Inconsistent numbers of objects using Get and Aggregate

Support
1

Description

I have three hierarchical collections:
Corpus ← Document ← Chunk

Chunks refer to the document they belong to using cross-references, documents refer to the corpus they belong to using cross-references.

I have recently come across the rare phenomenon that Aggregate queries, which we use to keep track of how many chunks are in a document and how many documents are in a corpus etc. return a larger number of objects using meta count than Get queries actually getting those objects. (Nota bene the limit for the maximum number of objects to get is definitely higher than both numbers so that it should have no influence.)

Example queries:

  • Aggregate query returns a count of 68 objects

    • query: {Aggregate{Chunk(where: {path: ["document", "Document", "id"] operator: Equal valueText: "3154c57d-9a70-415d-b3fe-480562d25c01"} limit: 1100 ){meta {count}}}}
    • result: {"data": {"Aggregate": {"Chunk": [{"meta": {"count": 68}}]}}

  • Get query return 63 Chunk objects

    • query: {Get{Chunk(where: {path: ["document", "Document", "id"] operator: Equal valueText: "3154c57d-9a70-415d-b3fe-480562d25c01"} limit: 1100 ){_additional {id}}}}
    • result:
{
    "data": {
        "Get": {
            "Chunk": [
                {
                    "_additional": {
                        "id": "1e1b6df2-c6dc-4aa4-8837-5781ebe04a23"
                    }
                },
                {
                    "_additional": {
                        "id": "85d23a48-c468-4422-aa24-1bea7294d3a8"
                    }
                },
                {
                    "_additional": {
                        "id": "a1a8c958-0199-4c1d-84b5-6fc023293003"
                    }
                },
                {
                    "_additional": {
                        "id": "4b3206c0-e8b4-417a-8466-964d946dcc7e"
                    }
                },
                {
                    "_additional": {
                        "id": "1fe3a960-cb7a-4a24-aaa4-33c87084b95f"
                    }
                },
                {
                    "_additional": {
                        "id": "c9aeb5cd-5618-4083-ba9b-7870934b4eac"
                    }
                },
                {
                    "_additional": {
                        "id": "26b3947d-22c0-4fe3-8037-c4f6f894b368"
                    }
                },
                {
                    "_additional": {
                        "id": "cab382b0-87f6-4d88-9554-c16a4c1ec1c9"
                    }
                },
                {
                    "_additional": {
                        "id": "574871d8-251e-49f0-92e8-a12ac126bff4"
                    }
                },
                {
                    "_additional": {
                        "id": "d96ba70b-1aef-4a50-9a83-2a4a1f267770"
                    }
                },
                {
                    "_additional": {
                        "id": "63eb0974-b18d-4224-a8ce-9fbb17aee6e7"
                    }
                },
                {
                    "_additional": {
                        "id": "226dc976-42f9-4362-8451-b270a03d5eae"
                    }
                },
                {
                    "_additional": {
                        "id": "61563590-39b1-4175-a716-894f10fc27bb"
                    }
                },
                {
                    "_additional": {
                        "id": "eacce4fc-f100-4f31-92e1-d56585107a38"
                    }
                },
                {
                    "_additional": {
                        "id": "3f6227b0-7415-4450-b0c3-9110e9a4ee37"
                    }
                },
                {
                    "_additional": {
                        "id": "4978d106-f70c-44ee-a73a-26dd3848e243"
                    }
                },
                {
                    "_additional": {
                        "id": "9c9e2ab8-ed31-4c89-b8bc-275485e70007"
                    }
                },
                {
                    "_additional": {
                        "id": "28fedc62-164c-43d9-8e9c-99e4e8e0d101"
                    }
                },
                {
                    "_additional": {
                        "id": "e8a29452-584c-4298-9bc8-64b81eeda7c6"
                    }
                },
                {
                    "_additional": {
                        "id": "b9a6adb5-55b5-4d6a-8fe3-cd587862ab71"
                    }
                },
                {
                    "_additional": {
                        "id": "fdbc016a-1c7b-448b-9593-3da2353e70d6"
                    }
                },
                {
                    "_additional": {
                        "id": "302ca135-02d4-4387-9e6c-45197e5360e3"
                    }
                },
                {
                    "_additional": {
                        "id": "e78e6758-d877-4210-b11d-8424381a9c7a"
                    }
                },
                {
                    "_additional": {
                        "id": "24c9b423-5c3e-42ac-bf16-35eb0a2b600f"
                    }
                },
                {
                    "_additional": {
                        "id": "3ab58cc5-47f4-4e95-9438-241ea6c0b6f0"
                    }
                },
                {
                    "_additional": {
                        "id": "c90c1aea-b68a-4918-b783-fad086266599"
                    }
                },
                {
                    "_additional": {
                        "id": "5642fc62-8a85-46c0-a2b0-de0b0edfe870"
                    }
                },
                {
                    "_additional": {
                        "id": "f6142f9b-d6e4-4b6a-8c35-17ef737f2336"
                    }
                },
                {
                    "_additional": {
                        "id": "0669522e-ebe8-4ed0-963a-f5204a2e793b"
                    }
                },
                {
                    "_additional": {
                        "id": "4f96f6e3-e0b0-4f22-b81d-4eba34521ab0"
                    }
                },
                {
                    "_additional": {
                        "id": "2ab46511-bb43-43ac-8c33-92ff054c561d"
                    }
                },
                {
                    "_additional": {
                        "id": "e55ac626-6c76-431f-9dd8-0fbb652e73dd"
                    }
                },
                {
                    "_additional": {
                        "id": "8efb0cdf-fc3f-4907-b7ce-9dfb632aac03"
                    }
                },
                {
                    "_additional": {
                        "id": "feebf499-d048-429f-b5cf-efaacaddba5d"
                    }
                },
                {
                    "_additional": {
                        "id": "a79188ca-cc7c-4161-a8b5-5c8cedfdb23c"
                    }
                },
                {
                    "_additional": {
                        "id": "fad39570-1131-40a7-863b-d561542f47c8"
                    }
                },
                {
                    "_additional": {
                        "id": "48811b2a-e9f3-4fe3-b89e-387d52f3d5e9"
                    }
                },
                {
                    "_additional": {
                        "id": "827080f6-019b-4f1c-b0da-d62fe8321d3d"
                    }
                },
                {
                    "_additional": {
                        "id": "8c0f8f4e-7127-46a9-bb27-6d78f53998c3"
                    }
                },
                {
                    "_additional": {
                        "id": "9c2102be-f376-46b6-8a38-b395dc782608"
                    }
                },
                {
                    "_additional": {
                        "id": "63c18dc8-ec58-4516-b784-468387c15c67"
                    }
                },
                {
                    "_additional": {
                        "id": "6330175f-de7f-4f10-8b18-de9a4bfdb45e"
                    }
                },
                {
                    "_additional": {
                        "id": "94d8feac-4e9f-40df-8d02-f9f9a961d5fe"
                    }
                },
                {
                    "_additional": {
                        "id": "7eefe892-27ed-4137-a833-e0df1e1eaafe"
                    }
                },
                {
                    "_additional": {
                        "id": "8c5cb0f6-2834-4c06-88c2-cd719edd9f29"
                    }
                },
                {
                    "_additional": {
                        "id": "3035ec04-dc4d-4941-8311-3940876af9d0"
                    }
                },
                {
                    "_additional": {
                        "id": "a02db6e1-ce4d-4a50-9522-b6cf0b904106"
                    }
                },
                {
                    "_additional": {
                        "id": "0e972e82-4d74-4470-8132-cc4b28fb1055"
                    }
                },
                {
                    "_additional": {
                        "id": "ecaef2f9-8671-4bbf-9f21-88e59aa50e18"
                    }
                },
                {
                    "_additional": {
                        "id": "efe30d68-fa3b-455c-a740-5e012f8edb28"
                    }
                },
                {
                    "_additional": {
                        "id": "6c7c96ee-e9de-45e6-8e8d-c13da731382a"
                    }
                },
                {
                    "_additional": {
                        "id": "ea3ec993-43fc-43e4-a038-ad83d6a3be43"
                    }
                },
                {
                    "_additional": {
                        "id": "c6cd581c-b831-4263-8128-6a048773c9ad"
                    }
                },
                {
                    "_additional": {
                        "id": "bda917ac-bae2-469f-9a3d-428a0b6c477a"
                    }
                },
                {
                    "_additional": {
                        "id": "7624105f-072f-4d71-ae29-63b497dd1a4e"
                    }
                },
                {
                    "_additional": {
                        "id": "635aac1d-b0d1-4ee3-8b6f-f2c69c1ac220"
                    }
                },
                {
                    "_additional": {
                        "id": "62a8f35f-c713-4b60-8e87-26bf1440a9a5"
                    }
                },
                {
                    "_additional": {
                        "id": "60a8cd09-7955-443f-981d-f5d9b6a07b36"
                    }
                },
                {
                    "_additional": {
                        "id": "5482791b-1af8-4d3e-b37c-d854445759e6"
                    }
                },
                {
                    "_additional": {
                        "id": "b8c595cd-0002-4505-b5db-c0eeb604fe5a"
                    }
                },
                {
                    "_additional": {
                        "id": "f28771dd-db46-4b04-995b-ee11e43c8bf3"
                    }
                },
                {
                    "_additional": {
                        "id": "dba63357-42d4-402a-bfe4-211d96dd9d05"
                    }
                },
                {
                    "_additional": {
                        "id": "537dd764-c989-4efa-a2e3-08840a0f3d27"
                    }
                }
            ]
        }
    }
}

Is there any good explanation for this kind of phenomenon? Or could this be a bug? I would be very thankful for any hints… Thank you very much!

Server Setup Information

  • Weaviate Server Version: 1.24.3
  • Deployment Method: k8s
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: Python v3
  • Multitenancy?: no

Any additional Information

None

2

hi @jan.strunk !!

Sorry for the delay here, missed this one :frowning:

Is this a big dataset?

Do you think we could reindex it on a second cluster and check if this still persists?

Was all that data added at 1.24.3 or you migrated it at some point?

And finally, what client version you are using? have you tried the latest python v3 client?

Thanks!