Replica search GET query returns different results

Hello,

I have Weaviate 1.20.1 running with 2 replicas and 2 shards for a particular class with ~4000000 documents. Doing a dense vector search for weaviate-0 replica skips results with a higher certainty. I am testing each replica by port-forwarding from the pod.
So for example i have two documents DocA /DocB and Query .

certainty of Query & DocA = 0.881561815738678
certainty of Query & DocB = 0.7634845674037933

When i query weaviate-1 to get the top result, it correctly returns DocA for the Query . But when i query weaviate-0 it returns DocB as the closest, even though when i examine DocA in weaviate-0 (by adding the where filter)the certainty is 0.881561815738678 .

Let me know if more details are needed.
Thank you in advanced!

Hi @darpan - that’s odd. I wonder if the query is being run with ConsistencyLevel.ONE somehow.

Could you please try running the same query with different consistency levels (ALL/QUORUM/ONE)?

In Python, for example - it would be set like:

.with_consistency_level(ConsistencyLevel.ALL)
.with_consistency_level(ConsistencyLevel.QUORUM)
.with_consistency_level(ConsistencyLevel.ONE)

Thanks @jphwang :slight_smile: for your quick reply.
Apologize for the delay in response, with all 3, it returns same results from each container.

To reproduce the issue run the steps below, let me know if you come across anything or have any issues in setting this up:
Related to Slack

  1. Create a new directory
  2. Grab the .gz from this dataset and add it to the new directory gfissore/arxiv-abstracts-2021 at main
  3. unzip contents of “reproduce_index.zip” (Located in this google drive in the new directory
  4. run “docker-compose up -d” to create a 2 instance weaviate
  5. run “python create_schema.py” to create the Article Schema
  6. run “python index.py” to index all documents. This should take a little bit. To speed up, you can bump up the replicas for t2v-transformers if you have more resources.
  7. Run the GQL query below against both http://localhost:6001/v1/graphql and http://localhost:6002/v1/graphql to see the difference
{
    Get {
        Article(
            nearText: {concepts: ["A pilgrimage to gravity on GPUs"]
            }
            limit: 12
        ) {
            title
            body
            _additional {
                isConsistent
                certainty
            }
        }
    }
}

Hi @darpan I wrote a reply but I see you’re getting assistance from Parker and he would know much better then me :slight_smile:

Thanks @jphwang :slight_smile:

I will update here when its resolved.

1 Like

@darpan would you be willing to upgrade to v1.21.2? We have since included some changes that improve the resiliency of replicated search.

I did setup a cluster to try and reproduce your issue according to the steps above, but everything seemed to work as expected for me.

Maybe the upgrade will clear up your issue

Thanks @parkerduckworth for taking a look at this, i know this is a difficult one.

I have had to scale down the instance to single node for our production for now to avoid complaints. When I upgrade/recreate the schema in the next few weeks, i will try to use the latest version and see if the issue is still occurring.

Will update here when i find something.