Replica search GET query returns different results

Support
1

Hello,

I have Weaviate 1.20.1 running with 2 replicas and 2 shards for a particular class with ~4000000 documents. Doing a dense vector search for weaviate-0 replica skips results with a higher certainty. I am testing each replica by port-forwarding from the pod.
So for example i have two documents DocA /DocB and Query .

certainty of Query & DocA = 0.881561815738678
certainty of Query & DocB = 0.7634845674037933

When i query weaviate-1 to get the top result, it correctly returns DocA for the Query . But when i query weaviate-0 it returns DocB as the closest, even though when i examine DocA in weaviate-0 (by adding the where filter)the certainty is 0.881561815738678 .

Let me know if more details are needed.
Thank you in advanced!

2

Hi @darpan - that’s odd. I wonder if the query is being run with ConsistencyLevel.ONE somehow.

Could you please try running the same query with different consistency levels (ALL/QUORUM/ONE)?

In Python, for example - it would be set like:

.with_consistency_level(ConsistencyLevel.ALL)
.with_consistency_level(ConsistencyLevel.QUORUM)
.with_consistency_level(ConsistencyLevel.ONE)

3

Thanks @jphwang :slight_smile: for your quick reply.
Apologize for the delay in response, with all 3, it returns same results from each container.

4

To reproduce the issue run the steps below, let me know if you come across anything or have any issues in setting this up:
Related to Slack

  1. Create a new directory
  2. Grab the .gz from this dataset and add it to the new directory gfissore/arxiv-abstracts-2021 at main
  3. unzip contents of “reproduce_index.zip” (Located in this google drive in the new directory
  4. run “docker-compose up -d” to create a 2 instance weaviate
  5. run “python create_schema.py” to create the Article Schema
  6. run “python index.py” to index all documents. This should take a little bit. To speed up, you can bump up the replicas for t2v-transformers if you have more resources.
  7. Run the GQL query below against both http://localhost:6001/v1/graphql and http://localhost:6002/v1/graphql to see the difference
{
    Get {
        Article(
            nearText: {concepts: ["A pilgrimage to gravity on GPUs"]
            }
            limit: 12
        ) {
            title
            body
            _additional {
                isConsistent
                certainty
            }
        }
    }
}
6

Hi @darpan I wrote a reply but I see you’re getting assistance from Parker and he would know much better then me :slight_smile:

7

Thanks @jphwang :slight_smile:

I will update here when its resolved.

1 Like
8

@darpan would you be willing to upgrade to v1.21.2? We have since included some changes that improve the resiliency of replicated search.

I did setup a cluster to try and reproduce your issue according to the steps above, but everything seemed to work as expected for me.

Maybe the upgrade will clear up your issue