Different score for the same entry on different replicas

General
1

hi! I have a question regarding results of nearText search. We can observe different scores for every replica we hit with our query (run on the localhost). The results are constant for given replica - we run it couple times to make sure the results are the same ). Results look like:

replica-0
1: {“certainty":0.9116471707820892,"id":"8097b3e0-2c14-4a18-801c-0446850dcb52"}

replica-1
1: {“certainty":0.916239321231842,"id":"8097b3e0-2c14-4a18-801c-0446850dcb52”}

replica-2
1: {“certainty":0.9145565330982208,"id":"088ed5fa-d6d7-4056-9723-5c34492fd39c”}
2: {“certainty":0.9143496453762054,"id":"63d00cde-8b3f-4518-9b0b-f8cc380e10b0”}
3: {“certainty":0.9127094149589539,"id":"99f2d72e-e297-4d7c-a2d5-d14aed11b1bb”}
4: {“certainty":0.9125392138957977,"id":"8097b3e0-2c14-4a18-801c-0446850dcb52”}

My questions:

  • is it expected to have different score for the same record on different replica?
  • if there are some issues with replications (we observed that groups of records can be found on one of the replica and not on the other) causing different structure of the search graph (we are using HNSW) can it lead to different score? (and can those graph be different anyway?)
  • can these differences be caused by implementation of ANN?

Our system consists of 3 replicas, 1 class containing about 10 millions of records. We experienced some issues with memory before (which caused a lot of crashing) and we experimented with LIMIT_RESOURCES (which is now on) and PQ (it also is on). Our current version of WV is 1.24.2.

Thanks in advance for any suggestions/answers. (in case that some info is missing please let me know).

2

hi @l.sajdak ! Welcome to our community :hugs:

This is not expected.

Maybe those crashes while ingesting prevented some objects to be indexed and now those calculations is differing :thinking:

I have asked internally, and should get some insights from our team.

Thanks!

1 Like