Why enabled PQ significantly impacted recall. (version 1.23.7)

Support
1

Description

Hey Weaviate community, I am benchmarking the recall of vector search for a dataset of 10M, with each vector dimension as 768 , Weaviate server version 1.23.7
I compared the impact on recall with Product Quantization(PQ) enabled/ disabled, the only different config change when pq is enabled:

‘pq’: {‘enabled’: True, ‘bitCompression’: False, ‘segments’: 96, ‘centroids’: 256, ‘trainingLimit’: 100000, ‘encoder’: {‘type’: ‘kmeans’, ‘distribution’: ‘log-normal’}}},

Or when pq is disabled,

‘pq’: {‘enabled’: False, …}

However, the recall result reduced significantly when the PQ enabled, see attached screen for reference. The result is inconsistent with this weaviate blog on PQ.
Could anyone provide some insight on why such significantly impact when the PQ enabled?

recall_PQ
recall_PQ1920×1185 190 KB

Server Setup Information

  • Weaviate Server Version: 1.23.7
  • Deployment Method: k8s through helm chart
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: Python client 3.21.0

Any additional Information

2

Hi @fairymane! Welcome to our community :hugs:

This is very interesting. I will relay that to our team.

Have you tried with a higher segments parameter?

Thanks!

3

image
image3770×1216 483 KB

@DudaNogueira
By setting the “segments” : 0 when PQ enabled, the result is actually not bad, as attached.

The only issue is changing such configuration needs to migrate data to a new collection with new config settings to be effective, which is time consuming.

4

Yes, that’s the AutoPQ kicking in, so it will evaluate your vectors and find out the best number of segments to train your data with.

What was the resulting segment value on that case?

Thanks!

5

I just configured “segments” : 0 at collection creation time.

How do I find out the resulting segment value?