How to use reference for filter effciently?

Support
1

Description

what does these warnning mean?

{"level":"warning","msg":"Number of found nested reference results exceeds configured QUERY_MAXIMUM_RESULTS. This may result in search performance degradation or even out of memory errors.","nested_reference_results":11922,"query_maximum_results":10000,"time":"2024-03-07T10:45:46Z"}

Server Setup Information

  • Weaviate Server Version: 1.24.1
  • Deployment Method: docker
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: python 4.5.1

Any additional Information

2

Hi @shadowlin !

I believe that you have a cross reference that was requested on a query, and those references (of all the queried objects), are bringing more objects than what is set for QUERY_MAXIMUM_RESULTS.

here is the code where this warning comes from:

Let me know if that helps.

3

I am using a schema like this:

File

  • auth_code

TextChunk

  • text
  • belong_to_file (reference to file)

is these a good idea that I have to use Filter.by_ref("belong_to_file ").by_property(“auth_code”) to do filter?
It could be a lot of qualified text chunks(could be 100k or millions ) after the filter.

4

Hi @shadowlin!

is these a good idea that I have to use Filter.by_ref("belong_to_file ").by_property(“auth_code”) to do filter? It could be a lot of qualified text chunks(could be 100k or millions ) after the filter.

That could be very unperformant and memory intensive.

What are you trying to achieve? Maybe there is a more efficient way to do that

5

I want to filter text chunk by auth_code and the auth_code is on file level.
I think another way is to let every text chunk has a auth_code field but when update auth_code I would have to update each text chunk for the file.

7

Is auth_codeused to identify different customers on your side that shouldn’t be mixed? If yes, multi-tenancy might be a good alternative to filtering

8

I do want to use multi-tenancy but the auth design need to mix them up some use have the permission to access almost all the file so can’t do that:<