GRPC trying to send message larger than max error : when trying collection.query.fetch_objects

Support
,
1

Description

I am in the process of migrating weaviate client from v3 to v4 and I am using haystack to integrate weaviate for my usecase. However when querying on collection using filters, I am getting the following grpc error


WeaviateQueryError: Query call with protocol GRPC search failed with message <AioRpcError of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUSTED
	details = "grpc: trying to send message larger than max (486438681 vs. 104858000)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"grpc: trying to send message larger than max (486438681 vs. 104858000)", grpc_status:8, created_time:"2024-08-15T18:22:12.13303888+00:00"}"

Here is the haystack link to query with filters. Haystack

I am filtering on a very small subset of documents in index and I see this error trying to get anything over 70-80 documents.

I also did increase my timeout config to 600 sec. With default timeout config, I can hardly query 5 docs.

Appreciate any help resolving this.

Server Setup Information

  • Weaviate Server Version:
  • Deployment Method: k8
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: Python
  • Multitenancy?: No

Any additional Information

2

Hi @pc10 !!

Welcome to our community :hugs:

What is the size of each objects?

This error message indicates you are hitting the limit for the GRPC endpoint. This limit comes from here:

My guess is that either payload in or out is passing that limit.

Let me know if that helps.

Thanks!

3

I am just trying to query using collection.query.fetch_objects with filter on a property that takes the documentName.

The query filter is something like this -

filters = {"field": "documentId", "operator": "in", "value": ["document_name1", "document_name2", "document_name3"...etc] }

is being sent to : collection.query.fetch_objects

So the payload to query isnt very huge.

On the flip side, When i extract ALL the documents in the index using

collection.iterator(include_vector=False, return_properties=properties) it returns the result. The index has document size about 9000 and return all of them without throwing an error.

It just with filtering, the underlying query.fecth_objects function is failing the request.

4

I see the limit is being set in the python-client. Is there a way to override it.

I am confused as why simple query is maxing out the limit.

5

How big are your objects? You can set which objects are returned using return_properties=[“prop1”,…]. by default all non-blob properties are returned

6

Sorry for delayed response. Realized one of the properties being queried from weaviate (using haystack) is a metadata field that is quite large. Excluding that solved this error.

1 Like
7

Glad to hear that, @pc10 !!

Thanks for sharing!