GRPC trying to send message larger than max error : when trying collection.query.fetch_objects

pc10 · August 15, 2024, 6:32pm

Description

I am in the process of migrating weaviate client from v3 to v4 and I am using haystack to integrate weaviate for my usecase. However when querying on collection using filters, I am getting the following grpc error


WeaviateQueryError: Query call with protocol GRPC search failed with message <AioRpcError of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUSTED
	details = "grpc: trying to send message larger than max (486438681 vs. 104858000)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"grpc: trying to send message larger than max (486438681 vs. 104858000)", grpc_status:8, created_time:"2024-08-15T18:22:12.13303888+00:00"}"

Here is the haystack link to query with filters. Haystack

I am filtering on a very small subset of documents in index and I see this error trying to get anything over 70-80 documents.

I also did increase my timeout config to 600 sec. With default timeout config, I can hardly query 5 docs.

Appreciate any help resolving this.

Server Setup Information

Weaviate Server Version:
Deployment Method: k8
Multi Node? Number of Running Nodes: 1
Client Language and Version: Python
Multitenancy?: No

Any additional Information

DudaNogueira · August 15, 2024, 9:20pm

Hi @pc10 !!

Welcome to our community

What is the size of each objects?

This error message indicates you are hitting the limit for the GRPC endpoint. This limit comes from here:

github.com

weaviate/weaviate/blob/502456b502bbffac6f47e880eb1343662c5be16e/adapters/handlers/grpc/server.go#L33


      
          	"github.com/weaviate/weaviate/usecases/auth/authentication/composer"
          	"google.golang.org/grpc"
          	"google.golang.org/grpc/credentials"
          	_ "google.golang.org/grpc/encoding/gzip" // Install the gzip compressor
          	"google.golang.org/grpc/health/grpc_health_v1"
          
          	v0 "github.com/weaviate/weaviate/adapters/handlers/grpc/v0"
          	v1 "github.com/weaviate/weaviate/adapters/handlers/grpc/v1"
          )
          
          const maxMsgSize = 104858000 // 10mb, needs to be synchronized with clients
          
          func CreateGRPCServer(state *state.State) *GRPCServer {
          	o := []grpc.ServerOption{
          		grpc.MaxRecvMsgSize(maxMsgSize),
          		grpc.MaxSendMsgSize(maxMsgSize),
          	}
          
          	// Add TLS creds for the GRPC connection, if defined.
          	if len(state.ServerConfig.Config.GRPC.CertFile) > 0 || len(state.ServerConfig.Config.GRPC.KeyFile) > 0 {
          		c, err := credentials.NewServerTLSFromFile(state.ServerConfig.Config.GRPC.CertFile,

My guess is that either payload in or out is passing that limit.

Let me know if that helps.

Thanks!

pc10 · August 15, 2024, 9:52pm

I am just trying to query using collection.query.fetch_objects with filter on a property that takes the documentName.

The query filter is something like this -

filters = {"field": "documentId", "operator": "in", "value": ["document_name1", "document_name2", "document_name3"...etc] }

is being sent to : collection.query.fetch_objects

So the payload to query isnt very huge.

On the flip side, When i extract ALL the documents in the index using

collection.iterator(include_vector=False, return_properties=properties) it returns the result. The index has document size about 9000 and return all of them without throwing an error.

It just with filtering, the underlying query.fecth_objects function is failing the request.

pc10 · August 15, 2024, 9:56pm

I see the limit is being set in the python-client. Is there a way to override it.

I am confused as why simple query is maxing out the limit.

github.com

weaviate/weaviate-python-client/blob/v4.7.1/weaviate/connect/base.py#L23


      
          
          from pydantic import BaseModel, field_validator, model_validator
          
          from weaviate.config import Proxies
          from weaviate.types import NUMBER
          
          
          JSONPayload = Union[dict, list]
          TIMEOUT_TYPE_RETURN = Tuple[NUMBER, NUMBER]
          MAX_GRPC_MESSAGE_LENGTH = 104858000  # 10mb, needs to be synchronized with GRPC server
          GRPC_DEFAULT_OPTIONS = [
              ("grpc.max_send_message_length", MAX_GRPC_MESSAGE_LENGTH),
              ("grpc.max_receive_message_length", MAX_GRPC_MESSAGE_LENGTH),
          ]
          
          
          class ProtocolParams(BaseModel):
              host: str
              port: int
              secure: bool

Dirk · August 16, 2024, 6:53am

How big are your objects? You can set which objects are returned using return_properties=[“prop1”,…]. by default all non-blob properties are returned

pc10 · August 27, 2024, 6:15pm

Sorry for delayed response. Realized one of the properties being queried from weaviate (using haystack) is a metadata field that is quite large. Excluding that solved this error.

DudaNogueira · August 27, 2024, 6:23pm

Glad to hear that, @pc10 !!

Thanks for sharing!

dch239 · September 20, 2024, 11:29pm

Why is the GRPC MAX_GRPC_MESSAGE_LENGTH hardcoded on both weaviate core and client? I have a simple weaviate object with two properties, title and poster. My avg images are around 3 mb. When i try indexing using batch.dynamic() i see the same error described in this thread. Using batch.fixed() is a hacky workaround which can fail. It seems like such a basic feature. Not to mention we have been bottlenecked and forced to abandon Weaviate just because of this issue.

DudaNogueira · September 23, 2024, 8:51am

hi @dch239 !! Welcome to our community

Using the fixed size batch is not a hacky solution at all.

the dynamic size batch will calculate the amount of objects to send based on the latency of the connection to the server.

However, if your objects are big (for example, images), the size of your batches shouldn’t be defined by the number of objects to send, but by the size of each objects that are being sent.

On that case, using fixed size is more interesting, as you have a clear control of the size of the batch, and avoid to have a few big sized batches in favor of correctly sized batches depending on your objects.

Also, given enough resources, that is not a bottleneck, as you can increase the concurrent requests and have enough nodes on a cluster that can handle the load of your data.

Let me know if that helps.

Thanks!

dch239 · September 25, 2024, 8:21am

Nevertheless, the 10 mb harcoded limit still applies, right? If each of my image is >10 mb would I be able to import even with a fixed batch size of 1? My point is why is MAX_GRPC_MESSAGE_LENGTH hardcoded in the python client (weaviate/connect/base.py) and server side? Why is this not a variable in docker compose since it is technically possible to set the max message size?

gfwgfw · December 4, 2024, 4:48am

Meet this issue today, and fix with set this environment variable:
GRPC_MAX_MESSAGE_SIZE=104857600

DudaNogueira · December 6, 2024, 11:28am

hi @gfwgfw !!

This environment variable GRPC_MAX_MESSAGE_SIZE was recently reproduced, so I have changed the solution to this thread to yours!

Thanks for sharing!

Topic		Replies	Views
Issue with python client v4 - GRPC batch failed with message Sent message larger than max Support	2	446	February 9, 2024
Error: 'WeaviateBatchError('Query call with protocol GRPC batch failed with message <>) Support	2	25	December 17, 2024
GRPC Resource Exhausted Error Support	1	188	August 22, 2024
[Question] WeaviateBatchError('Query call with protocol GRPC batch failed with message Deadline Exceeded') while writing store Support integration , wcs	3	358	April 10, 2024
GRCP connection failure when processing data-intensive batches Support documentation	5	185	August 19, 2024

GRPC trying to send message larger than max error : when trying collection.query.fetch_objects

Description

Server Setup Information

Any additional Information

Related topics