Error: 'WeaviateBatchError('Query call with protocol GRPC batch failed with message <>)

Support
1

Hi Team,
I am getting the following Errors while writing object to weaviate.

  • Error: 'WeaviateBatchError(‘Query call with protocol GRPC batch failed with message CLIENT: Sent message larger than max (296704410 vs. 104858000).’)
  • Error: 'WeaviateBatchError(‘Query call with protocol GRPC batch failed with message Deadline Exceeded.’)
  • Error: 'WeaviateBatchError(‘Query call with protocol GRPC batch failed with message GOAWAY received; Error code: 1; Debug Text: [p]DATA: stream idle.’)

The following error occours when the batch size is set to >=100.

Server Setup Information

  • Weaviate Server Version:
  • Deployment Method: Kubernetes
  • Number of Running Nodes: One node
  • Weaviate Version: 1.25.0
  • weaviate python client: 4.6.5
2

hi @riturajraman !!

Welcome to our community :hugs:

This will happen if the amount of data you have per object, combined in a batch of X objects, is greater than GRPC_MAX_MESSAGE_SIZE.

If you inspect the server logs, you should see a tip that your need to increase this environment variable.

Increasing this number should result in bigger batches being allowed.

Let me know if that helps!

Thanks!

3

Hi @DudaNogueira
Thankyou for the clarification.
Can you please clarifiy the other two, the deadline exceed and goaway received error one or it is also related to the same GRPC_MAX_MESSAGE_SIZE?

4

Hi @DudaNogueira
After reducing the batch size, I am getting the Error: 'WeaviateBatchError(‘Query call with protocol GRPC batch failed with message Deadline Exceeded.’) and Error ‘Sent message larger than max’ is no more reflecting which was earlier an issue.

What could be the resolution.

5

hi!

You can either increase the environment variable GRPC_MAX_MESSAGE_SIZE , as described here

or reduce the batch size further.

The problem here is that your objects, combined by the batch size, are bigger than the default GRPC max message.

Let me know if that helps!

Thanks!