Error: 'WeaviateBatchError('Query call with protocol GRPC batch failed with message <>)

Support
1

Hi Team,
I am getting the following Errors while writing object to weaviate.

  • Error: 'WeaviateBatchError(‘Query call with protocol GRPC batch failed with message CLIENT: Sent message larger than max (296704410 vs. 104858000).’)
  • Error: 'WeaviateBatchError(‘Query call with protocol GRPC batch failed with message Deadline Exceeded.’)
  • Error: 'WeaviateBatchError(‘Query call with protocol GRPC batch failed with message GOAWAY received; Error code: 1; Debug Text: [p]DATA: stream idle.’)

The following error occours when the batch size is set to >=100.

Server Setup Information

  • Weaviate Server Version:
  • Deployment Method: Kubernetes
  • Number of Running Nodes: One node
  • Weaviate Version: 1.25.0
  • weaviate python client: 4.6.5
2

hi @riturajraman !!

Welcome to our community :hugs:

This will happen if the amount of data you have per object, combined in a batch of X objects, is greater than GRPC_MAX_MESSAGE_SIZE.

If you inspect the server logs, you should see a tip that your need to increase this environment variable.

Increasing this number should result in bigger batches being allowed.

Let me know if that helps!

Thanks!

3

Hi @DudaNogueira
Thankyou for the clarification.
Can you please clarifiy the other two, the deadline exceed and goaway received error one or it is also related to the same GRPC_MAX_MESSAGE_SIZE?

4

Hi @DudaNogueira
After reducing the batch size, I am getting the Error: 'WeaviateBatchError(‘Query call with protocol GRPC batch failed with message Deadline Exceeded.’) and Error ‘Sent message larger than max’ is no more reflecting which was earlier an issue.

What could be the resolution.

5

hi!

You can either increase the environment variable GRPC_MAX_MESSAGE_SIZE , as described here

or reduce the batch size further.

The problem here is that your objects, combined by the batch size, are bigger than the default GRPC max message.

Let me know if that helps!

Thanks!

6

Hello,
I got similar error when doing batch insert.
Weaviate is running in a docker container:

the yaml:
services:
weaviate2:
command:

My insert code:
try:
batch_size = 1
# with client.batch.dynamic() as batch:
# with client.batch.fixed_size(batch_size=batch_size, concurrent_requests=5) as batch:
with client.batch.fixed_size(batch_size=batch_size) as batch:
for i, obj_data in enumerate(data):
try:
batch.add_object(
properties=obj_data,
collection=collection_name,
)

            if (i + 1) % batch_size == 0:
                print(f"Inserted: {i + 1} objects")

        except weaviate.exceptions.WeaviateBaseError as e:
            print(f"Error adding object: {e}")
            print(f"Data that caused the error: {obj_data}")
            #client.close()
            continue

failed_objs = client.batch.failed_objects  # Get failed objects
if failed_objs:
    print(f"Number of failed objects in the first batch: {len(failed_objs)}")
    for i, failed_obj in enumerate(failed_objs, 1):
        print(f"Failed object {i}:")
        print(f"Error message: {failed_obj.message}")
else:
    print("Finished insertig all objects.")

I change the size from 104857600 to 296704410 and got the following error with the code client.batch.dynamic():
{‘message’: ‘Failed to send all objects in a batch of 48’, ‘error’: “WeaviateBatchError(‘Query call with protocol GRPC batch failed with message Received RST_STREAM with error code 8.’)”}
{‘message’: ‘Failed to send 48 objects in a batch of 48. Please inspect client.batch.failed_objects or collection.batch.failed_objects for the failed objects.’}

when I change automatic with client.batch.fixed_size(batch_size=batch_size)

Note: the error I got when I lower the batch_size=1…!
‘Failed to send all objects in a batch of 1’, ‘error’: 'WeaviateInsertManyAllFailedError('Every object failed during insertion. Here is the set of all errors: send POST request: Post “http://t2v-transformers:8080/vectors”: context deadline exceeded (Client.Timeout exceeded while awaiting headers

The code with batch.automatic run fine in another linux server but failed on my windows laptop with 32 GB of Ram and 2 gb Samsung 980 ssd.

Any solution?

7

Hi! The error indicates that the transformer is timing out.

batch.dynamic will calculate the the batch size based on server latency upon ingestion. Considering you are on a local environment, this can easily overwhelm your server or the vectorizer.

Do you have any readings on resources like memory and cpu for those containers? Is the transformer running on GPU?

Also, try to always use 1.XX.latest as you will benefit from all latest patches.

Let me know if this helps!