# GRPC trying to send message larger than max error : when trying collection.query.fetch\_objects

**URL:** <https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362>\
**Category:** Support\
**Tags:** bug, python\
**Created:** [August 15, 2024, 6:32pm UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362 "2024-08-15T18:32:03Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![pc10](https://avatars.discourse-cdn.com/v4/letter/p/73ab20/32.png) [@pc10](https://forum.weaviate.io/u/pc10)\
**Post date:** [August 15, 2024, 6:32pm UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/1 "2024-08-15T18:32:03Z")

</div>

### Description

I am in the process of migrating weaviate client from v3 to v4 and I am using haystack to integrate weaviate for my usecase. However when querying on collection using filters, I am getting the following grpc error

```auto

WeaviateQueryError: Query call with protocol GRPC search failed with message <AioRpcError of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUSTED
	details = "grpc: trying to send message larger than max (486438681 vs. 104858000)"
	debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"grpc: trying to send message larger than max (486438681 vs. 104858000)", grpc_status:8, created_time:"2024-08-15T18:22:12.13303888+00:00"}"

```

Here is the haystack link to query with filters. [Haystack](https://github.com/deepset-ai/haystack-core-integrations/blob/integrations/weaviate-v2.2.0/integrations/weaviate/src/haystack_integrations/document_stores/weaviate/document_store.py#L348)

I am filtering on a very small subset of documents in index and I see this error trying to get anything over 70-80 documents.

I also did increase my timeout config to 600 sec. With default timeout config, I can hardly query 5 docs.

Appreciate any help resolving this.

### Server Setup Information

- Weaviate Server Version:
- Deployment Method: k8
- Multi Node? Number of Running Nodes: 1
- Client Language and Version: Python
- Multitenancy?: No

### Any additional Information

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [August 15, 2024, 9:20pm UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/2 "2024-08-15T21:20:26Z")

</div>

Hi @pc10 !!

Welcome to our community 🤗

What is the size of each objects?

This error message indicates you are hitting the limit for the GRPC endpoint. This limit comes from here:

> <https://github.com/weaviate/weaviate/blob/502456b502bbffac6f47e880eb1343662c5be16e/adapters/handlers/grpc/server.go#L33>

My guess is that either payload in or out is passing that limit.

Let me know if that helps.

Thanks!

---

<div class="post-metadata">

**Author:** ![pc10](https://avatars.discourse-cdn.com/v4/letter/p/73ab20/32.png) [@pc10](https://forum.weaviate.io/u/pc10)\
**Post date:** [August 15, 2024, 9:52pm UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/3 "2024-08-15T21:52:53Z")

</div>

I am just trying to query using `collection.query.fetch_objects` with filter on a property that takes the documentName.

The query filter is something like this -

`filters = {"field": "documentId", "operator": "in", "value": ["document_name1", "document_name2", "document_name3"...etc] }`

is being sent to : [collection.query.fetch\_objects](https://github.com/deepset-ai/haystack-core-integrations/blob/integrations/weaviate-v2.2.0/integrations/weaviate/src/haystack_integrations/document_stores/weaviate/document_store.py#L364-L371)

So the payload to query isnt very huge.

On the flip side, When i extract ALL the documents in the index using

`collection.iterator(include_vector=False, return_properties=properties)` it returns the result. The index has document size about 9000 and return all of them without throwing an error.

It just with filtering, the underlying query.fecth\_objects function is failing the request.

---

<div class="post-metadata">

**Author:** ![pc10](https://avatars.discourse-cdn.com/v4/letter/p/73ab20/32.png) [@pc10](https://forum.weaviate.io/u/pc10)\
**Post date:** [August 15, 2024, 9:56pm UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/4 "2024-08-15T21:56:23Z")

</div>

I see the limit is being set in the python-client. Is there a way to override it.

I am confused as why simple query is maxing out the limit.

> <https://github.com/weaviate/weaviate-python-client/blob/v4.7.1/weaviate/connect/base.py#L23>

---

<div class="post-metadata">

**Author:** ![Dirk](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dirk/32/46_2.png) [@Dirk](https://forum.weaviate.io/u/Dirk)\
**Post date:** [August 16, 2024, 6:53am UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/5 "2024-08-16T06:53:52Z")

</div>

How big are your objects? You can set which objects are returned using return\_properties=[“prop1”,…]. by default all non-blob properties are returned

---

<div class="post-metadata">

**Author:** ![pc10](https://avatars.discourse-cdn.com/v4/letter/p/73ab20/32.png) [@pc10](https://forum.weaviate.io/u/pc10)\
**Post date:** [August 27, 2024, 6:15pm UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/6 "2024-08-27T18:15:09Z")

</div>

Sorry for delayed response. Realized one of the properties being queried from weaviate (using haystack) is a metadata field that is quite large. Excluding that solved this error.

> [@Dirk](#):
>
> How big are your objects? You can set which objects are returned using return\_properties=[“prop1”,…]. by default all non-blob properties are returned

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [August 27, 2024, 6:23pm UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/7 "2024-08-27T18:23:57Z")

</div>

Glad to hear that, @pc10 !!

Thanks for sharing!

---

<div class="post-metadata">

**Author:** ![dch239](https://avatars.discourse-cdn.com/v4/letter/d/e79b87/32.png) [@dch239](https://forum.weaviate.io/u/dch239)\
**Post date:** [September 20, 2024, 11:29pm UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/8 "2024-09-20T23:29:47Z")

</div>

Why is the GRPC MAX\_GRPC\_MESSAGE\_LENGTH hardcoded on both weaviate core and client? I have a simple weaviate object with two properties, title and poster. My avg images are around 3 mb. When i try indexing using batch.dynamic() i see the same error described in this thread. Using batch.fixed() is a hacky workaround which can fail. It seems like such a basic feature. Not to mention we have been bottlenecked and forced to abandon Weaviate just because of this issue.

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [September 23, 2024, 8:51am UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/9 "2024-09-23T08:51:56Z")

</div>

hi @dch239 !! Welcome to our community 🤗

Using the fixed size batch is not a hacky solution at all.

the dynamic size batch will calculate the amount of objects to send based on the latency of the connection to the server.

However, if your objects are big (for example, images), the size of your batches shouldn’t be defined by the number of objects to send, but by the size of each objects that are being sent.

On that case, using fixed size is more interesting, as you have a clear control of the size of the batch, and avoid to have a few big sized batches in favor of correctly sized batches depending on your objects.

Also, given enough resources, that is not a bottleneck, as you can increase the concurrent requests and have enough nodes on a cluster that can handle the load of your data.

Let me know if that helps.

Thanks!

---

<div class="post-metadata">

**Author:** ![dch239](https://avatars.discourse-cdn.com/v4/letter/d/e79b87/32.png) [@dch239](https://forum.weaviate.io/u/dch239)\
**Post date:** [September 25, 2024, 8:21am UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/10 "2024-09-25T08:21:58Z")

</div>

Nevertheless, the 10 mb harcoded limit still applies, right? If each of my image is \>10 mb would I be able to import even with a fixed batch size of 1? My point is why is MAX\_GRPC\_MESSAGE\_LENGTH hardcoded in the python client (weaviate/connect/base.py) and server side? Why is this not a variable in docker compose since it is technically possible to set the max message size?

---

<div class="post-metadata">

**Author:** ![gfwgfw](https://avatars.discourse-cdn.com/v4/letter/g/c0e974/32.png) [@gfwgfw](https://forum.weaviate.io/u/gfwgfw)\
**Post date:** [December 4, 2024, 4:48am UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/11 "2024-12-04T04:48:55Z")

</div>

Meet this issue today, and fix with set this environment variable:  
GRPC\_MAX\_MESSAGE\_SIZE=104857600

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [December 6, 2024, 11:28am UTC](https://forum.weaviate.io/t/grpc-trying-to-send-message-larger-than-max-error-when-trying-collection-query-fetch-objects/3362/12 "2024-12-06T11:28:08Z")

</div>

hi @gfwgfw !!

This environment variable `GRPC_MAX_MESSAGE_SIZE` was recently reproduced, so I have changed the solution to this thread to yours!

Thanks for sharing!