Cannot process batch: not enough memory

Support
1

Hi , configured batch on
client.batch.configure(batch_size=100,dynamic=True,consistency_level=“ALL”,connection_error_retries=3,num_workers=2)
perform batch insertion and getting exception
UnexpectedStatusCodeException: batch response! Unexpected status code: 500, with response body: {‘error’: [{‘message’: ‘batch objects: &fmt.wrapError{msg:“cannot process batch: not enough memory”, err:(*errors.errorString)(0xc00004c020)}’}]}.

Weaviate version :1.23.11
Deployment type: Kubernetes cluster
Python client : 3.24.2
available memory per container: 4GB

2

hi @Dharanish !

Can you try using the new python v4 client?

Also, does the server has enough memory considering the amount of objects to be ingested?

And finally, does it happens if you reduce the batch size?

If you try to the python v4 client, I suggest using the dynamic batch size, so Weaviate server can adjust the batch size while communicating with the client.

Thanks!

3

Hi @DudaNogueira , My cluster has enough memory and batch size is 20 and using client 4.5.7, still getting this error

4

This is the origin of this error:

It may be hitting the limits while ingesting.

Does it also happen if you reduce the size of the batch or increase the memory?

Also, have you tried this on recent version?

5

Hi @DudaNogueira , @Dharanish ,
I’ve also seen the same issue today.
Running weaviate: 1.24.11

Restarting the weaviate service has resolved this(temporarily I guess).

6

hi @bharath97 !!

In order to mitigate this, some recommendations would be upgrading to latest version (a lot have improved from from 1.24 to 1.28). Also, we need to be aware that while ingesting data with vectors, Weaviate will also index that data, while write it accordingly. For example, Weaviate 1.28 version, that was just released, had some interesting improvements on ASYNC_INDEXING

So allocating more memory so the ingestion can happen smoothly is an option.

Another route here is to enable ASYNC_INDEXING. This will allow Weaviate to “take it’s time” to index everything and will also make the ingestion process quicker, as it will not perform the indexing right away, but asynchronously.

Let me know if that helps!

THanks!

7

Hey @DudaNogueira ,
Thanks for responding to this.
Let me try setting the ASYNC_INDEXING config to true and see if this improved as I see it’s available from 1.22.
If not, I’ll try upgrading to 1.28.

Thanks!

1 Like
8

Hello @DudaNogueira,
I have set the ASYNC_INDEXING to true and also upgraded weaviate to 1.28, however the issue still seems to persist.
Can you help please?
FYI - my client version is: 3.24.1 (Python SDK)

Some logs from weaviate that could be of any help:

cannot load vector into cache due to memory pressure
find and connect neighbors: at level 0: search layer at level 0: calculate distance between candidate and query: not enough memory
9

It is best to use the python version 4+.

Also, do you have any observability on this server?

What is the memory and cpu allocated? have you done something as documented here?

Ps: it is best to open a new thread, so we can answer it from there.

Thanks!