Cannot process batch: not enough memory

Dharanish · March 25, 2024, 8:18am

Hi , configured batch on
client.batch.configure(batch_size=100,dynamic=True,consistency_level=“ALL”,connection_error_retries=3,num_workers=2)
perform batch insertion and getting exception
UnexpectedStatusCodeException: batch response! Unexpected status code: 500, with response body: {‘error’: [{‘message’: ‘batch objects: &fmt.wrapError{msg:“cannot process batch: not enough memory”, err:(*errors.errorString)(0xc00004c020)}’}]}.

Weaviate version :1.23.11
Deployment type: Kubernetes cluster
Python client : 3.24.2
available memory per container: 4GB

DudaNogueira · March 25, 2024, 10:38am

hi @Dharanish !

Can you try using the new python v4 client?

Also, does the server has enough memory considering the amount of objects to be ingested?

And finally, does it happens if you reduce the batch size?

If you try to the python v4 client, I suggest using the dynamic batch size, so Weaviate server can adjust the batch size while communicating with the client.

Thanks!

Dharanish · August 8, 2024, 10:28am

Hi @DudaNogueira , My cluster has enough memory and batch size is 20 and using client 4.5.7, still getting this error

DudaNogueira · August 8, 2024, 2:31pm

This is the origin of this error:

github.com

weaviate/weaviate/blob/6fd24322f15ecf46d40e1e35754b524fe96e61f5/adapters/repos/db/batch.go#L39


      
          	originalIndex []int
          }
          
          func (db *DB) BatchPutObjects(ctx context.Context, objs objects.BatchObjects,
          	repl *additional.ReplicationProperties, schemaVersion uint64,
          ) (objects.BatchObjects, error) {
          	objectByClass := make(map[string]batchQueue)
          	indexByClass := make(map[string]*Index)
          
          	if err := db.memMonitor.CheckAlloc(estimateBatchMemory(objs)); err != nil {
          		db.logger.WithError(err).Errorf("memory pressure: cannot process batch")
          		return nil, fmt.Errorf("cannot process batch: %w", err)
          	}
          
          	for _, item := range objs {
          		if item.Err != nil {
          			// item has a validation error or another reason to ignore
          			continue
          		}
          		queue := objectByClass[item.Object.Class]
          		queue.objects = append(queue.objects, storobj.FromObject(item.Object, item.Object.Vector, item.Object.Vectors))

It may be hitting the limits while ingesting.

Does it also happen if you reduce the size of the batch or increase the memory?

Also, have you tried this on recent version?

Topic		Replies	Views
Payload Too Large Support	1	189	February 23, 2024
Weaviate Cloud Serverless - Batch Insert 502 Server Side errors with v4 client Support	3	14	August 29, 2024
Weaviate Batch Errors during Batch Insertion with v4 client Support bug , developer-experience , wcs , python , documentation	11	472	May 15, 2024
Getting the error "UnexpectedStatusCodeException: batch response! Unexpected status code: 502, with response body: None." Support	3	711	January 2, 2024
Error in Batch Addition Support	14	2174	April 8, 2024

Cannot process batch: not enough memory

Related Topics