Description
When inserting a large number of objects(> 1M total vectors) with a batch insert I’ve been intermittently getting
the error.
{‘message’: ‘Failed to send x objects in a batch of x. Please inspect client.batch.failed_objects or collection.batch.failed_objects for the failed objects.’}
Insertion Code:
with self.collection.batch.fixed_size(batch_size=300)as batch:
for i, data_row in enumerate(property_rows):
batch.add_object(
properties=data_row,
vector={
“title_vector”: title_vectors[i],
“body_vector”: body_vectors[i],
“keywords_vector”: keyword_vectors[i],
},
uuid=generate_uuid5(data_row[‘ext_id’])
)
Connection Code:
self.client = weaviate.connect_to_wcs(
cluster_url=URL,
auth_credentials=weaviate.auth.AuthApiKey(APIKEY),
additional_config=AdditionalConfig(
connection=ConnectionConfig(
session_pool_connections=30,
session_pool_maxsize=200,
session_pool_max_retries=3,
),
timeout=Timeout(init=1440, query=1440, insert=1440) # Values in seconds
)
)
Errors:
WeaviateBatchError(‘Query call with protocol GRPC batch failed with message <AioRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = “Received http2 header with status: 502”\n\tdebug_error_string = “UNKNOWN:Error received from peer {created_time:“2024-08-29T08:31:03.197046647+00:00”, grpc_status:14, grpc_message:“Received http2 header with status: 502”}”\n>.’)
WeaviateBatchError(‘Query call with protocol GRPC batch failed with message <AioRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = “recvmsg:Connection reset by peer”\n\tdebug_error_string = “UNKNOWN:Error received from peer {created_time:“2024-08-28T23:03:28.550055837+00:00”, grpc_status:14, grpc_message:“recvmsg:Connection reset by peer”}”\n>.’)
<AioRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = “sendmsg: Broken pipe (32)”\n\tdebug_error_string = “UNKNOWN:Error received from peer {created_time:“2024-08-28T18:23:59.686140478+00:00”, grpc_status:14, grpc_message:“sendmsg: Broken pipe (32)”}”\n>
Server Setup Information
- Weaviate Server Version: 1.25.10
- Deployment Method: Weaviate Serverless Cloud
- Multi Node? Number of Running Nodes:
- Client Language and Version: Python v4 Client
- Multitenancy?: No
Any Additional Information
No vectorizer is being used, objects are being sent with preloaded embeddings. There are multiple-named vectors in this collection.
Asynchronous Indexing is Enabled