Description
I’m encountering an error when trying to add objects to my Weaviate collection using the VoyageAI vectorizer. Despite verifying that my API key works and the module appears to be available, I’m still getting the following error:
UnexpectedStatusCodeError: Object was not added! Unexpected status code: 500, with response body: {'error': [{'message': 'vectorize target vector content_embeddings: update vector: connection to VoyageAI failed with status: 404 error: Not Found'}]}.
I’ve verified the following:
-
My VoyageAI API key works (confirmed with a successful curl request to VoyageAI API)
-
I’m using the correct header format with both the API key and baseurl
-
The model names I’m using are correct
-
The VoyageAI module appears to be available on my cluster
Server Setup Information
- Weaviate Server Version: 1.29.0
- Deployment Method: Weaviate Cloud Service
- Client Language and Version: Python, weaviate-client v4 (4.11.1)
- Multitenancy?: Yes
Code Example
Here’s how I’m initializing my client:
self.client = weaviate.connect_to_weaviate_cloud(
cluster_url=self.url,
auth_credentials=Auth.api_key(api_key=self.auth_key),
additional_config=AdditionalConfig(
timeout=Timeout(init=120, query=120, insert=300)
),
headers={"X-VoyageAI-Api-Key": self.voyage_key, "X-VoyageAI-Baseurl": self.voyage_baseurl},
skip_init_checks=False,
)
self.client.integrations.configure(
[
Integrations.voyageai(
api_key=self.voyage_key,
requests_per_minute_embeddings=self.config["Voyage_tpm"],
tokens_per_minute_embeddings=self.config["Voyage_rpm_embeddings"],
),
]
)
And here’s how I’m creating my collection:
self.client.collections.create(
name=self.config["SuperBotCollection"],
multi_tenancy_config=Configure.multi_tenancy(
enabled=True, auto_tenant_creation=False
),
reranker_config=Configure.Reranker.voyageai(
model=self.config["Voyage_Reranker_Model"]
),
properties=[
Property(name="chunk_id", data_type=DataType.TEXT),
Property(name="content", data_type=DataType.TEXT),
Property(
name="file_name",
data_type=DataType.TEXT,
skip_vectorization=True,
),
Property(name="meta_data", data_type=DataType.TEXT),
],
vectorizer_config=[
Configure.NamedVectors.text2vec_voyageai(
model=self.config["Voyage_Text_Embedding_Model"],
vectorize_collection_name=False,
name="content_embeddings",
source_properties=["content"],
vector_index_config=Configure.VectorIndex.hnsw(
quantizer=None,
ef_construction=128,
distance_metric=VectorDistances.COSINE,
max_connections=64,
filter_strategy=VectorFilterStrategy.ACORN,
),
)
],
)
When I try to insert data:
document = self.client.collections.get(class_name)
document_ten = document.with_tenant(str(tenant))
document_ten.data.insert({
"chunk_id": "DUMMY TEXT",
"content": "DUMMY TEXT",
"meta_data": "{'DUMMY TEXT': 'DUMMY TEXT'}",
"file_name": "DUMMY TEXT"
})
Also While try to make use of batch be it fixed , rate_limit or dynamic getting this error:-
ErrorObject(message='WeaviateBatchError(\'Query call with protocol GRPC batch failed with message <AioRpcError of RPC that terminated with:\\n\\tstatus = StatusCode.UNAVAILABLE\\n\\tdetails = "Received http2 header with status: 502"\\n\\tdebug_error_string = "UNKNOWN:Error received from peer {created_time:"2025-03-17T13:09:03.914196782+05:30", grpc_status:14, grpc_message:"Received http2 header with status: 502"}"\\n>.\')'
Even After retry mechanism it doesn’t help:-
ERROR:weaviate-client:{'message': 'Failed to send all objects in a batch of 10', 'error': 'WeaviateBatchError(\'Query call with protocol GRPC batch failed with message <AioRpcError of RPC that terminated with:\\n\\tstatus = StatusCode.UNAVAILABLE\\n\\tdetails = "Received http2 header with status: 502"\\n\\tdebug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Received http2 header with status: 502", grpc_status:14, created_time:"2025-03-17T13:37:05.529298246+05:30"}"\\n>.\')'}
ERROR:weaviate-client:{'message': 'Failed to send 10 objects in a batch of 10. Please inspect client.batch.failed_objects or collection.batch.failed_objects for the failed objects.'}
2025-03-17 08:07:05,543944 l=WARNING ip=Unkknown m=Unkknown url=Unkknown msg=Attempt 1: Still failed to insert 10 objects for tenant: 679cc320af0f91e204fdd470
WARNING:rq.worker:Attempt 1: Still failed to insert 10 objects for tenant: 679cc320af0f91e204fdd470
ERROR:weaviate-client:{'message': 'Failed to send all objects in a batch of 10', 'error': 'WeaviateBatchError(\'Query call with protocol GRPC batch failed with message <AioRpcError of RPC that terminated with:\\n\\tstatus = StatusCode.UNAVAILABLE\\n\\tdetails = "Received http2 header with status: 502"\\n\\tdebug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Received http2 header with status: 502", grpc_status:14, created_time:"2025-03-17T13:37:36.658112152+05:30"}"\\n>.\')'}
ERROR:weaviate-client:{'message': 'Failed to send 10 objects in a batch of 10. Please inspect client.batch.failed_objects or collection.batch.failed_objects for the failed objects.'}
2025-03-17 08:07:36,670347 l=WARNING ip=Unkknown m=Unkknown url=Unkknown msg=Attempt 2: Still failed to insert 10 objects for tenant: 679cc320af0f91e204fdd470
WARNING:rq.worker:Attempt 2: Still failed to insert 10 objects for tenant: 679cc320af0f91e204fdd470
ERROR:weaviate-client:{'message': 'Failed to send all objects in a batch of 10', 'error': 'WeaviateBatchError(\'Query call with protocol GRPC batch failed with message <AioRpcError of RPC that terminated with:\\n\\tstatus = StatusCode.UNAVAILABLE\\n\\tdetails = "Received http2 header with status: 502"\\n\\tdebug_error_string = "UNKNOWN:Error received from peer {created_time:"2025-03-17T13:38:07.774087886+05:30", grpc_status:14, grpc_message:"Received http2 header with status: 502"}"\\n>.\')'}
ERROR:weaviate-client:{'message': 'Failed to send 10 objects in a batch of 10. Please inspect client.batch.failed_objects or collection.batch.failed_objects for the failed objects.'}
2025-03-17 08:08:07,785726 l=WARNING ip=Unkknown m=Unkknown url=Unkknown msg=Attempt 3: Still failed to insert 10 objects for tenant: 679cc320af0f91e204fdd470
WARNING:rq.worker:Attempt 3: Still failed to insert 10 objects for tenant: 679cc320af0f91e204fdd470
2025-03-17 08:08:07,786009 l=ERROR ip=Unkknown m=Unkknown url=Unkknown msg=Failed to insert 10 objects after 3 attempts for tenant: 679cc320af0f91e204fdd470
ERROR:rq.worker:Failed to insert 10 objects after 3 attempts for tenant: 679cc320af0f91e204fdd470
The strange part is my token and API counts on voyage ai are increasing with every request.
