Weaviate python client v4.12.0 keep throwing GRPC Deadline exceeded lately

Description

I make the client like this:

weaviate_client = weaviate.connect_to_weaviate_cloud(
            cluster_url=WEAVIATE_URL,
            auth_credentials=Auth.api_key(WEAVIATE_API_KEY),
            skip_init_checks=True
        )

Deploy my python backend on azure container app but it keeps throwing in the console:
stderr F debug_error_string = “UNKNOWN:Error received from peer {created_time:“2025-04-08T13:56:17.76001071+00:00”, grpc_status:4, grpc_message:“Deadline Exceeded”}”

Server Setup Information

Serverless cluster hosted by Weaviate.

Any additional Information

2025-04-08T13:55:36.0551084Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_weaviate/vectorstores.py", line 278, in _perform_search
2025-04-08T13:55:36.0551115Z stderr F     raise ValueError(f"Error during query: {e}")
2025-04-08T13:55:36.0551138Z stderr F ValueError: Error during query: Query call with protocol GRPC search failed with message <AioRpcError of RPC that terminated with:
2025-04-08T13:55:36.0551160Z stderr F   status = StatusCode.DEADLINE_EXCEEDED
2025-04-08T13:55:36.0551181Z stderr F   details = "Deadline Exceeded"
2025-04-08T13:55:36.0551202Z stderr F   debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2025-04-08T13:55:35.991674066+00:00"}"
2025-04-08T13:55:36.0551391Z stderr F >.
2025-04-08T13:55:47.2393238Z stdout F INFO:     100.100.0.199:35286 - "OPTIONS /chat/stream_log HTTP/1.1" 200 OK
2025-04-08T13:55:47.3962134Z stdout F INFO:     100.100.0.199:35286 - "POST /chat/stream_log HTTP/1.1" 200 OK
2025-04-08T13:55:47.7571409Z stderr F INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
2025-04-08T13:56:17.7885657Z stderr F ERROR:    Exception in ASGI application
2025-04-08T13:56:17.7886664Z stderr F Traceback (most recent call last):
2025-04-08T13:56:17.7886871Z stderr F   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
2025-04-08T13:56:17.7886904Z stderr F     result = await app(  # type: ignore[func-returns-value]
2025-04-08T13:56:17.7904535Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/tracers/log_stream.py", line 279, in tap_output_aiter
2025-04-08T13:56:17.7904556Z stderr F     async for chunk in output:
2025-04-08T13:56:17.7904795Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 4915, in _atransform
2025-04-08T13:56:17.7904839Z stderr F     async for ichunk in input:
2025-04-08T13:56:17.7904870Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/utils/aiter.py", line 128, in tee_peer
2025-04-08T13:56:17.7904891Z stderr F     item = await iterator.__anext__()
2025-04-08T13:56:17.7904912Z stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7904932Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/utils/aiter.py", line 128, in tee_peer
2025-04-08T13:56:17.7904952Z stderr F     item = await iterator.__anext__()
2025-04-08T13:56:17.7905019Z stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7905042Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/utils/aiter.py", line 128, in tee_peer
2025-04-08T13:56:17.7905062Z stderr F     item = await iterator.__anext__()
2025-04-08T13:56:17.7905082Z stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7905102Z stderr F   [Previous line repeated 2 more times]
2025-04-08T13:56:17.7905122Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/passthrough.py", line 650, in atransform
2025-04-08T13:56:17.7905142Z stderr F     async for chunk in self._atransform_stream_with_config(
2025-04-08T13:56:17.7905163Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2319, in _atransform_stream_with_config
2025-04-08T13:56:17.7905444Z stderr F     chunk: Output = await asyncio.create_task(  # type: ignore[call-arg]
2025-04-08T13:56:17.7905548Z stderr F                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7905579Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/tracers/log_stream.py", line 279, in tap_output_aiter
2025-04-08T13:56:17.7905601Z stderr F     async for chunk in output:
2025-04-08T13:56:17.7905621Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/passthrough.py", line 639, in _atransform
2025-04-08T13:56:17.7905641Z stderr F     yield await first_map_chunk_task
2025-04-08T13:56:17.7905662Z stderr F           ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7905683Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/utils/aiter.py", line 78, in anext_impl
2025-04-08T13:56:17.7905703Z stderr F     return await __anext__(iterator)
2025-04-08T13:56:17.7905723Z stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7905744Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 3967, in atransform
2025-04-08T13:56:17.7905764Z stderr F     async for chunk in self._atransform_stream_with_config(
2025-04-08T13:56:17.7905790Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2319, in _atransform_stream_with_config
2025-04-08T13:56:17.7905812Z stderr F     chunk: Output = await asyncio.create_task(  # type: ignore[call-arg]
2025-04-08T13:56:17.7906097Z stderr F                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7906228Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/tracers/log_stream.py", line 279, in tap_output_aiter
2025-04-08T13:56:17.7906247Z stderr F     async for chunk in output:
2025-04-08T13:56:17.7906260Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 3953, in _atransform
2025-04-08T13:56:17.7906272Z stderr F     chunk = AddableDict({step_name: task.result()})
2025-04-08T13:56:17.7907015Z stderr F                                     ^^^^^^^^^^^^^
2025-04-08T13:56:17.7907250Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 3936, in get_next_chunk
2025-04-08T13:56:17.7907445Z stderr F     return await py_anext(generator)
2025-04-08T13:56:17.7907470Z stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7907602Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 5669, in atransform
2025-04-08T13:56:17.7907622Z stderr F     async for item in self.bound.atransform(
2025-04-08T13:56:17.7907641Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1478, in atransform
2025-04-08T13:56:17.7907662Z stderr F     async for output in self.astream(final, config, **kwargs):
2025-04-08T13:56:17.7908226Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/branch.py", line 454, in astream
2025-04-08T13:56:17.7908295Z stderr F     async for chunk in self.default.astream(
2025-04-08T13:56:17.7908325Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 5630, in astream
2025-04-08T13:56:17.7908347Z stderr F     async for item in self.bound.astream(
2025-04-08T13:56:17.7908367Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 3467, in astream
2025-04-08T13:56:17.7908474Z stderr F     async for chunk in self.atransform(input_aiter(), config, **kwargs):
2025-04-08T13:56:17.7908499Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 3449, in atransform
2025-04-08T13:56:17.7908519Z stderr F     async for chunk in self._atransform_stream_with_config(
2025-04-08T13:56:17.7908540Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2319, in _atransform_stream_with_config
2025-04-08T13:56:17.7908562Z stderr F     chunk: Output = await asyncio.create_task(  # type: ignore[call-arg]
2025-04-08T13:56:17.7908582Z stderr F                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7908602Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/tracers/log_stream.py", line 279, in tap_output_aiter
2025-04-08T13:56:17.7908623Z stderr F     async for chunk in output:
2025-04-08T13:56:17.7908935Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 3416, in _atransform
2025-04-08T13:56:17.7908972Z stderr F     async for output in final_pipeline:
2025-04-08T13:56:17.7909047Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1478, in atransform
2025-04-08T13:56:17.7909071Z stderr F     async for output in self.astream(final, config, **kwargs):
2025-04-08T13:56:17.7909091Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1023, in astream
2025-04-08T13:56:17.7909113Z stderr F     yield await self.ainvoke(input, config, **kwargs)
2025-04-08T13:56:17.7909134Z stderr F           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7909154Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/retrievers.py", line 322, in ainvoke
2025-04-08T13:56:17.7909174Z stderr F     result = await self._aget_relevant_documents(
2025-04-08T13:56:17.7909194Z stderr F              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7909215Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/vectorstores/base.py", line 1101, in _aget_relevant_documents
2025-04-08T13:56:17.7909236Z stderr F     docs = await self.vectorstore.asimilarity_search(query, **_kwargs)
2025-04-08T13:56:17.7909303Z stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7910518Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/vectorstores/base.py", line 647, in asimilarity_search
2025-04-08T13:56:17.7910624Z stderr F     return await run_in_executor(None, self.similarity_search, query, k=k, **kwargs)
2025-04-08T13:56:17.7910674Z stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7910690Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/config.py", line 616, in run_in_executor
2025-04-08T13:56:17.7910703Z stderr F     return await asyncio.get_running_loop().run_in_executor(
2025-04-08T13:56:17.7910715Z stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7910810Z stderr F   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
2025-04-08T13:56:17.7910825Z stderr F     result = self.fn(*self.args, **self.kwargs)
2025-04-08T13:56:17.7910838Z stderr F              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7910850Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/config.py", line 607, in wrapper
2025-04-08T13:56:17.7910862Z stderr F     return func(*args, **kwargs)
2025-04-08T13:56:17.7910874Z stderr F            ^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7910886Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_weaviate/vectorstores.py", line 318, in similarity_search
2025-04-08T13:56:17.7910899Z stderr F     result = self._perform_search(query, k, **kwargs)
2025-04-08T13:56:17.7910911Z stderr F              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-08T13:56:17.7911213Z stderr F   File "/usr/local/lib/python3.11/site-packages/langchain_weaviate/vectorstores.py", line 278, in _perform_search
2025-04-08T13:56:17.7911299Z stderr F     raise ValueError(f"Error during query: {e}")
2025-04-08T13:56:17.7911328Z stderr F ValueError: Error during query: Query call with protocol GRPC search failed with message <AioRpcError of RPC that terminated with:
2025-04-08T13:56:17.7911351Z stderr F   status = StatusCode.DEADLINE_EXCEEDED
2025-04-08T13:56:17.7911372Z stderr F   details = "Deadline Exceeded"
2025-04-08T13:56:17.7911393Z stderr F   debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-04-08T13:56:17.76001071+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"

hi @bamibestelt !!

Welcome to our community :hugs:

For any issues with a cluster hosted in our cloud, the best place for support is opening a support ticket from:

With that we can check the cluster logs and identify.

This can happen if the cluster is overwhelmed or while it is scaling (and HA is not enabled).

Thanks!

Ok I have sent an email to the customer support.

1 Like