Description
We are sometimes experiencing timeouts while executing a BM25 search in Weaviate: “DoBlockMaxWand: search timed out, returning partial results”. I’m unsure of the reason this is happening. I found the corresponding code in Weaviate, but I’m still not sure what that “timeout” actually means. Does anyone know what could cause this error? I could not find anyone else having this problem. We currently have a lot of collections, almost 1000, which is something we want to try to improve. Could this be the cause of the error? But the search performance has been good so far.
Server Setup Information
- Weaviate Server Version: 1.32.1
- Deployment Method: Kubernetes
- Multi Node? Number of Running Nodes: 2
- Client Language and Version: Python, weaviate-client 4.15.4
- Multitenancy?: No
Any additional Information
Weaviate log:
2025-07-31 09:55:04.543
{"action":"bm25_search","build_git_commit":"55e01d0","build_go_version":"go1.24.5","build_image_tag":"v1.32.1","build_wv_version":"1.32.1","class":"INDEX_6282","filterCardinality":-1,"firstNonExhausted":0,"iterations":295900000,"lenResults":78,"level":"warning","limit":12,"msg":"DoBlockMaxWand: search timed out, returning partial results","pivotID":80,"pivotPoint":13,"segment":"/var/lib/weaviate/index_6282/awSl82KpBrrR/lsm/property_text_searchable/segment-1753351102207519787.db","shard":"awSl82KpBrrR","terms":"b:74:41, 1:76:24, den:78:34, der:78:44, des:79:27, die:79:48, 3:79:22, oder:83:28, ist:83:30, wir:80:32, ihr:80:4, nach:80:24, bis:80:16, 5:80:16, 17:81:2, sie:83:28, das:83:12, für:83:30, haben:83:12, bei:84:19, von:84:29, dies:84:3, zu:84:24, 20:85:1, und:86:31, eine:86:19, versicherungen:86:1, dürfen:-1:1, 2025:-1:5, beitrag:-1:11, kann:-1:6, diebstahl:-1:5, d:-1:2, diese:-1:8, versicherung:-1:5, e:-1:6, z:-1:2, nur:-1:10, ohne:-1:7, de:-1:1, deckung:-1:2, wäre:-1:1, so:-1:6, h:-1:1, bitte:-1:1, erhalten:-1:3, mindestens:-1:2, dann:-1:1, wie:-1:1, wird:-1:8, versichert:-1:4, einen:-1:10, neu:-1:2, erfahren:-1:1, versicherer:-1:3, muss:-1:5, vom:-1:2, ob:-1:2, um:-1:2, allgemeine:-1:1, hierzu:-1:1, ende:-1:2, 6:-1:11, versicherungsschutz:-1:8, m:-1:1, 80:-1:1, weg:-1:1, siehe:-1:1, am:-1:3, deutschland:-1:1, j:-1:2, welche:-1:5, damit:-1:1, kosten:-1:8, 14:-1:2, auch:-1:13, mit:-1:9, nicht:-1:21, ","time":"2025-07-31T07:55:04Z","upperBound":16.032675}
Python service log:
2025-07-31 09:55:04.544 2025-07-31 07:55:04.544 | ERROR | project.main:automate:768 - Unexpected error. Traceback details: Traceback (most recent call last):
2025-07-31 09:55:04.544 File "/project/.venv/lib/python3.10/site-packages/weaviate/connect/v4.py", line 985, in grpc_search
2025-07-31 09:55:04.544 res = _Retry(4).with_exponential_backoff(
2025-07-31 09:55:04.544 File "/project/.venv/lib/python3.10/site-packages/weaviate/retry.py", line 54, in with_exponential_backoff
2025-07-31 09:55:04.544 raise e
2025-07-31 09:55:04.544 File "/project/.venv/lib/python3.10/site-packages/weaviate/retry.py", line 50, in with_exponential_backoff
2025-07-31 09:55:04.544 return f(*args, **kwargs)
2025-07-31 09:55:04.544 File "/project/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 68, in __call__
2025-07-31 09:55:04.544 return self._interceptor.intercept_unary(
2025-07-31 09:55:04.544 File "/project/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 190, in intercept_unary
2025-07-31 09:55:04.544 return self._intercept(request, metadata, client_info, invoker)
2025-07-31 09:55:04.544 File "/project/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 171, in _intercept
2025-07-31 09:55:04.544 raise exc
2025-07-31 09:55:04.544 File "/project/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 157, in _intercept
2025-07-31 09:55:04.544 result = invoker(request, metadata)
2025-07-31 09:55:04.544 File "/project/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 58, in invoker
2025-07-31 09:55:04.544 return self._base_callable(
2025-07-31 09:55:04.544 File "/project/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 1181, in __call__
2025-07-31 09:55:04.544 return _end_unary_response_blocking(state, call, False, None)
2025-07-31 09:55:04.544 File "/project/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
2025-07-31 09:55:04.544 raise _InactiveRpcError(state) # pytype: disable=not-instantiable
2025-07-31 09:55:04.544 grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
2025-07-31 09:55:04.544 status = StatusCode.DEADLINE_EXCEEDED
2025-07-31 09:55:04.544 details = "Received RST_STREAM with error code 8"
2025-07-31 09:55:04.544 debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Received RST_STREAM with error code 8", grpc_status:4}"
2025-07-31 09:55:04.544 >
2025-07-31 09:55:04.544
2025-07-31 09:55:04.544 During handling of the above exception, another exception occurred:
2025-07-31 09:55:04.544
2025-07-31 09:55:04.544 Traceback (most recent call last):
...
2025-07-31 09:55:04.545 response = collection.query.bm25(
2025-07-31 09:55:04.545 File "/project/.venv/lib/python3.10/site-packages/weaviate/collections/queries/bm25/query/executor.py", line 375, in bm25
2025-07-31 09:55:04.545 return executor.execute(
2025-07-31 09:55:04.545 File "/project/.venv/lib/python3.10/site-packages/weaviate/connect/executor.py", line 99, in execute
2025-07-31 09:55:04.545 return cast(T, exception_callback(e))
2025-07-31 09:55:04.545 File "/project/.venv/lib/python3.10/site-packages/weaviate/connect/executor.py", line 38, in raise_exception
2025-07-31 09:55:04.545 raise e
2025-07-31 09:55:04.545 File "/project/.venv/lib/python3.10/site-packages/weaviate/connect/executor.py", line 80, in execute
2025-07-31 09:55:04.545 call = method(*args, **kwargs)
2025-07-31 09:55:04.545 File "/project/.venv/lib/python3.10/site-packages/weaviate/connect/v4.py", line 998, in grpc_search
2025-07-31 09:55:04.545 raise WeaviateQueryError(str(error.details()), "GRPC search") # pyright: ignore
2025-07-31 09:55:04.545 weaviate.exceptions.WeaviateQueryError: Query call with protocol GRPC search failed with message Received RST_STREAM with error code 8.