Bm25_search - DoBlockMaxWand: search timed out, returning partial results

Support
1

Description

We are sometimes experiencing timeouts while executing a BM25 search in Weaviate: “DoBlockMaxWand: search timed out, returning partial results”. I’m unsure of the reason this is happening. I found the corresponding code in Weaviate, but I’m still not sure what that “timeout” actually means. Does anyone know what could cause this error? I could not find anyone else having this problem. We currently have a lot of collections, almost 1000, which is something we want to try to improve. Could this be the cause of the error? But the search performance has been good so far.

Server Setup Information

  • Weaviate Server Version: 1.32.1
  • Deployment Method: Kubernetes
  • Multi Node? Number of Running Nodes: 2
  • Client Language and Version: Python, weaviate-client 4.15.4
  • Multitenancy?: No

Any additional Information

Weaviate log:

2025-07-31 09:55:04.543	
{"action":"bm25_search","build_git_commit":"55e01d0","build_go_version":"go1.24.5","build_image_tag":"v1.32.1","build_wv_version":"1.32.1","class":"INDEX_6282","filterCardinality":-1,"firstNonExhausted":0,"iterations":295900000,"lenResults":78,"level":"warning","limit":12,"msg":"DoBlockMaxWand: search timed out, returning partial results","pivotID":80,"pivotPoint":13,"segment":"/var/lib/weaviate/index_6282/awSl82KpBrrR/lsm/property_text_searchable/segment-1753351102207519787.db","shard":"awSl82KpBrrR","terms":"b:74:41, 1:76:24, den:78:34, der:78:44, des:79:27, die:79:48, 3:79:22, oder:83:28, ist:83:30, wir:80:32, ihr:80:4, nach:80:24, bis:80:16, 5:80:16, 17:81:2, sie:83:28, das:83:12, für:83:30, haben:83:12, bei:84:19, von:84:29, dies:84:3, zu:84:24, 20:85:1, und:86:31, eine:86:19, versicherungen:86:1, dürfen:-1:1, 2025:-1:5, beitrag:-1:11, kann:-1:6, diebstahl:-1:5, d:-1:2, diese:-1:8, versicherung:-1:5, e:-1:6, z:-1:2, nur:-1:10, ohne:-1:7, de:-1:1, deckung:-1:2, wäre:-1:1, so:-1:6, h:-1:1, bitte:-1:1, erhalten:-1:3, mindestens:-1:2, dann:-1:1, wie:-1:1, wird:-1:8, versichert:-1:4, einen:-1:10, neu:-1:2, erfahren:-1:1, versicherer:-1:3, muss:-1:5, vom:-1:2, ob:-1:2, um:-1:2, allgemeine:-1:1, hierzu:-1:1, ende:-1:2, 6:-1:11, versicherungsschutz:-1:8, m:-1:1, 80:-1:1, weg:-1:1, siehe:-1:1, am:-1:3, deutschland:-1:1, j:-1:2, welche:-1:5, damit:-1:1, kosten:-1:8, 14:-1:2, auch:-1:13, mit:-1:9, nicht:-1:21, ","time":"2025-07-31T07:55:04Z","upperBound":16.032675}

Python service log:

2025-07-31 09:55:04.544	2025-07-31 07:55:04.544 | ERROR    | project.main:automate:768 - Unexpected error. Traceback details: Traceback (most recent call last):
2025-07-31 09:55:04.544	  File "/project/.venv/lib/python3.10/site-packages/weaviate/connect/v4.py", line 985, in grpc_search
2025-07-31 09:55:04.544	    res = _Retry(4).with_exponential_backoff(
2025-07-31 09:55:04.544	  File "/project/.venv/lib/python3.10/site-packages/weaviate/retry.py", line 54, in with_exponential_backoff
2025-07-31 09:55:04.544	    raise e
2025-07-31 09:55:04.544	  File "/project/.venv/lib/python3.10/site-packages/weaviate/retry.py", line 50, in with_exponential_backoff
2025-07-31 09:55:04.544	    return f(*args, **kwargs)
2025-07-31 09:55:04.544	  File "/project/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 68, in __call__
2025-07-31 09:55:04.544	    return self._interceptor.intercept_unary(
2025-07-31 09:55:04.544	  File "/project/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 190, in intercept_unary
2025-07-31 09:55:04.544	    return self._intercept(request, metadata, client_info, invoker)
2025-07-31 09:55:04.544	  File "/project/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 171, in _intercept
2025-07-31 09:55:04.544	    raise exc
2025-07-31 09:55:04.544	  File "/project/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 157, in _intercept
2025-07-31 09:55:04.544	    result = invoker(request, metadata)
2025-07-31 09:55:04.544	  File "/project/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 58, in invoker
2025-07-31 09:55:04.544	    return self._base_callable(
2025-07-31 09:55:04.544	  File "/project/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 1181, in __call__
2025-07-31 09:55:04.544	    return _end_unary_response_blocking(state, call, False, None)
2025-07-31 09:55:04.544	  File "/project/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
2025-07-31 09:55:04.544	    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
2025-07-31 09:55:04.544	grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
2025-07-31 09:55:04.544		status = StatusCode.DEADLINE_EXCEEDED
2025-07-31 09:55:04.544		details = "Received RST_STREAM with error code 8"
2025-07-31 09:55:04.544		debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Received RST_STREAM with error code 8", grpc_status:4}"
2025-07-31 09:55:04.544	>
2025-07-31 09:55:04.544	
2025-07-31 09:55:04.544	During handling of the above exception, another exception occurred:
2025-07-31 09:55:04.544	
2025-07-31 09:55:04.544	Traceback (most recent call last):
...
2025-07-31 09:55:04.545	    response = collection.query.bm25(
2025-07-31 09:55:04.545	  File "/project/.venv/lib/python3.10/site-packages/weaviate/collections/queries/bm25/query/executor.py", line 375, in bm25
2025-07-31 09:55:04.545	    return executor.execute(
2025-07-31 09:55:04.545	  File "/project/.venv/lib/python3.10/site-packages/weaviate/connect/executor.py", line 99, in execute
2025-07-31 09:55:04.545	    return cast(T, exception_callback(e))
2025-07-31 09:55:04.545	  File "/project/.venv/lib/python3.10/site-packages/weaviate/connect/executor.py", line 38, in raise_exception
2025-07-31 09:55:04.545	    raise e
2025-07-31 09:55:04.545	  File "/project/.venv/lib/python3.10/site-packages/weaviate/connect/executor.py", line 80, in execute
2025-07-31 09:55:04.545	    call = method(*args, **kwargs)
2025-07-31 09:55:04.545	  File "/project/.venv/lib/python3.10/site-packages/weaviate/connect/v4.py", line 998, in grpc_search
2025-07-31 09:55:04.545	    raise WeaviateQueryError(str(error.details()), "GRPC search")  # pyright: ignore
2025-07-31 09:55:04.545	weaviate.exceptions.WeaviateQueryError: Query call with protocol GRPC search failed with message Received RST_STREAM with error code 8.
2

Hey Torben,

André Mourão from Weaviate here, working on keyword search.

Sorry you are having this problem.

This is got nothing to do with the number of collections you have, it seems like you found a bug in Weaviate’s keyword search algorithm.

The error means that the keyword search algorithm couldn’t “finish” and stopped when the timeout was reached.

This is an hard issue to reproduce, as in my experience, it is tied to the specific segment and query. Do you have access to the file listed in the logs?

index_6282/awSl82KpBrrR/lsm/property_text_searchable/segment-1753351102207519787.db

If so, it would really help if you could send it to me directly on Weaviate’s community Slack. If not, I’ll keep testing to try and reproduce the test.

Best,

André Mourão

3

Thank you @amourao for your quick response!

Sadly, I’m not allowed to give you the segment file and the query. The segment file contains a document of a customer and the query is a customer email. I have not been able to reproduce the problem with a different collection / different data or an anonymized email.

I noticed the same query does not always fail. Yesterday it was failing most of the time. Right now it only failed in 1 out of 5 tries.

I think the error might also be related to load on Weaviate. We are quickly executing BM25 search on almost 200 collections. But if it fails it always fails on the same collection.

4

Hey Torben,

Sadly, I’m not allowed to give you the segment file and the query.

No problem, we’ll figure it out :slight_smile:

I noticed the same query does not always fail. Yesterday it was failing most of the time. Right now it only failed in 1 out of 5 tries.

Interesting, are you also ingesting, updating or deleting data for that specific user/collection when doing queries?

Is the error always on the same segment?

I think the error might also be related to load on Weaviate. We are quickly executing BM25 search on almost 200 collections. But if it fails it always fails on the same collection.

That’s helpful, have you checked whether doing only the affected queries makes them fail less often?

I’m trying to find similar conditions to reproduce the error locally.

Until then, if you want, you can try to reduce the query timeout to make the query fail faster (following example reduces it to 5 seconds):

client = weaviate.connect_to_local(additional_config=AdditionalConfig(timeout=Timeout(init=30, query=5, insert=120)  # Values in seconds))

Best,

André

5

No, the collection is not changed. It currently only contains a single document and does not change. This is the reason for the high number of collections we have right now. But as I mentioned, we want to change that.

Yes, even though the search is in almost 200 collections it’s always the same collection and segment file that causes the problem.

I tried to reproduce the problem with only 2 collections, 1 of them containing the same document as the problematic collection. But I did not get any errors that way. (I did not do hundreds of searches, though.) But I had to do the ingestion for the document again for that, so I cannot 100% ensure that the segment files are the same.

Thanks again for your help!

6

Hi Toben,

sorry for the delayed response.

I was able to reproduce an issue that looks similar to the one you are experiencing. I’m working on a fix and will let you know when it is released so we can see if it fixes your issue as well.

Thanks a lot for your report and very helpful answers.

2 Likes
7

Thanks a lot. I’m looking forward to testing your fix.