[Question] WeaviateBatchError('Query call with protocol GRPC batch failed with message Deadline Exceeded') while writing store

Support
,
1

Summary: not able to write doc store

Context in short:

  • 127.0.0.1:8080/v1/nodes shows health status
  • grpc_startup log shows 50051 is listening
  • 127.0.0.1:8080/v1/meta shows all transformers r loaded successfully
  • much more longer Timeout is given in weaviate constructor
  • client lib: weaviate-haystack
  • weaviate python lib version: v4

Error: ‘WeaviateBatchError(‘Query call with protocol GRPC batch failed with message Deadline Exceeded.’)’

from haystack import Document
from haystack_integrations.document_stores.weaviate import WeaviateDocumentStore
from weaviate.config import AdditionalConfig, ConnectionConfig, Timeout
from weaviate.collections.classes import grpc

document_store = WeaviateDocumentStore(
  url="http://localhost:8080", 
  additional_config=AdditionalConfig(
    grpc_secure=False,
    connection=ConnectionConfig(
      session_pool_connections=30,
      session_pool_maxsize=200,
      session_pool_max_retries=3,
    ),
    timeout=Timeout(query=500, insert=180),
  )
)
document_store.write_documents([
    Document(content="Allo"),
    Document(content="World")
])

print(document_store.count_documents())
2

I tried to insert data using weaviate its own lib and methods. worked. proving that it’s a 3rd integration issue rather than weaviate itself.

3

hi @eli ! Welcome to our community! :hugs:

Can you paste here the entire error stack?

This may mean that the payload you sent is too big to be handled in a single request. It is usually a symptom of a batch size too big, and the GRPC will complain exactly on that.

Assuming there is no “timeout” on that error message.

I tried this code, and it worked as expected, so I am assuming this error happened with other dataset.

I see on Haystack code they seems to be using of the dynamic batch sizing:

that, according to our docs:

The batch size and the number of concurrent requests are dynamically adjusted on-the-fly during import, depending on the server load.

So this may not be it :thinking:

Let me know if this helps, or if you can create a reproducible notebook and/or have more relevant logs.

Thanks!

4

Hi @DudaNogueira , appreciate your detailed response and trying.

  • Data set:
    I can confirm that the dataset are as simple as u can see
document_store.write_documents([
    Document(content="Allo"),
    Document(content="World")
])

  • Error log:
    File “/app/tests/weaviate_docstore.py”, line 24, in
    document_store.write_documents([
    File “/env_py_12/lib/python3.12/site-packages/haystack_integrations/document_stores/weaviate/document_store.py”, line 444, in write_documents
    return self._batch_write(documents)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File “/env_py_12/lib/python3.12/site-packages/haystack_integrations/document_stores/weaviate/document_store.py”, line 394, in _batch_write
    raise DocumentStoreError(msg)
    haystack.document_stores.errors.errors.DocumentStoreError: Failed to write object with id ‘f3b66e62b3b5d3e0942135c5cce25f713cf8415328c26ba992ba8f37d1b750e4’. Error: ‘WeaviateBatchError(‘Query call with protocol GRPC batch failed with message Deadline Exceeded.’)’
    Failed to write object with id ‘ec4e7974f6a1f33960b3d53a147158c0aeb554488ea808a74037aa5dcfd89a21’. Error: ‘WeaviateBatchError(‘Query call with protocol GRPC batch failed with message Deadline Exceeded.’)’
    sys:1: ResourceWarning: unclosed <socket.socket fd=7, family=30, type=1, proto=6, laddr=(‘::1’, 60236, 0, 0), raddr=(‘::1’, 8080, 0, 0)>

  • reproducible notebook:
    on it, will update once make it ready

thx again

1 Like