Weaviate Timeout After No Calls for Period of Time - Semantic Search Hosted as API

Support
1

Description

Hey! We’re serving a semantic search built using Weaviate 1.24 and Haystack 1.25 through a Fast API endpoint. They’re both hosted in separate containers. The Weaviate Container is always running. We’re finding that after the endpoint hasn’t been called for a while and then gets called, there’s a timeout (tried increasing the timeout) when Haystack is trying to make calls to the DB. The calls then work and the same issue happens again after a few mins of not being called. Has anyone else encountered an issue like this? We’re instantiating the WeaviateDocumentStore and EmbeddingRetriever when the app is run to avoid latency on each call. I can’t see any obvious configuration to set to avoid this from happening on the Weaviate DB. Is there a recommended setup for Weaviate for serving these types of applications?

Server Setup Information

  • Weaviate Server Version: 1.24
  • Deployment Method: Docker hosted on Azure Container App (k8s)
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: Python, v3 of Python client

Any additional Information

2

hi @patrick_brainpool !! Welcome to our community :hugs:

Can you isolate this issue to Weaviate alone, meaning, reproducing this query directly into Weaviate?

With all those layers on top of it (FastApi + Haystack), this can easily be something before Weaviate, as we don’t see this kind of issue in the wild while using Weaviate directly.

Thanks!

3

Hi @DudaNogueira , thanks for the welcome and replying to the query!

I’ve replicated the issue by using the same Weaviate container and running a separate container calling the Weaviate DB using v3 of the Python client. This is the script calling the Weaviate DB:

import weaviate
import time

client = weaviate.Client(url="<Weaviate URL>")

def do_dummy_query():
    return client.query.get("<class>", properties="<id>").do()


print(f"Verify call after client setup {do_dummy_query()}")

time.sleep(10)

print(f"Verify call after 10 seconds {do_dummy_query()}")

time.sleep(60)
print(f"Verify call after 60 seconds {do_dummy_query()}")

time.sleep(300)
print(f"Verify call after 300 seconds {do_dummy_query()}")

The first 3 queries work as expected, but after sleeping after 300 seconds we experience the timeout issue. Here is the stacktrace:

/usr/local/lib/python3.8/site-packages/weaviate/warnings.py:121: DeprecationWarning: Dep005: You are using weaviate-client version 3.26.2. The latest version is 4.5.4.
Verify call after client setup {'data': {'Get': <expected data> }}
Please consider upgrading to the latest version. See https://weaviate.io/developers/weaviate/client-libraries/python for details.
warnings.warn(
Verify call after 10 seconds {'data': {'Get': <expected data>}}
Verify call after 60 seconds {'data': <expected data> }
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 537, in _make_request
response = conn.getresponse()
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 466, in getresponse
httplib_response = super().getresponse()
File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 847, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 470, in increment
raise reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.8/site-packages/urllib3/util/util.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 539, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='<Weaviate URL>', port=80): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./scripts/weaviate_client_timeout_debug.py", line 20, in <module>
print(f"Verify call after 300 seconds {do_dummy_query()}")
File "./scripts/weaviate_client_timeout_debug.py", line 7, in do_dummy_query
return client.query.get("<class>", properties="<id property>").do()
File "/usr/local/lib/python3.8/site-packages/weaviate/gql/get.py", line 1905, in do
return super().do()
File "/usr/local/lib/python3.8/site-packages/weaviate/gql/filter.py", line 120, in do
response = self._connection.post(path="/graphql", weaviate_object={"query": query})
File "/usr/local/lib/python3.8/site-packages/weaviate/connect/connection.py", line 462, in post
return self._session.post(
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host=<Weaviate URL>, port=80): Read timed out. (read timeout=60)

Best wishes,

Patrick

4

Hi,

We believe the issue was related to the internal networking of the tool (Azure Container App) we’re using for deployments rather than the Weaviate client itself.

In case anyone else encounters this issue when using Weaviate on Azure Container App, using the name of the Container App rather than the internal application URL fixes the timeout issue after the DB has not been called for a while.

Best wishes,

Patrick

1 Like
5

hi @patrick_brainpool !

Ohhhh, that’s a great information to have.

Thanks a lot for sharing! We really appreciate it.

This kind of issue is really hard to tackle.

Kudos!