Issue with Connecting to Weaviate Cloud Using gevent Worker Class on AWS Ubuntu Instance

Support
, ,
1

Problem Description:

We are currently facing challenges with our Weaviate setup on an AWS Ubuntu instance when using version 4 of Weaviate. While our setup works smoothly with Weaviate v3 under the same conditions, v4 encounters issues when connecting to the Weaviate Cloud.

Here are the details:

  1. Environment:

    • Instance: AWS Ubuntu instance.
    • Gunicorn Workers Tested: sync, gthread, gevent.
    • Weaviate Versions:
      • v3: Works fine under all tested configurations.
      • v4: Works with sync and gthread but fails with gevent.

  2. Error Details:
    When using gevent with Weaviate v4, the connection attempt hangs indefinitely at the following line:

    client = weaviate.connect_to_weaviate_cloud(
    cluster_url=wcd_url,  # Replace with your Weaviate Cloud URL
    auth_credentials=Auth.api_key(wcd_api_key),  # Replace with your Weaviate Cloud key
    headers={'X-OpenAI-Api-key': openai_api_key}  # Replace with your OpenAI API key
)
    • The process eventually times out after 120 seconds, and the worker thread is killed with the following error message:
      [2024-10-09 10:56:37 +0000] [445826] [ERROR] Worker (pid:445830) was sent SIGKILL! Perhaps out of memory?

  3. Replication Steps:

    • Create a file named vec.py with relevant connection code.
    • Run the following command:
      gunicorn --workers 4 --worker-class gevent --bind 0.0.0.0:8000 --timeout 120 vec:app
    • Observe that with Weaviate v4 and gevent, the connection hangs and fails, while v3 works without any issues.

  4. Additional Notes:

    • This issue only arises with the gevent worker class in Gunicorn.
    • Using sync or gthread with v4 works as expected.
    • The problem does not seem related to instance memory, as sufficient resources are allocated, and no OOM conditions are observed.
    • The same gevent configuration works for v3 but fails for v4.