Query call with protocol GRPC search failed with message sendmsg: Socket operation on non-socket (88)

Support
1

Description

Hi,
When the weaviate sever is deployed, in the first 15 minutes if we perform a hybrid search, we are getting this error from weaviate:

Traceback (most recent call last): File "/app/venv/lib/python3.10/site-packages/weaviate/collections/grpc/query.py", line 649, in __call res, _ = self._connection.grpc_stub.Search.with_call( File "/app/venv/lib/python3.10/site-packages/grpc/_channel.py", line 1198, in with_call return _end_unary_response_blocking(state, call, True, None) File "/app/venv/lib/python3.10/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking raise _InactiveRpcError(state) # pytype: disable=not-instantiable grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: \tstatus = StatusCode.UNAVAILABLE \tdetails = "sendmsg: Socket operation on non-socket (88)" \tdebug_error_string = "UNKNOWN:
Error
received from peer {created_time:"2024-08-21T10:04:43.533570886+00:00", grpc_status:14, grpc_message:"sendmsg: Socket operation on non-socket (88)"}" >
File "/app/venv/lib/python3.10/site-packages/weaviate/collections/queries/hybrid/query.py", line 105, in hybrid res = self._query.hybrid( File "/app/venv/lib/python3.10/site-packages/weaviate/collections/grpc/query.py", line 246, in hybrid return self.__call(request) File "/app/venv/lib/python3.10/site-packages/weaviate/collections/grpc/query.py", line 658, in __call raise WeaviateQueryError(e.details(), "GRPC search") # pyright: ignore weaviate.exceptions.WeaviateQueryError: Query call with protocol GRPC search failed with message sendmsg: Socket operation on non-socket (88).

This is getting resolved in next 15-20 minutes on its own but what is the reason for this error and how to resolve this?

I am getting error at this line :
collection.query.hybrid(**hybrid_params).objects

This is how the client is created:
weaviate_connection_config = AdditionalConfig(
connection=ConnectionConfig(
session_pool_connections=20,
session_pool_maxsize=20,
),
timeout=(
60,180),
)
client = weaviate.connect_to_custom(
http_host=url,
http_port=8080,
http_secure=False,
grpc_host=url,
grpc_port=50051,
grpc_secure=False,
additional_config=weaviate_connection_config,
)

Server Setup Information

  • Weaviate Server Version: 1.23.7
  • Deployment Method: docker
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version:4.6.5

TIA.

2

hi @swetag !!

Do you see any logs in the server side?

Also, do you have any readings on resource and have you tweaked any of the parameters as stated in Resource Planning?

1.23.7 is fairly “old”. :grimacing:

Is it possible to test this on latest version?

3

Hi @DudaNogueira,

  1. No logs on the server side.
  2. These are the resource related env variables for weaviate.
    LIMIT_RESOURCES: true
    GOMEMLIMIT: 7680MiB
    QUERY_MAXIMUM_RESULTS: 1000000
    DISK_USE_READONLY_PERCENTAGE: 95
  3. I had to use this version because most of the 1.24.x, 1.25.x, 1.26.0 (all of which I tried) are leading me to this issue, but 1.23.7 is not.
    Additionally, I was also getting the same behavior with 1.25.6
4

Do you still have this kind of issue even when allocating more memory from the host?

Is this cluster exposed somehow, for example, using a load balancer, or is the client connecting directly to the server within the same docker network?