WeaviateQueryError: Query call with protocol GRPC search failed with message Socket closed

Support
1

Server

I have weaviate hosted on k8s with grpc enabled as follows:

# The service controls how weaviate gRPC endpoint is exposed to the outside world.
# If you don't want a public load balancer, you can also choose 'ClusterIP' to make
# weaviate gRPC port be only accessible within your cluster.
grpcService:
  # Set this to true in order to deploy Weaviate gRPC service
  enabled: true
  name: weaviate-grpc
  ports:
    - name: grpc
      protocol: TCP
      port: 50052
      # Target port is going to be the same for every port
  type: LoadBalancer
  loadBalancerSourceRanges: []
  # optionally set cluster IP if you want to set a static IP
  clusterIP:
  annotations: {}

NAME                        TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)           AGE
service/weaviate            LoadBalancer   a.b.c.d   e.f.g.h   80:30357/TCP      8d
service/weaviate-grpc       LoadBalancer   i.j.k.l   m.n.o.p  50052:31927/TCP   8d
service/weaviate-headless   ClusterIP      None           <none>          80/TCP            8d

Client

I’m on weaviate-client-4.4b8 python client.

When I try to query weaviate with bm25 or hybrid search using the python v4 client I am getting following error:

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Socket closed"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Socket closed", grpc_status:14, created_time:"2024-01-27T19:46:26.102631-05:00"}"
>

During handling of the above exception, another exception occurred:

WeaviateQueryError                        Traceback (most recent call last)
.
.
.
  561     return res
    563 except grpc.RpcError as e:
--> 564     raise WeaviateQueryError(e.details(), "GRPC search")

WeaviateQueryError: Query call with protocol GRPC search failed with message Socket closed.

I am instantiating weaviate as follows:

target_client = weaviate.connect_to_custom(
    http_host="e.f.g.h",
    http_port="80",
    http_secure=False,
    grpc_host="m.n.o.p",
    grpc_port="50052",
    grpc_secure=False,
    auth_credentials=AuthApiKey("xxxxxxxxxx")
)

Any help is appreciated. Thank you.

2

Hi @vamsi.

Were you able to fix this?

Weaviate Python v4 Client is now available. Also, make sure to run it with the latest Weaviate server, so you need to change the tag in your values.yml too

You shouldn’t have any issues deploying on K8s with our helm chart, unless your loadbalancer is interfering.

Let me know if I can help.

Thanks!

3

This issue has been resolved with the latest versions of python client and weaviate @DudaNogueira . Thanks for the help

1 Like
5

Can you tell me which versions of weaviate and the python client you were using?

I am facing this same issue using weaviate==1.24.5 and weaviate-client==4.6.4

6

hi @zbloss !!

You need to expose both ports for HTTP and GRPC.

How is your cluster deployed?

7

I am running weaviate as a statefulset

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: weaviate-statefulset
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: weaviate
    spec:
      containers:
        - name: weaviate-container
          image: semitechnologies/weaviate:1.25.4
          command: ["/bin/weaviate"]
          args:
            - '--host'
            - '0.0.0.0'
            - '--port'
            - '8080'
            - '--scheme'
            - 'http'
            - --read-timeout=600s 
            - --write-timeout=600s
          ports:
            - containerPort: 8080
            - containerPort: 50051

And have a service object exposing both ports as below:

apiVersion: v1
kind: Service
metadata:
  name: weaviate-service
spec:
  selector:
    matchLabels:
      app: weaviate
  ports:
    - port: 8080
      targetPort: 8080
      name: http
    - port: 2112
      targetPort: 2112
      name: metrics
    - port: 50051
      targetPort: 50051
      name: grpc
  type: ClusterIP
  selector:
    app: weaviate
8

This is how you can check if your GRPC service is up and running:

For non TLS/SSL:

# lets test our grpc connection
❯ wget https://raw.githubusercontent.com/grpc/grpc/master/src/proto/grpc/health/v1/health.proto
❯ grpcurl --plaintext -d '{"service": "Weaviate"}' -proto health.proto grpc.weaviate.mydomain.com:50051 grpc.health.v1.Health/Check
{
  "status": "SERVING"
}

Let me know if you can do this test and have this same outcome?

Thanks!

9

Yes I’m also getting

{
    "status": "SERVING"
}
10

Ok, can you share the code you have that is getting this error?

Thanks!

11

Instantiating a client will produce the error unless I set skip_init_checks=True.

When I enable skip_init_checks I don’t get the error until I run a query.


import weaviate
import weaviate.classes as wvc
from weaviate.collections.classes.internal import Object, QueryReturn

class MyWeaviateClient:

    weaviate_host: str
    weaviate_http_port: int
    weaviate_grpc_port: int
    weaviate_query_timeout_seconds: int
    weaviate_insert_timeout_seconds: int
    weaviate_init_timeout_seconds: int
    weaviate_skip_init_checks: bool

    @property
    def client(self) -> weaviate.WeaviateClient:
        weaviate_client: weaviate.WeaviateClient = weaviate.connect_to_custom(
            http_host=self.weaviate_host,
            http_port=self.weaviate_http_port,
            http_secure=False,
            grpc_host=self.weaviate_host,
            grpc_port=self.weaviate_grpc_port,
            grpc_secure=False,
            additional_config=wvc.init.AdditionalConfig(
                timeout=wvc.init.Timeout(
                    query=self.weaviate_query_timeout_seconds,
                    insert=self.weaviate_insert_timeout_seconds,
                    init=self.weaviate_init_timeout_seconds,
                )
            ),
            skip_init_checks=self.weaviate_skip_init_checks,
        )

        return weaviate_client
    
    @property
    def example_collection(self) -> weaviate.collections.Collection:
        return self.client.collections.get("Example")
    
    def query_example_collection(self, limit: int = 100) -> list[Object]:
        
        query_return: QueryReturn = self.example_collection.query.fetch_objects(
            limit=limit
        )

        return query_return.objects

if __name__ == "__main__":

    client: MyWeaviateClient = MyWeaviateClient(
        weaviate_host="localhost",
        weaviate_http_port=8080,
        weaviate_grpc_port=50051,
        weaviate_query_timeout_seconds=60,
        weaviate_insert_timeout_seconds=60,
        weaviate_init_timeout_seconds=30,
        weaviate_skip_init_checks=True,
    )

    objects: list[Object] = client.query_example_collection(limit=10)
12

can you specify those in your custom_to_custom

http_secure=False,
grpc_secure=False

Thanks!

13

Yes, I get the same error

14

Are you running this locally? Using something like Kind?

Also, are you using our helm chart?

I could try reproducing this deployment.

Thanks!

15

I am not using the helm chart. I have this deployed in google kubernetes engine. The error does not always appear which is equally confusing

16

Strange. Nothing on logs from Weaviate or firewall?

17

Hey @zbloss , did you get your issue solved? what was the RCA? I am also facing similar issue with query when I set skip_init_checks=True.

18

hi @ajay !

Not sure those are related :thinking: