GRPC Health Check Failing

Support
1

Hi Guys, I am attempting to set up Weaviate V4 within a Kubernetes cluster. I have also set up an ingress (with ssl redirect: true, backend protocol: grpcs) so I can communicate with it, but I keep failing the health checks during connection. For example, if I run:

grpcurl -d '{"service": "Weaviate"}' -proto health.proto my-url.org:50051 grpc.health.v1.Health/Check

I get: Failed to dial target host "my-url:50051": context deadline exceeded

I have tried multiple ways to fix the issue, but none of them worked:

  • skip init checks, but this causes the query to fail with error: An unexpected error occurred: Query call with protocol GRPC search failed with message failed to connect to all addresses; last error: UNKNOWN: ipv4:xxx:50051: Failed to connect to remote host: FD Shutdown.. Something strange is that loading data into collections is perfectly fine, with no errors.
  • setting the environment variables REQUESTS_CA_BUNDLE, SSL_CERT_FILE and GRPC_DEFAULT_SSL_ROOTS_FILE_PATH to my CA certificate (containing my self-signed certificates for weaviate and weaviate gRPC in the same file), but this did not fix the health check issue
  • trying grpc_secure to be false / true, but this did not work

I figured out a way to run the client locally, and that is to run kubectl port-forward svc/weaviate-grpc 50051:50051. However, I do not want to do this when running the client from within the same namespace as my weaviate service since port forwarding should not be needed.
Any help would be greatly appreciated!

Full logs on health-check error:

2024-04-24 14:14:52.568 | ERROR    | __main__:<module>:110 - An unexpected error occurred: gRPC health check could not be completed. This could have several reasons:
        - gRPC is not enabled or incorrectly configured on the server with version v1.24.8 or the client
        - your connection is unstable or has a high latency. In this case you can:
            - increase init-timeout in `weaviate.connect_to_local(additional_config=wvc.init.AdditionalConfig(timeout=wvc.init.Timeout(init=X)))`
            - disable startup checks by connecting using `skip_init_checks=True`
2

Hi!

How is the type of your grpcService? in the values.yaml?

Are you deploying it in your own K8s or some service?

Thanks!

3

Hi, thank you for replying! It is of type NodePort, and in my own K8s.

Currently I can query using my local machine via port forwarding, but if I run the query script in a pod within the same cluster & namespace as my weaviate-0 pod, I face the issues. But loading is fine - so it probably isn’t using gRPC for data insertions?

4

Would it be possible to query the grpc service directly? e.g. client = Client("weaviate-grpc:50051") . I am running my benchmark pod in the same cluster & namespace as my weaviate services, so this can bypass the ingress issues. Thanks a lot again for your help.

5

Issue fixed! I set grpc_host to the service name directly, and set secure to False

1 Like
6

thanks for sharing!!