Connect weaviate instance installed on EKS

Support
1

Description

We deployed weavaite instance through helm chart weaviate-helm/weaviate/values.yaml at 853a9c212679931fc750e2550e570fe4ce0e92f9 · weaviate/weaviate-helm · GitHub. We make sure grpc is enabled in the yaml file. While we could connect client , ingest data, and search data through python v3 sdk, we could not connect the client with python v4 sdk. It keeps giving us ConnectTimeout: timed out error.

The host (weaviate.xxx.com) is from kubernetes ingress, which is redirecting it to the apps in kubernetes.

Python v3:
from weaviate import Client
client = Client(“https://host”)
I could check the schema, add data, and use hybrid search.

Python v4:
client = weaviate.connect_to_custom(
http_host=“host”,
http_port=8080,
http_secure=False,
grpc_host=“host”,
grpc_port=50051,
grpc_secure=False,
additional_config=weaviate.config.AdditionalConfig(timeout=(5, 15))
)

ConnectTimeout: timed out

I also tried to use connect_to_local, but got the same connect timeout error.

client = weaviate.connect_to_local(
host=“host”,
port=8080,
grpc_port=50051,
additional_config=weaviate.config.AdditionalConfig(timeout=(5, 15))
)
Same error: ConnectTimeout: timed out
Did we miss sth here?

1 Like
2

Hi @zmliu213 !

Can you try adding, to connect_to_custom:

skip_init_checks=True

If this works, there is probably a high latency between client and the server.

Another way to troubleshoot it this:

# lets test our grpc connection
❯ wget https://raw.githubusercontent.com/grpc/grpc/master/src/proto/grpc/health/v1/health.proto
❯ grpcurl --plaintext -d '{"service": "Weaviate"}' -proto health.proto grpc.weaviate.mydomain.com:50051 grpc.health.v1.Health/Check
{
  "status": "SERVING"
}

Note: This command considers that there is not APIKEY required. If you require a Bearer token, you need to adjust this grpcurl call accordingly

Let me know if this helps

3

Thank you for the response. (1) After adding skip_init_checks=True, I still get connection timeout error. (2) Failed to dial target host “grpc.weaviate.mydomain.com:50051”: dial tcp: lookup grpc.weaviate.mydomain.com: no such host. So the grpc connection is not successful?

4

Not really. This message will usually surface if the client couldn’t connect at that host at all.

Can you make sure that url is publicly reachable?

5

I am also getting a similar error. Couldnt connect using python v4 library. Works fine with v3 library.

I used pre-configured AWS cloudformation and followed steps from this link: AWS Marketplace | Weaviate - Vector Database

6

hi @suryansh ! What is the version installed?

I believe those steps will install a version that doesn’t have GRPC.

The new python v4 client requires Weaviate 1.23.7+.

So for this deployment, in order to use python v4, you will not only have to make sure are running this version (1.23.7 or higher) and also to expose the GRPC port accordingly.

Let me know if this helps!

7

Running into a similar issue. I was able to skip the GRPC error while connecting by adding skip_init_checks=True. But then during ingestion I get a GRPC timeout anyway. The ideal solution is to properly expose the port on AWS.

From what I can see GRPC is not configured for the AWS marketplace deployment option. I don’t see any option related to GRPC in the CloudFormation Stack parameters either. Only port 80 is exposed by the load balancers.

The first post mentions We make sure grpc is enabled in the yaml file. but I’m not sure what that is about. I didn’t interact with any yaml file at all.

weaviate-client==4.6.5
cr.weaviate.io/semitechnologies/weaviate:1.24.8