Not able to connect to weaviate instance from vectordbBench

Support
1

I am trying to run vectored bench on weaviate which is running on Kubernetes on EC2 instance. when vectordb is trying to connect to weaviate its getting the below error:

“failed to run, reason=Weaviate did not start up in 5 seconds. Either the Weaviate URL http://10.103.208.135:80/v1 is wrong or Weaviate did not start up in the interval given in 'startup_period”

Now, when I opened weaviate logs below are the output:

ubuntu@ip-172-30-7-158:~$ kubectl logs weaviate-0 --namespace weaviate

Defaulted container “weaviate” out of: weaviate, configure-sysctl (init)

{“action”:“config_load”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“config_file_path”:“/weaviate-config/conf.yaml”,“level”:“info”,“msg”:“Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.”,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“deprecation”:{“apiType”:“Configuration”,“id”:“config-files”,“locations”:[“–config-file=""”],“mitigation”:“Configure Weaviate using environment variables.”,“msg”:“use of deprecated command line argument --config-file”,“sinceTime”:“2020-09-08T09:46:00.000Z”,“sinceVersion”:“0.22.16”,“status”:“deprecated”},“level”:“warning”,“msg”:“use of deprecated command line argument --config-file”,“time”:“2024-11-04T08:59:46Z”}

{“action”:“startup”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“default_vectorizer_module”:“none”,“level”:“info”,“msg”:“the default vectorizer modules is set to "none", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer”,“time”:“2024-11-04T08:59:46Z”}

{“action”:“startup”,“auto_schema_enabled”:true,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“auto schema enabled setting is set to "true"”,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true”,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“module offload-s3 is enabled”,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“open cluster service”,“servers”:{“weaviate-0”:8300},“time”:“2024-11-04T08:59:46Z”}

{“address”:“10.244.0.13:8301”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“starting cloud rpc server …”,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“starting raft sub-system …”,“time”:“2024-11-04T08:59:46Z”}

{“address”:“10.244.0.13:8300”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“tcp transport”,“tcpMaxPool”:3,“tcpTimeout”:10000000000,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“loading local db”,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“local DB successfully loaded”,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“schema manager loaded”,“n”:0,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“metadata_only_voters”:false,“msg”:“construct a new raft node”,“name”:“weaviate-0”,“time”:“2024-11-04T08:59:46Z”}

{“action”:“raft”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“index”:10,“level”:“info”,“msg”:“raft initial configuration”,“servers”:“[[{Suffrage:Voter ID:weaviate-0 Address:10.244.0.10:8300}]]”,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“last_snapshot_index”:0,“last_store_applied_index_on_start”:0,“level”:“info”,“msg”:“raft node constructed”,“raft_applied_index”:0,“raft_last_index”:10,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“hasState”:true,“level”:“info”,“msg”:“raft init”,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:[“10.244.0.13:8300”],“time”:“2024-11-04T08:59:46Z”}

{“action”:“raft”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“follower”:{},“leader-address”:“”,“leader-id”:“”,“level”:“info”,“msg”:“raft entering follower state”,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“10.244.0.13:8300”,“status”:8,“time”:“2024-11-04T08:59:46Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:[“10.244.0.13:8300”],“time”:“2024-11-04T08:59:47Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“10.244.0.13:8300”,“status”:8,“time”:“2024-11-04T08:59:47Z”}

{“action”:“raft”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“last-leader-addr”:“”,“last-leader-id”:“”,“level”:“warning”,“msg”:“raft heartbeat timeout reached, starting election”,“time”:“2024-11-04T08:59:47Z”}

{“action”:“raft”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“raft entering candidate state”,“node”:{},“term”:7,“time”:“2024-11-04T08:59:47Z”}

{“action”:“raft”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“raft pre-vote successful, starting election”,“refused”:0,“tally”:1,“term”:7,“time”:“2024-11-04T08:59:47Z”,“votesNeeded”:1}

{“action”:“raft”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“raft election won”,“tally”:1,“term”:7,“time”:“2024-11-04T08:59:47Z”}

{“action”:“raft”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“leader”:{},“level”:“info”,“msg”:“raft entering leader state”,“time”:“2024-11-04T08:59:47Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“configured versions”,“server_version”:“1.27.0”,“time”:“2024-11-04T08:59:48Z”,“version”:“1.27.0”}

{“action”:“grpc_startup”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“grpc server listening at [::]:50051”,“time”:“2024-11-04T08:59:48Z”}

{“address”:“10.244.0.13:8300”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“current Leader”,“time”:“2024-11-04T08:59:48Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“starting migration from old schema”,“time”:“2024-11-04T08:59:48Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“legacy schema is empty, nothing to migrate”,“time”:“2024-11-04T08:59:48Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“migration from the old schema has been successfully completed”,“time”:“2024-11-04T08:59:48Z”}

{“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:[“10.244.0.13:8300”],“time”:“2024-11-04T08:59:48Z”}

{“action”:“raft”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“command”:0,“level”:“info”,“msg”:“raft updating configuration”,“server-addr”:“10.244.0.13:8300”,“server-id”:“weaviate-0”,“servers”:“[[{Suffrage:Voter ID:weaviate-0 Address:10.244.0.13:8300}]]”,“time”:“2024-11-04T08:59:48Z”}

{“action”:“restapi_management”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“Serving weaviate at http://[::]:8080”,“time”:“2024-11-04T08:59:48Z”,“version”:“1.27.0”}

{“action”:“telemetry_push”,“build_git_commit”:“6c571ff”,“build_go_version”:“go1.22.8”,“build_image_tag”:“”,“build_wv_version”:“”,“level”:“info”,“msg”:“telemetry started”,“payload”:“\u0026{MachineID:c02cd97c-78aa-4ad7-bd89-9f3ec42b0524 Type:INIT Version:1.27.0 NumObjects:0 OS:linux Arch:amd64 UsedModules:}”,“time”:“2024-11-04T08:59:48Z”}

2

Hi!

Are you using our oficial helm chart?

3

yes , I was using your official helm chart. using : helm repo add weaviate Weaviate Helm Charts | Weaviate Helm chart repo

4

Ok!

What is the software you want to connect it with Weaviate?

Were you able to connect using the python client for example?

5

I am trying to run vectorDBbench on weaviate for the performance run. (GitHub - zilliztech/VectorDBBench: A Benchmark Tool for VectorDB).

Were you able to connect using the python client for example? >> I haven’t done this.

6

Oh, I see.

That code/benchmark is using our old client that doesn’t leverage the recent improvements over GRPC. It also targets our cloud, that should be instantiated first and with the proper resources allocated.

You should be able to connect to your cluster like that:

import weaviate
client = weaviate.Client(
    url="http://localhost:8080",
    auth_client_secret=weaviate.auth.AuthApiKey("YOUR API KEY HERE"),
)

that from the same pod running that code.

Note that your error log suggests that you may be passing http://10.103.208.135:80/v1 as the url instead of http://10.103.208.135:80

Let me know if this helps.

7

And by the way, if you want to learn about our take on benchmarking Weaviate, there are some nice contents here:

8

Thanks @DudaNogueira for replying.so , I have three questions here based on ur reply:

  1. whether the vectiordbBench benchmark code is compatible with latest weaviate code base?

  2. since u have mentioned " It also targets our cloud" , does it mean it cannot connect to my Kubernetes cluster locally?

  3. will u be able to tell me which weaviate version (lower version) is compatible with the vectordbBench code?

I will get more clarity on how to move forward, based on these answers.

9

Hi @DudaNogueira, I tried running the solution u proposed from my local machine using python code.

import weaviate
client = weaviate.Client(
url=“http://localhost:8080”,
auth_client_secret=weaviate.auth.AuthApiKey(“YOUR API KEY HERE”),
)

Its showing the error:
weaviate.exceptions.WeaviateStartUpError: Weaviate did not start up in 5 seconds. Either the Weaviate URL http://localhost:50051 is wrong or Weaviate did not start up in the interval given in ‘startup_period’.

So , giving u more info on my setup on which weaviate is running:

I followed: Create a Kubernetes cluster | Weaviate

so , I installed Minikube and kubectl and dowloaded hem chart. so , in the doc it told me to use Minikube tunnel to expose external ip. so after that when I run the python client its giving the above error.

10

Oh I see. You will need to forward the http service port from that pod to 8080.

This is not covered on that docs.

Do you know how to forward those?

Are you also using minikube?

11

yes @DudaNogueira , I am using minikuve . I have followed the exact steps mentioned in: Create a Kubernetes cluster | Weaviate. I am facing issue forwarding request inside Minikube into weaviate cluster

12

I was able to send requests on http port exposed via Minikube tunnel s external IP. whereas for the grpc port I am not able to get it through to the weaviate running in minikube. I tried port forwarding but that is also not working.

13

@pavan @DudaNogueira I also tried minikube, and I followed this blog (Achieve Zero-Downtime Upgrades with Weaviate’s Multi-Node Setup | Weaviate). I am unsure if this is already a bit dated (since we have v4 client). I successfully reached step “minikube service weaviate --namespace weaviate” to expose to “outside world”, and it launched http://127.0.0.1:/v1and there was no apparent error anywhere. But if I tried to connect using client v4:

client = weaviate.connect_to_local(‘localhost’, …)

I got errors, and this may be pertinent:

AioRpcError: <AioRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = “failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: Connection refused”
debug_error_string = “UNKNOWN:Error received from peer {grpc_message:“failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: Connection refused”, grpc_status:14, created_time:“2024-11-20T13:31:57.896269-05:00”}”

it seems it says there’s no grpc available. I thought it was mentioned this is used in the client and not REST. So I suspect the blog is missing another step where grpc should also be exposed?

14

Hi!

Here is how I do to connect to a minikube cluster with 3 nodes:

>> minikube start --nodes 3

>> kubectl get nodes
NAME           STATUS   ROLES           AGE     VERSION
minikube       Ready    control-plane   4m17s   v1.31.0
minikube-m02   Ready    <none>          4m7s    v1.31.0
minikube-m03   Ready    <none>          3m59s   v1.31.0

>> helm repo add weaviate https://weaviate.github.io/weaviate-helm

>> helm show values weaviate/weaviate > values.yaml

edit the values.yaml and enable the openai and set the replicas to 3

create the namespace:

>> kubectl create namespace weaviate

deploy the helm chart:

>> helm upgrade --install \
  "weaviate" \
  weaviate/weaviate \
  --namespace "weaviate" \
  --values ./values.yaml

watch your cluster starting, and wait all pods to be ready

>> kubectl -n weaviate get pods -w

those are the services we have:

>> kubectl -n weaviate get services
NAME                TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
weaviate            LoadBalancer   10.107.66.168   <pending>     80:32443/TCP      106s
weaviate-grpc       LoadBalancer   10.97.59.100    <pending>     50051:30758/TCP   106s
weaviate-headless   ClusterIP      None            <none>        80/TCP            106s

we need to forward both http and grpc ports
note that the service for http is on port 80 and grpc 50051

kubectl port-forward service/weaviate 8080:80 -n weaviate
kubectl port-forward service/weaviate-grpc 50051:50051 -n weaviate

In order to test the GRPC endpoint, you can use grpcurl. Here is how to test without any auth or ssl

>> wget https://raw.githubusercontent.com/grpc/grpc/master/src/proto/grpc/health/v1/health.proto
>> grpcurl -plaintext -d '{"service": "Weaviate"}' -proto health.proto localhost:50051 grpc.health.v1.Health/Check
{
  "status": "SERVING"
}

Let me know if this helps!

15

This works! thanks a lot.

1 Like