Single-node in k8s deployment issue

Anton · September 8, 2024, 10:08am

Description

I’ve deployed the latest Weaviate helm chart (v17.2.1) in our k8s cluster. I’m trying to run it in a single-node configuration. During a startup error message is displayed in logs:
“error”:“could not join a cluster from [172.16.10.120:8300]”

But after that Weaviate seems to start:
Serving weaviate at http://[::]:8080

Unfortunately shortly after that the server fails with the following error messages:

“error”:“cannot find peer”,“level”:“error”,“msg”:“transferring leadership”

Since this is a single-node application there’re no peers, the process fails and a node restarts.

Server Setup Information

Weaviate Server Version: 1.26.3
Deployment Method: k8s
Multi Node? Number of Running Nodes: 1
Client Language and Version: Python 3.12
Multitenancy?: Yes

Any additional Information

I’m running Weaviate with default configuration. And any help will be much appreciated.

Mohamed_Shahin · September 9, 2024, 12:35pm

Hey @Anton,

Welcome to our community! It’s great to have you with us.

I understand you’re seeing RAFT-related error logs. To explain, RAFT implementation is designed for fault tolerance in multi-node clusters. In a single-node setup, RAFT doesn’t have peers to elect as leader or followers, so when it can’t find peers, leadership transfer can’t proceed, which is why you’re seeing those logs.

Have you noticed any other errors aside from RAFT?

While these RAFT errors are being logged, in some cases, the leadership failure messages might not cause critical issues, and Weaviate can still function as expected for single nodes.

A 3-node setup is generally recommended for more stability with RAFT, especially in production environments, but a single-node setup should still work fine in non-critical deployments. If you’re planning to expand or increase fault tolerance later on, a multi-node cluster will give you better stability.

Could you share more details or the entire log stack with us? Additionally, are there any other errors from the node’s events?

kubectl describe pod weaviate-0

Looking forward to helping you resolve this!

Anton · September 9, 2024, 9:15pm

Hey @Mohamed_Shahin ,

Thank you for getting back to me. The following is Weaviate pod’s description:

Name:             weaviate-0
Namespace:        infra
Priority:         0
Service Account:  default
Node:             gke-main-default-87b2c456-jgm8/10.128.0.71
Start Time:       Sun, 08 Sep 2024 12:36:56 +0300
Labels:           app=weaviate
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=weaviate
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=weaviate-7bccb69488
                  statefulset.kubernetes.io/pod-name=weaviate-0
Annotations:      checksum/config: 7355a42cca09eb72943a4bf637082e7f5b24197d19d2f851109b01875a53fba4
Status:           Running
IP:               172.16.10.52
IPs:
  IP:           172.16.10.52
Controlled By:  StatefulSet/weaviate
Init Containers:
  configure-sysctl:
    Container ID:  containerd://23be7241f52d8a01cc168f0d9776b50c49dcdc0fe26345be6a128ce0eed50d4a
    Image:         docker.io/alpine:latest
    Image ID:      docker.io/library/alpine@sha256:0a4eaa0eecf5f8c050e5bba433f58c052be7587ee8af3e8b3910ef9ab5fbe9f5
    Port:          <none>
    Host Port:     <none>
    Command:
      sysctl
      -w
      vm.max_map_count=524288
      vm.overcommit_memory=1
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 09 Sep 2024 23:59:47 +0300
      Finished:     Mon, 09 Sep 2024 23:59:47 +0300
    Ready:          True
    Restart Count:  333
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tlbt6 (ro)
Containers:
  weaviate:
    Container ID:  containerd://e410a14a74e5b68552ae0d7a3272fa28faa99f372436da3e462f84c1a5fa5c4b
    Image:         cr.weaviate.io/semitechnologies/weaviate:1.26.3
    Image ID:      cr.weaviate.io/semitechnologies/weaviate@sha256:ef1fe73f1918a1a4d52187b738f8fbc795f6a1d0cd4d5d6d0706a3ae5a50a03e
    Ports:         8080/TCP, 50051/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /bin/weaviate
    Args:
      --host
      0.0.0.0
      --port
      8080
      --scheme
      http
      --config-file
      /weaviate-config/conf.yaml
      --read-timeout=60s
      --write-timeout=60s
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 09 Sep 2024 23:58:16 +0300
      Finished:     Mon, 09 Sep 2024 23:59:46 +0300
    Ready:          False
    Restart Count:  338
    Liveness:       http-get http://:8080/v1/.well-known/live delay=900s timeout=3s period=10s #success=1 #failure=30
    Readiness:      http-get http://:8080/v1/.well-known/ready delay=3s timeout=3s period=10s #success=1 #failure=3
    Environment:
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED:  true
      CLUSTER_DATA_BIND_PORT:                   7001
      CLUSTER_DATA_PORT:                        
      CLUSTER_GOSSIP_BIND_PORT:                 7000
      CLUSTER_GOSSIP_PORT:                      
      CLUSTER_HOST:                             
      CLUSTER_IN_LOCALHOST:                     false
      DISABLE_TELEMETRY:                        true
      ENABLE_CLUSTER:                           false
      GOGC:                                     100
      LOG_LEVEL:                                debug
      PROMETHEUS_MONITORING_ENABLED:            false
      PROMETHEUS_MONITORING_GROUP:              false
      QUERY_MAXIMUM_RESULTS:                    100000
      REINDEX_VECTOR_DIMENSIONS_AT_STARTUP:     false
      TRACK_VECTOR_DIMENSIONS:                  false
      CLUSTER_BASIC_AUTH_USERNAME:              <set to the key 'username' in secret 'weaviate-cluster-api-basic-auth'>  Optional: false
      CLUSTER_BASIC_AUTH_PASSWORD:              <set to the key 'password' in secret 'weaviate-cluster-api-basic-auth'>  Optional: false
      PERSISTENCE_DATA_PATH:                    /var/lib/weaviate
      DEFAULT_VECTORIZER_MODULE:                none
      RAFT_JOIN:                                weaviate-0
      RAFT_BOOTSTRAP_EXPECT:                    1
      CLUSTER_JOIN:                             weaviate-headless.infra.svc.cluster.local.
    Mounts:
      /var/lib/weaviate from weaviate-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tlbt6 (ro)
      /weaviate-config from weaviate-config (rw)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  weaviate-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  weaviate-data-weaviate-0
    ReadOnly:   false
  weaviate-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      weaviate-config
    Optional:  false
  kube-api-access-tlbt6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Normal   Killing  7m59s (x338 over 35h)  kubelet  Stopping container weaviate
  Warning  BackOff  4m1s (x8205 over 35h)  kubelet  Back-off restarting failed container weaviate in pod weaviate-0_infra(1eb740cc-e11f-40b2-b30a-ebbcc771716d)

And the full log:

{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"created startup context, nothing done so far","startup_time_left":"59m59.998618881s","time":"2024-09-09T21:04:49Z"}
{"action":"config_load","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","config_file_path":"/weaviate-config/conf.yaml","level":"info","msg":"Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","deprecation":{"apiType":"Configuration","id":"config-files","locations":["--config-file=\"\""],"mitigation":"Configure Weaviate using environment variables.","msg":"use of deprecated command line argument --config-file","sinceTime":"2020-09-08T09:46:00.000Z","sinceVersion":"0.22.16","status":"deprecated"},"level":"warning","msg":"use of deprecated command line argument --config-file","time":"2024-09-09T21:04:49Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-09-09T21:04:49Z"}
{"action":"startup","auto_schema_enabled":true,"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-09-09T21:04:49Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"config loaded","startup_time_left":"59m59.998157221s","time":"2024-09-09T21:04:49Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"configured OIDC and anonymous access client","startup_time_left":"59m59.998140071s","time":"2024-09-09T21:04:49Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"initialized schema","startup_time_left":"59m59.998120461s","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":" memberlist: Initiating push/pull sync with:  172.16.10.52:7000","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":" memberlist: Stream connection from=172.16.10.52:46836","time":"2024-09-09T21:04:49Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"startup routine complete","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-09-09T21:04:49Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"start registering modules","time":"2024-09-09T21:04:49Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"completed registering modules","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"module offload-s3 is not enabled","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"module offload-s3 is enabled","time":"2024-09-09T21:04:49Z"}
{"action":"cluster_api_startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"serving cluster api on port 7001","port":7001,"time":"2024-09-09T21:04:49Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"start initializing modules","time":"2024-09-09T21:04:49Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"finished initializing modules","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"open cluster service","servers":{"weaviate-0":8300},"time":"2024-09-09T21:04:49Z"}
{"address":"172.16.10.52:8301","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"starting cloud rpc server ...","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"starting raft sub-system ...","time":"2024-09-09T21:04:49Z"}
{"address":"172.16.10.52:8300","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"tcp transport","tcpMaxPool":3,"tcpTimeout":10000000000,"time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"loading local db","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"local DB successfully loaded","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"schema manager loaded","n":0,"time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","metadata_only_voters":false,"msg":"construct a new raft node","name":"weaviate-0","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":"1449-1452-1725915496176","last-index":1452,"last-term":1449,"level":"info","msg":"starting restore from snapshot","size-in-bytes":78,"time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"restoring schema from snapshot","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"successfully restored schema from snapshot","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"reload local db: update schema ...","time":"2024-09-09T21:04:49Z"}
{"action":"graphql_rebuild","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"rebuilding the graphql schema","schema":{"Objects":{"classes":[]}},"time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","last_applied_index":1452,"last_snapshot_index":1452,"last_store_log_applied_index":1452,"level":"info","msg":"successfully reloaded indexes from snapshot","n":0,"time":"2024-09-09T21:04:49Z"}
{"action":"","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":"1449-1452-1725915496176","last-index":1452,"last-term":1449,"level":"info","msg":"snapshot restore progress","percent-complete":"[100.00]%","read-bytes":78,"size-in-bytes":78,"time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":"1449-1452-1725915496176","last-index":1452,"last-term":1449,"level":"info","msg":"restored from snapshot","size-in-bytes":78,"time":"2024-09-09T21:04:49Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","index":1,"level":"info","msg":"raft initial configuration","servers":"[[{Suffrage:Voter ID:weaviate-0 Address:172.16.10.51:8301}]]","time":"2024-09-09T21:04:49Z"}
{"action":"raft_cluster_recovery","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","existed_single_cluster_node":{"Suffrage":0,"ID":"weaviate-0","Address":"172.16.10.51:8301"},"level":"info","msg":"perform cluster recovery","new_single_cluster_node":{"Suffrage":0,"ID":"weaviate-0","Address":"172.16.10.52:8301"},"time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"restoring schema from snapshot","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"successfully restored schema from snapshot","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"reload local db: update schema ...","time":"2024-09-09T21:04:49Z"}
{"action":"graphql_rebuild","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"rebuilding the graphql schema","schema":{"Objects":{"classes":[]}},"time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","last_applied_index":1452,"last_snapshot_index":1452,"last_store_log_applied_index":1452,"level":"info","msg":"successfully reloaded indexes from snapshot","n":0,"time":"2024-09-09T21:04:49Z"}
{"action":"","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":"1449-1452-1725915496176","last-index":1452,"last-term":1449,"level":"info","msg":"snapshot restore progress","percent-complete":"[100.00]%","read-bytes":78,"size-in-bytes":78,"time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"persisting snapshot","time":"2024-09-09T21:04:49Z"}
2024-09-09T21:04:49.184Z [INFO]  snapshot: creating new snapshot: path=/var/lib/weaviate/raft/snapshots/1450-1453-1725915889184.tmp
2024-09-09T21:04:49.194Z [INFO]  snapshot: reaping snapshot: path=/var/lib/weaviate/raft/snapshots/1447-1450-1725914796170
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":"1450-1453-1725915889184","last-index":1453,"last-term":1450,"level":"info","msg":"starting restore from snapshot","size-in-bytes":78,"time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"restoring schema from snapshot","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"successfully restored schema from snapshot","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"reload local db: update schema ...","time":"2024-09-09T21:04:49Z"}
{"action":"graphql_rebuild","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"rebuilding the graphql schema","schema":{"Objects":{"classes":[]}},"time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","last_applied_index":1452,"last_snapshot_index":1453,"last_store_log_applied_index":1452,"level":"info","msg":"successfully reloaded indexes from snapshot","n":0,"time":"2024-09-09T21:04:49Z"}
{"action":"","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":"1450-1453-1725915889184","last-index":1453,"last-term":1450,"level":"info","msg":"snapshot restore progress","percent-complete":"[100.00]%","read-bytes":78,"size-in-bytes":78,"time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":"1450-1453-1725915889184","last-index":1453,"last-term":1450,"level":"info","msg":"restored from snapshot","size-in-bytes":78,"time":"2024-09-09T21:04:49Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","index":1,"level":"info","msg":"raft initial configuration","servers":"[[{Suffrage:Voter ID:weaviate-0 Address:172.16.10.52:8301}]]","time":"2024-09-09T21:04:49Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","follower":{},"leader-address":"","leader-id":"","level":"info","msg":"raft entering follower state","time":"2024-09-09T21:04:49Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","last_snapshot_index":1453,"last_store_applied_index_on_start":1452,"level":"info","msg":"raft node constructed","raft_applied_index":1453,"raft_last_index":1453,"time":"2024-09-09T21:04:49Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","last-leader-addr":"","last-leader-id":"","level":"warning","msg":"raft heartbeat timeout reached, starting election","time":"2024-09-09T21:04:50Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"raft entering candidate state","node":{},"term":1451,"time":"2024-09-09T21:04:50Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":"weaviate-0","level":"debug","msg":"raft voting for self","term":1451,"time":"2024-09-09T21:04:50Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"raft calculated votes needed","needed":1,"term":1451,"time":"2024-09-09T21:04:50Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","from":"weaviate-0","level":"debug","msg":"raft vote granted","tally":1,"term":1451,"time":"2024-09-09T21:04:50Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"raft election won","tally":1,"term":1451,"time":"2024-09-09T21:04:50Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","leader":{},"level":"info","msg":"raft entering leader state","time":"2024-09-09T21:04:50Z"}
{"action":"bootstrap","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"node reporting ready, node has probably recovered cluster from raft config. Exiting bootstrap process","time":"2024-09-09T21:04:51Z"}
{"action":"inverted filter2search migration","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"migration skip flag set, skipping migration","time":"2024-09-09T21:04:51Z"}
{"action":"inverted filter2search migration","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"starting switching fallback mode","time":"2024-09-09T21:04:51Z"}
{"action":"inverted filter2search migration","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"no missing filterable indexes, fallback mode skipped","time":"2024-09-09T21:04:51Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","docker_image_tag":"1.26.3","level":"info","msg":"configured versions","server_version":"1.26.3","time":"2024-09-09T21:04:51Z"}
{"action":"grpc_startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-09-09T21:04:51Z"}
{"address":"172.16.10.52:8300","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"current Leader","time":"2024-09-09T21:04:51Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"starting migration from old schema","time":"2024-09-09T21:04:51Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"legacy schema is empty, nothing to migrate","time":"2024-09-09T21:04:51Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"migration from the old schema has been successfully completed","time":"2024-09-09T21:04:51Z"}
{"action":"restapi_management","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","docker_image_tag":"1.26.3","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-09-09T21:04:51Z"}
{"action":"restapi_management","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","docker_image_tag":"1.26.3","level":"info","msg":"Shutting down... ","time":"2024-09-09T21:06:12Z"}
{"action":"restapi_management","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","docker_image_tag":"1.26.3","level":"info","msg":"Stopped serving weaviate at http://[::]:8080","time":"2024-09-09T21:06:12Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing raft FSM store ...","time":"2024-09-09T21:06:12Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"shutting down raft sub-system ...","time":"2024-09-09T21:06:12Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"transferring leadership to another server","time":"2024-09-09T21:06:12Z"}
{"action":"raft","address":null,"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":null,"level":"debug","msg":"raft starting leadership transfer","time":"2024-09-09T21:06:12Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"raft cannot find peer","time":"2024-09-09T21:06:12Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"cannot find peer","level":"error","msg":"transferring leadership","time":"2024-09-09T21:06:12Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing raft-net ...","time":"2024-09-09T21:06:12Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing log store ...","time":"2024-09-09T21:06:12Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing data store ...","time":"2024-09-09T21:06:12Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing loaded database ...","time":"2024-09-09T21:06:12Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing raft-rpc client ...","time":"2024-09-09T21:06:12Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing raft-rpc server ...","time":"2024-09-09T21:06:12Z"}

Please let me know if you need any additional information and thanks for the help.

Mohamed_Shahin · September 11, 2024, 3:30pm

Hey @Anton,

Could you please provide us with the .yaml file? I’d like to try reproducing the issue.

Also, is this a fresh installation, or are you editing the config and restarting the node?

Anton · September 11, 2024, 10:31pm

Hey @Mohamed_Shahin ,

In terms of configuration, we’ve deployed the latest (17.2.1) Weaviate helm chart to our k8s cluster (v1.29) running in GCP. The installation is fresh and the following is a configuration:

  replicas: 3
  env:
    LOG_LEVEL: 'debug'

Currently in three-node configuration the following is the full log of one of the replicas between crashes:

{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"created startup context, nothing done so far","startup_time_left":"59m59.99858618s","time":"2024-09-11T22:28:05Z"}
{"action":"config_load","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","config_file_path":"/weaviate-config/conf.yaml","level":"info","msg":"Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","deprecation":{"apiType":"Configuration","id":"config-files","locations":["--config-file=\"\""],"mitigation":"Configure Weaviate using environment variables.","msg":"use of deprecated command line argument --config-file","sinceTime":"2020-09-08T09:46:00.000Z","sinceVersion":"0.22.16","status":"deprecated"},"level":"warning","msg":"use of deprecated command line argument --config-file","time":"2024-09-11T22:28:05Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-09-11T22:28:05Z"}
{"action":"startup","auto_schema_enabled":true,"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-09-11T22:28:05Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"config loaded","startup_time_left":"59m59.99818449s","time":"2024-09-11T22:28:05Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"configured OIDC and anonymous access client","startup_time_left":"59m59.99815745s","time":"2024-09-11T22:28:05Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"initialized schema","startup_time_left":"59m59.99814683s","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":" memberlist: Failed to join 172.16.17.159:7000: dial tcp 172.16.17.159:7000: connect: connection refused","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":" memberlist: Failed to join 172.16.4.116:7000: dial tcp 172.16.4.116:7000: connect: connection refused","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":" memberlist: Initiating push/pull sync with:  172.16.9.130:7000","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":" memberlist: Stream connection from=172.16.9.130:41992","time":"2024-09-11T22:28:05Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"startup routine complete","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-09-11T22:28:05Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"start registering modules","time":"2024-09-11T22:28:05Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"completed registering modules","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"module offload-s3 is not enabled","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"module offload-s3 is enabled","time":"2024-09-11T22:28:05Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"start initializing modules","time":"2024-09-11T22:28:05Z"}
{"action":"startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"finished initializing modules","time":"2024-09-11T22:28:05Z"}
{"action":"cluster_api_startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"serving cluster api on port 7001","port":7001,"time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"open cluster service","servers":{"weaviate-0":8300,"weaviate-1":8300,"weaviate-2":8300},"time":"2024-09-11T22:28:05Z"}
{"address":"172.16.9.130:8301","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"starting cloud rpc server ...","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"starting raft sub-system ...","time":"2024-09-11T22:28:05Z"}
{"address":"172.16.9.130:8300","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"tcp transport","tcpMaxPool":3,"tcpTimeout":10000000000,"time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"loading local db","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"local DB successfully loaded","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"schema manager loaded","n":0,"time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","metadata_only_voters":false,"msg":"construct a new raft node","name":"weaviate-1","time":"2024-09-11T22:28:05Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","index":340,"level":"info","msg":"raft initial configuration","servers":"[[{Suffrage:Voter ID:weaviate-0 Address:172.16.4.110:8300} {Suffrage:Voter ID:weaviate-2 Address:172.16.17.144:8300} {Suffrage:Voter ID:weaviate-1 Address:172.16.9.51:8300}]]","time":"2024-09-11T22:28:05Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","last_snapshot_index":0,"last_store_applied_index_on_start":0,"level":"info","msg":"raft node constructed","raft_applied_index":0,"raft_last_index":346,"time":"2024-09-11T22:28:05Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","follower":{},"leader-address":"","leader-id":"","level":"info","msg":"raft entering follower state","time":"2024-09-11T22:28:05Z"}
{"action":"bootstrap","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"could not join a cluster from [172.16.9.130:8300]","level":"warning","msg":"failed to join cluster, will notify next if voter","servers":["172.16.9.130:8300"],"time":"2024-09-11T22:28:06Z","voter":true}
{"action":"bootstrap","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","expect":3,"got":{"weaviate-1":"172.16.9.130:8300"},"level":"debug","msg":"number of candidates lower than bootstrap expect param, stopping notify","time":"2024-09-11T22:28:06Z"}
{"action":"bootstrap","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"notified peers this node is ready to join as voter","servers":["172.16.9.130:8300"],"time":"2024-09-11T22:28:06Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","last-leader-addr":"","last-leader-id":"","level":"warning","msg":"raft heartbeat timeout reached, starting election","time":"2024-09-11T22:28:06Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"raft entering candidate state","node":{},"term":7112,"time":"2024-09-11T22:28:06Z"}
{"action":"raft","address":"172.16.4.110:8300","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","from":"weaviate-0","level":"debug","msg":"raft asking for vote","term":7112,"time":"2024-09-11T22:28:06Z"}
{"action":"raft","address":"172.16.17.144:8300","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","from":"weaviate-2","level":"debug","msg":"raft asking for vote","term":7112,"time":"2024-09-11T22:28:06Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":"weaviate-1","level":"debug","msg":"raft voting for self","term":7112,"time":"2024-09-11T22:28:06Z"}
{"action":"raft-net","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"could not resolve server id weaviate-0","fallback":"172.16.4.110:8300","id":"weaviate-0","level":"warning","msg":"raft-net unable to get address for server, using fallback address","time":"2024-09-11T22:28:06Z"}
{"action":"raft-net","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"could not resolve server id weaviate-2","fallback":"172.16.17.144:8300","id":"weaviate-2","level":"warning","msg":"raft-net unable to get address for server, using fallback address","time":"2024-09-11T22:28:06Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"raft calculated votes needed","needed":2,"term":7112,"time":"2024-09-11T22:28:06Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","from":"weaviate-1","level":"debug","msg":"raft vote granted","tally":1,"term":7112,"time":"2024-09-11T22:28:06Z"}
{"action":"inverted filter2search migration","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"migration skip flag set, skipping migration","time":"2024-09-11T22:28:07Z"}
{"action":"inverted filter2search migration","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"starting switching fallback mode","time":"2024-09-11T22:28:07Z"}
{"action":"inverted filter2search migration","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"no missing filterable indexes, fallback mode skipped","time":"2024-09-11T22:28:07Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","docker_image_tag":"1.26.3","level":"info","msg":"configured versions","server_version":"1.26.3","time":"2024-09-11T22:28:07Z"}
{"action":"grpc_startup","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-09-11T22:28:07Z"}
{"action":"restapi_management","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","docker_image_tag":"1.26.3","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-09-11T22:28:07Z"}
{"action":"bootstrap","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"could not join a cluster from [172.16.9.130:8300]","level":"warning","msg":"failed to join cluster, will notify next if voter","servers":["172.16.9.130:8300"],"time":"2024-09-11T22:28:07Z","voter":true}
{"action":"bootstrap","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","expect":3,"got":{"weaviate-1":"172.16.9.130:8300"},"level":"debug","msg":"number of candidates lower than bootstrap expect param, stopping notify","time":"2024-09-11T22:28:07Z"}
{"action":"bootstrap","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"notified peers this node is ready to join as voter","servers":["172.16.9.130:8300"],"time":"2024-09-11T22:28:07Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":" memberlist: Stream connection from=172.16.4.116:42192","time":"2024-09-11T22:28:07Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"warning","msg":"raft Election timeout reached, restarting election","time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"raft entering candidate state","node":{},"term":7113,"time":"2024-09-11T22:28:08Z"}
{"action":"raft","address":"172.16.4.110:8300","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","from":"weaviate-0","level":"debug","msg":"raft asking for vote","term":7113,"time":"2024-09-11T22:28:08Z"}
{"action":"raft","address":"172.16.17.144:8300","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","from":"weaviate-2","level":"debug","msg":"raft asking for vote","term":7113,"time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":"weaviate-1","level":"debug","msg":"raft voting for self","term":7113,"time":"2024-09-11T22:28:08Z"}
{"action":"raft-net","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"could not resolve server id weaviate-2","fallback":"172.16.17.144:8300","id":"weaviate-2","level":"warning","msg":"raft-net unable to get address for server, using fallback address","time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","msg":"raft calculated votes needed","needed":2,"term":7113,"time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","from":"weaviate-1","level":"debug","msg":"raft vote granted","tally":1,"term":7113,"time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","from":"weaviate-0","level":"debug","msg":"raft vote granted","tally":2,"term":7113,"time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"raft election won","tally":2,"term":7113,"time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","leader":{},"level":"info","msg":"raft entering leader state","time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"raft added peer, starting replication","peer":"weaviate-0","time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"raft added peer, starting replication","peer":"weaviate-2","time":"2024-09-11T22:28:08Z"}
{"action":"raft-net","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"could not resolve server id weaviate-2","fallback":"172.16.17.144:8300","id":"weaviate-2","level":"warning","msg":"raft-net unable to get address for server, using fallback address","time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"warning","msg":"raft appendEntries rejected, sending older logs","next":346,"peer":{"Suffrage":0,"ID":"weaviate-0","Address":"172.16.4.110:8300"},"time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"raft pipelining replication","peer":{"Suffrage":0,"ID":"weaviate-0","Address":"172.16.4.110:8300"},"time":"2024-09-11T22:28:08Z"}
{"address":"172.16.9.130:8300","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"current Leader","time":"2024-09-11T22:28:08Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"starting migration from old schema","time":"2024-09-11T22:28:08Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"legacy schema is empty, nothing to migrate","time":"2024-09-11T22:28:08Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"migration from the old schema has been successfully completed","time":"2024-09-11T22:28:08Z"}
{"action":"raft-net","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"could not resolve server id weaviate-2","fallback":"172.16.17.144:8300","id":"weaviate-2","level":"warning","msg":"raft-net unable to get address for server, using fallback address","time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":501887355,"level":"warning","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:08Z"}
{"action":"bootstrap","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"node reporting ready, node has probably recovered cluster from raft config. Exiting bootstrap process","time":"2024-09-11T22:28:08Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":970415071,"level":"warning","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:09Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":1415676468,"level":"warning","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:09Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":1849860337,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:09Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":2298746284,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:10Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":2734670652,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:10Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":3233415067,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:11Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":3716153072,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:11Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":4190246908,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:12Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":4684010383,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:12Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":5120811571,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:13Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":5592213807,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:13Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":6086457392,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:14Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":6563357968,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:14Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","fields.time":7028038025,"level":"debug","msg":"raft failed to contact","server-id":"weaviate-2","time":"2024-09-11T22:28:15Z"}
{"action":"restapi_management","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","docker_image_tag":"1.26.3","level":"info","msg":"Shutting down... ","time":"2024-09-11T22:28:15Z"}
{"action":"restapi_management","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","docker_image_tag":"1.26.3","level":"info","msg":"Stopped serving weaviate at http://[::]:8080","time":"2024-09-11T22:28:15Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing raft FSM store ...","time":"2024-09-11T22:28:15Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"shutting down raft sub-system ...","time":"2024-09-11T22:28:15Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"transferring leadership to another server","time":"2024-09-11T22:28:15Z"}
{"action":"raft","address":null,"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","id":null,"level":"debug","msg":"raft starting leadership transfer","time":"2024-09-11T22:28:15Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"successfully transferred leadership to another server","time":"2024-09-11T22:28:15Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"raft aborting pipeline replication","peer":{"Suffrage":0,"ID":"weaviate-0","Address":"172.16.4.110:8300"},"time":"2024-09-11T22:28:15Z"}
{"action":"raft-net","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","local-address":"172.16.9.130:8300","msg":"raft-net accepted connection","remote-address":"172.16.4.116:46818","time":"2024-09-11T22:28:15Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"dial tcp 172.16.4.110:8300: i/o timeout","level":"error","msg":"raft failed to make requestVote RPC","target":{"Suffrage":0,"ID":"weaviate-0","Address":"172.16.4.110:8300"},"term":7112,"time":"2024-09-11T22:28:16Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"dial tcp 172.16.17.144:8300: i/o timeout","level":"error","msg":"raft failed to make requestVote RPC","target":{"Suffrage":0,"ID":"weaviate-2","Address":"172.16.17.144:8300"},"term":7112,"time":"2024-09-11T22:28:16Z"}
{"action":"raft-net","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","local-address":"172.16.9.130:8300","msg":"raft-net accepted connection","remote-address":"172.16.4.116:46832","time":"2024-09-11T22:28:17Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"dial tcp 172.16.17.144:8300: i/o timeout","level":"error","msg":"raft failed to make requestVote RPC","target":{"Suffrage":0,"ID":"weaviate-2","Address":"172.16.17.144:8300"},"term":7113,"time":"2024-09-11T22:28:18Z"}
{"action":"raft","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"dial tcp 172.16.17.144:8300: i/o timeout","level":"error","msg":"raft failed to appendEntries to","peer":{"Suffrage":0,"ID":"weaviate-2","Address":"172.16.17.144:8300"},"time":"2024-09-11T22:28:18Z"}
{"action":"raft-net","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"debug","local-address":"172.16.9.130:8300","msg":"raft-net accepted connection","remote-address":"172.16.4.116:46846","time":"2024-09-11T22:28:18Z"}
{"action":"raft","backoff time":10000000,"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"dial tcp 172.16.17.144:8300: i/o timeout","level":"error","msg":"raft failed to heartbeat to","peer":"172.16.17.144:8300","time":"2024-09-11T22:28:18Z"}
{"action":"raft-net","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"transport shutdown","level":"error","msg":"raft-net failed to decode incoming command","time":"2024-09-11T22:28:18Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing raft-net ...","time":"2024-09-11T22:28:18Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing log store ...","time":"2024-09-11T22:28:18Z"}
{"action":"raft-net","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"transport shutdown","level":"error","msg":"raft-net failed to decode incoming command","time":"2024-09-11T22:28:18Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing data store ...","time":"2024-09-11T22:28:18Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing loaded database ...","time":"2024-09-11T22:28:18Z"}
{"action":"raft-net","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","error":"transport shutdown","level":"error","msg":"raft-net failed to decode incoming command","time":"2024-09-11T22:28:18Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing raft-rpc client ...","time":"2024-09-11T22:28:18Z"}
{"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"info","msg":"closing raft-rpc server ...","time":"2024-09-11T22:28:18Z"}

Thanks for the help, mate!

Mohamed_Shahin · September 20, 2024, 9:27am

Hey @Anton ,

I’ve attempted the same deployment, and everything worked fine on my end! I wonder if it’s specific to your GCP.

Is the issue still happening? Try deleting the pod and initializing it again to see if that resolves it.

Check the logs for this pod:

Anton · September 23, 2024, 7:51am

Hey @Mohamed_Shahin ,

I agree that it looks like the issue is related to our infrastructure configuration. I’m going to look into this with our DevOps team. For now we’re going forward with testing an Enterprise Cloud solution and will revisit VPC deployment at a later stage.

Thank you for your help!

Topic		Replies	Views
Fatal error during Weaviate 1.25 startup: could not open cloud meta store Support	10	166	July 16, 2024
Weaviate scalability Support	5	457	January 18, 2024
Single node down/failure Support developer-experience	2	187	February 2, 2024
Issue: Weaviate Cluster Setup with Docker on Different Servers Failing Support	3	55	August 12, 2024
Helm cluster, node spontaneously stops and restarts, shard unavailable for 1 min Support	5	401	December 4, 2023

Single-node in k8s deployment issue

Description

Server Setup Information

Any additional Information

Related Topics