# Issues with the nodes joining the cluster **URL:** \ **Category:** Support\ **Created:** [August 29, 2025, 6:11am UTC](https://forum.weaviate.io/t/issues-with-the-nodes-joining-the-cluster/21916 "2025-08-29T06:11:46Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Harishankar](https://avatars.discourse-cdn.com/v4/letter/h/d26b3c/32.png) [@Harishankar](https://forum.weaviate.io/u/Harishankar)\ **Post date:** [August 29, 2025, 6:11am UTC](https://forum.weaviate.io/t/issues-with-the-nodes-joining-the-cluster/21916/1 "2025-08-29T06:11:46Z")
### Description I am using a 3-node cluster with the following Docker Compose. # cat docker-compose.yml version: ‘3.8’ services: weaviate-node-1: image: cr[dot]weaviate[dot]io/semitechnologies/weaviate:1.30.3 ports: - 8080:8080 - 6050:6060 - 50051:50051 # - 8300:8300 restart: on-failure:0 volumes: - weaviate-node-1-data:/var/lib/weaviate environment: CLUSTER\_HOSTNAME: ‘node1’ CLUSTER\_GOSSIP\_BIND\_PORT: ‘7100’ CLUSTER\_DATA\_BIND\_PORT: ‘7101’ RAFT\_BOOTSTRAP\_EXPECT: 3 RAFT\_JOIN: ‘node1,node2,node3’ ENABLE\_API\_BASED\_MODULES: ‘false’ ENABLE\_MODULES: ‘’ DISABLE\_MODULES: ‘text2vec-contextionary,text2vec-transformers,text2vec-openai,text2vec-huggingface’ AUTHENTICATION\_ANONYMOUS\_ACCESS\_ENABLED: ‘true’ PERSISTENCE\_DATA\_PATH: /var/lib/weaviate QUERY\_DEFAULTS\_LIMIT: 1000 QUERY\_MAXIMUM\_RESULTS: 100000 LOG\_LEVEL: debug ASYNC\_INDEXING: ‘true’ ENABLE\_FULL\_METRICS: ‘true’ deploy: resources: limits: cpus: ‘7.0’ memory: ‘14G’ networks: - weaviate-net weaviate-node-2: image: cr[dot]weaviate[dot]io/semitechnologies/weaviate:1.30.3 ports: - 8081:8080 - 6051:6060 - 50052:50051 # - 8301:8300 restart: on-failure:0 volumes: - weaviate-node-2-data:/var/lib/weaviate environment: CLUSTER\_HOSTNAME: ‘node2’ CLUSTER\_GOSSIP\_BIND\_PORT: ‘7102’ CLUSTER\_DATA\_BIND\_PORT: ‘7103’ CLUSTER\_JOIN: ‘weaviate-node-1:7100’ RAFT\_BOOTSTRAP\_EXPECT: 3 RAFT\_JOIN: ‘node1,node2,node3’ ENABLE\_API\_BASED\_MODULES: ‘false’ ENABLE\_MODULES: ‘’ DISABLE\_MODULES: ‘text2vec-contextionary,text2vec-transformers,text2vec-openai,text2vec-huggingface’ AUTHENTICATION\_ANONYMOUS\_ACCESS\_ENABLED: ‘true’ PERSISTENCE\_DATA\_PATH: /var/lib/weaviate QUERY\_DEFAULTS\_LIMIT: 1000 QUERY\_MAXIMUM\_RESULTS: 100000 LOG\_LEVEL: debug ASYNC\_INDEXING: ‘true’ ENABLE\_FULL\_METRICS: ‘true’ deploy: resources: limits: cpus: ‘7.0’ memory: ‘14G’ networks: - weaviate-net weaviate-node-3: image: cr[dot]weaviate[dot]io/semitechnologies/weaviate:1.30.3 ports: - 8082:8080 - 6052:6060 - 50053:50051 # - 8302:8300 restart: on-failure:0 volumes: - weaviate-node-3-data:/var/lib/weaviate environment: CLUSTER\_HOSTNAME: ‘node3’ CLUSTER\_GOSSIP\_BIND\_PORT: ‘7104’ CLUSTER\_DATA\_BIND\_PORT: ‘7105’ CLUSTER\_JOIN: ‘weaviate-node-1:7100’ RAFT\_BOOTSTRAP\_EXPECT: 3 RAFT\_JOIN: ‘node1,node2,node3’ ENABLE\_API\_BASED\_MODULES: ‘false’ ENABLE\_MODULES: ‘’ DISABLE\_MODULES: ‘text2vec-contextionary,text2vec-transformers,text2vec-openai,text2vec-huggingface’ AUTHENTICATION\_ANONYMOUS\_ACCESS\_ENABLED: ‘true’ PERSISTENCE\_DATA\_PATH: /var/lib/weaviate QUERY\_DEFAULTS\_LIMIT: 1000 QUERY\_MAXIMUM\_RESULTS: 100000 LOG\_LEVEL: debug ASYNC\_INDEXING: ‘true’ ENABLE\_FULL\_METRICS: ‘true’ deploy: resources: limits: cpus: ‘7.0’ memory: ‘14G’ networks: - weaviate-net volumes: weaviate-node-1-data: driver: local driver\_opts: type: none device: /mnt/weaviate/pod1 o: bind weaviate-node-2-data: driver: local driver\_opts: type: none device: /mnt/weaviate/pod2 o: bind weaviate-node-3-data: driver: local driver\_opts: type: none device: /mnt/weaviate/pod3 o: bind networks: weaviate-net: driver: bridge My cluster had been running fine for the last three months, but suddenly last night it started showing issues with the nodes joining the cluster. Server Setup Information - Weaviate Server Version: weaviate:1.30.3 - Deployment Method: docker compose - Multi Node? Number of Running Nodes: Yes, 3 nodes - Client Language and Version: English - Multitenancy?: none ### Any additional Information ==================== Logs for weaviate-cluster-weaviate-node-1-1 ==================== {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.19.0.4:8300”,“status”:13,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“fields.time”:117158505668,“level”:“debug”,“msg”:“failed to contact”,“server-id”:“node3”,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“backoff time”:500000000,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“error”:“dial tcp: address 99999999: invalid port”,“level”:“error”,“msg”:“failed to heartbeat to”,“peer”:“172.19.0.2:8300”,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“fields.time”:117609801307,“level”:“debug”,“msg”:“failed to contact”,“server-id”:“node3”,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“backoff time”:500000000,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“error”:“dial tcp: address 99999999: invalid port”,“level”:“error”,“msg”:“failed to heartbeat to”,“peer”:“172.19.0.2:8300”,“time”:“2025-08-29T05:55:12Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:{“node1”:“172.19.0.4:8300”,“node2”:“172.19.0.3:8300”},“time”:“2025-08-29T05:55:12Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.19.0.4:8300”,“status”:13,“time”:“2025-08-29T05:55:12Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.19.0.3:8300”,“status”:8,“time”:“2025-08-29T05:55:12Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“fields.time”:118093069224,“level”:“debug”,“msg”:“failed to contact”,“server-id”:“node3”,“time”:“2025-08-29T05:55:12Z”} {“action”:“raft”,“backoff time”:500000000,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“error”:“dial tcp: address 99999999: invalid port”,“level”:“error”,“msg”:“failed to heartbeat to”,“peer”:“172.19.0.2:8300”,“time”:“2025-08-29T05:55:12Z”} ==================== Logs for weaviate-cluster-weaviate-node-2-1 ==================== {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.19.0.3:8300”,“status”:8,“time”:“2025-08-29T05:55:09Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:{“node1”:“172.19.0.4:8300”,“node2”:“172.19.0.3:8300”},“time”:“2025-08-29T05:55:10Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.19.0.4:8300”,“status”:13,“time”:“2025-08-29T05:55:10Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.19.0.3:8300”,“status”:8,“time”:“2025-08-29T05:55:10Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:{“node1”:“172.19.0.4:8300”,“node2”:“172.19.0.3:8300”},“time”:“2025-08-29T05:55:11Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.19.0.4:8300”,“status”:13,“time”:“2025-08-29T05:55:11Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.19.0.3:8300”,“status”:8,“time”:“2025-08-29T05:55:11Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:{“node1”:“172.19.0.4:8300”,“node2”:“172.19.0.3:8300”},“time”:“2025-08-29T05:55:12Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.19.0.4:8300”,“status”:13,“time”:“2025-08-29T05:55:12Z”} {“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.19.0.3:8300”,“status”:8,“time”:“2025-08-29T05:55:12Z”} ==================== Logs for weaviate-cluster-weaviate-node-3-1 ==================== {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“debug”,“msg”:“calculated votes needed”,“needed”:2,“term”:77,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“from”:“node3”,“level”:“debug”,“msg”:“pre-vote received”,“tally”:0,“term”:77,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“from”:“node3”,“level”:“debug”,“msg”:“pre-vote granted”,“tally”:1,“term”:77,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“error”:“dial tcp: address 99999999: invalid port”,“level”:“error”,“msg”:“failed to make requestVote RPC”,“target”:{“Suffrage”:0,“ID”:“node2”,“Address”:“172.19.0.4:8300”},“term”:77,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“from”:“node2”,“level”:“debug”,“msg”:“pre-vote received”,“tally”:1,“term”:77,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“from”:“node2”,“level”:“debug”,“msg”:“pre-vote denied”,“tally”:1,“term”:77,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“error”:“dial tcp: address 99999999: invalid port”,“level”:“error”,“msg”:“failed to make requestVote RPC”,“target”:{“Suffrage”:0,“ID”:“node1”,“Address”:“172.19.0.3:8300”},“term”:77,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“from”:“node1”,“level”:“debug”,“msg”:“pre-vote received”,“tally”:1,“term”:77,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“from”:“node1”,“level”:“debug”,“msg”:“pre-vote denied”,“tally”:1,“term”:77,“time”:“2025-08-29T05:55:11Z”} {“action”:“raft”,“build\_git\_commit”:“ca35d6b”,“build\_go\_version”:“go1.24.3”,“build\_image\_tag”:“v1.30.3”,“build\_wv\_version”:“1.30.3”,“level”:“info”,“msg”:“pre-vote campaign failed, waiting for election timeout”,“refused”:2,“tally”:1,“term”:77,“time”:“2025-08-29T05:55:11Z”,“votesNeeded”:2} Please suggest steps to resolve the issue, as this is a production setup. I am hesitant to clear the Raft state because in the past, I’ve observed that deleting the Raft directory on all nodes after clearing the Raft state leads to a stable cluster, but we risk losing data and its relationship with the schema. ---
**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\ **Post date:** [August 29, 2025, 1:52pm UTC](https://forum.weaviate.io/t/issues-with-the-nodes-joining-the-cluster/21916/2 "2025-08-29T13:52:46Z")
hi @Harishankar ! Here is where in our docs you can check a docker compose for multi node in docker: > **[Docker | Weaviate Documentation](https://docs.weaviate.io/deploy/installation-guides/docker-installation#multi-node-configuration)** > > Weaviate supports deployment with Docker. Do you have any readings on memory and cpu usage? What could be happening is that Weaviate may be struggling to get resources and not able to answer back for raft heart beats and also can’t join elections as they cannot respond when it get s requested. Also, you can try increasing `RAFT_BOOTSTRAP_TIMEOUT` to give it time in order to abandon the election pool. Also, check if you orchestrator is not killing the container for not being responsible on health checks. Let me know if that helps! ---
**Author:** ![Harishankar](https://avatars.discourse-cdn.com/v4/letter/h/d26b3c/32.png) [@Harishankar](https://forum.weaviate.io/u/Harishankar)\ **Post date:** [August 29, 2025, 2:08pm UTC](https://forum.weaviate.io/t/issues-with-the-nodes-joining-the-cluster/21916/3 "2025-08-29T14:08:03Z")
> [@Harishankar](#): > > Please suggest steps to resolve the issue, as this is a production setup. I am hesitant to clear the Raft state because in the past, I’ve observed that deleting the Raft directory on all nodes after clearing the Raft state leads to a stable cluster, but we risk losing data and its relationship with the schema. Hi @DudaNogueira , Thank you for your reply. No issues found related to OOM , my server got enough CPU and memory resources. Right now the cluster is stable and I am able to see my indexed data and able to indexed new data. Kindly suggest if there is any tool/utility to fix this type of issue or suggest the steps we should perform when this kind of issue happens. Thanks and regards Hari ---
**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\ **Post date:** [September 1, 2025, 12:13pm UTC](https://forum.weaviate.io/t/issues-with-the-nodes-joining-the-cluster/21916/4 "2025-09-01T12:13:28Z")
Hi @Harishankar !! You should first try to update your cluster, as this could have been fixed. Check this issue for more context: [Bug: In cluster mode, the error "dial tcp: address 99999999: invalid port" might occur. · Issue #6555 · weaviate/weaviate · GitHub](https://github.com/weaviate/weaviate/issues/6555) Thanks!