Issue with multi-node setup

Support
1

Description

We have a 5-node setup hosted through docker compose.

The issue I am facing is:

All the nodes are not added to cluster when I do docker compose up -d. There will always be at least 1 node missing (checking /v1/nodes) from the cluster even though all the containers are up and running.

If I start each node one after the other then they are being added to the cluster immediately.

docker compose up weaviate-node-1 -d
docker compose up weaviate-node-2 -d
docker compose up weaviate-node-3 -d
docker compose up weaviate-node-4 -d
docker compose up weaviate-node-5 -d

Adding healthcheck on first node, depends_on on all nodes did not resolve this issue.

If any container restarts for some reason it is not joining the cluster again as well.

Any help is appreciated in this regard.

Server Setup Information

  • Weaviate Server Version: 1.24.1
  • Deployment Method: docker
  • Multi Node? Number of Running Nodes: 5
  • Client Language and Version:

Any additional Information

2

Hi @vamsi !

Do you see any outstanding logs on the node that has not joined the cluster?

Fortunately, in our next 1.25 version we’ll be delivering raft consensus along with many other improvements that will greatly improve a cluster resilience.

here for more info:

Thanks!