Should founding node be able to rejoin cluster in multinode setup?

Dominik_Doberski · June 10, 2025, 7:10am

Hello,

I’m trying to set up a 3-node Weaviate cluster on ECS. I followed the Docker tutorial (Docker | Weaviate) and other related topics on this forum. Everything works fine, but when I restart the founding node (to simulate a crash), it doesn’t reconnect to the existing cluster with the other nodes. Instead, it creates a new cluster.

Is this the expected behavior, or should the founding node be able to rejoin the original cluster? I’m using Weaviate version 1.31.0.

Thanks!

Mohamed_Shahin · June 10, 2025, 8:31am

Good morning @Dominik_Doberski,

Welcome to our community! It’s lovely to have you here, mate I hope you’re having a great week.

What you’re experiencing isn’t the expected behavior. In a correctly configured Weaviate cluster, a node should be able to rejoin after a restart rather than forming a new cluster. Cluster membership and state should persist across restarts—assuming the configuration is intact.

Could you please share your configuration file with me? I’d be happy to take a closer look and, if possible, try to replicate the issue on my side. Any additional details you can provide would also be helpful.

Best regards,

Mohamed Shahin
Weaviate Support Engineer
(Ireland, GMT/UTC timezone)

andrewisplinghoff · June 10, 2025, 12:06pm

I noticed the same behaviour. On the founding node, CLUSTER_JOIN is not being set and without that, the existing memberlist cluster will not be joined (code doesn’t go into this conditional):

github.com/weaviate/weaviate

usecases/cluster/state.go

3190fa73a


      
          			"hostname":  userConfig.Hostname,
          			"bind_port": userConfig.GossipBindPort,
          		}).WithError(err).Error("memberlist not created")
          		return nil, errors.Wrap(err, "create member list")
          	}
          	var joinAddr []string
          	if userConfig.Join != "" {
          		joinAddr = strings.Split(userConfig.Join, ",")
          	}
          
          	if len(joinAddr) > 0 {
          		_, err := net.LookupIP(strings.Split(joinAddr[0], ":")[0])
          		if err != nil {
          			logger.WithFields(logrus.Fields{
          				"action":          "cluster_attempt_join",
          				"remote_hostname": joinAddr[0],
          			}).WithError(err).Warn(
          				"specified hostname to join cluster cannot be resolved. This is fine" +
          					"if this is the first node of a new cluster, but problematic otherwise.")
          		} else {
          			_, err := state.list.Join(joinAddr)

Mohamed_Shahin · June 10, 2025, 12:29pm

@andrewisplinghoff Thank you so much! Absolutely — @Dominik_Doberski, in your configuration, please make sure to review the following documentation for multi-node setup:

See the CLUSTER_JOIN parameter. It must be set to the service name of the founding node in the cluster. This ensures proper cluster formation and rejoining behavior after restarts.

Best regards,

Mohamed Shahin
Weaviate Support Engineer
(Ireland, GMT/UTC timezone)

Dominik_Doberski · June 10, 2025, 12:40pm

Thank you for the reply. The documentation you referenced states that the CLUSTER_JOIN variable should only be set for nodes other than the founding node. However, my issue is that the founding node does not reconnect to the cluster after a restart. The rest of the nodes are functioning as expected.

Below is the configuration of environment variables we use for all nodes (we are deploying using Pulumi):

environment: [
{ name: 'QUERY_DEFAULTS_LIMIT', value: '25' },
{ name: 'AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED', value: 'true' },
{ name: 'PERSISTENCE_DATA_PATH', value: '/var/lib/weaviate' },
{ name: 'DEFAULT_VECTORIZER_MODULE', value: 'none' },
{ name: 'ENABLE_API_BASED_MODULES', value: 'true' },
{ name: 'LOG_LEVEL', value: 'debug' },
{ name: 'CLUSTER_HOSTNAME', value: nodeName },
{ name: 'CLUSTER_GOSSIP_BIND_PORT', value: gossipBindPort },
{ name: 'CLUSTER_DATA_BIND_PORT', value: dataBindPort },
{ name: 'RAFT_JOIN', value: nodeJoinList }, // "node1,node2,node3"
{
  name: 'RAFT_BOOTSTRAP_EXPECT',
  value: WeaviateService.NODE_NAMES.length.toString(),
},
{
  name: 'REPLICATION_MINIMUM_FACTOR',
  value: WeaviateService.NODE_NAMES.length.toString(),
},
{ name: 'DEFAULT_SHARD_COUNT', value: '1' },
...(isFirstNode
  ? []
  : [
      {
        name: 'CLUSTER_JOIN',
        value: `node1.weaviate.local:${WeaviateService.BASE_GOSSIP_PORT}`,
      },
    ]),
],

We are using AWS Cloud Map to ensure proper IP resolution between nodes. This is why node1.weaviate.local:${WeaviateService.BASE_GOSSIP_PORT} is used for the CLUSTER_JOIN variable.

Mohamed_Shahin · June 13, 2025, 8:17am

I’m not entirely sure how to replicate this yet, let’s see if someone has a similar environment who can help even from community.

Would you mind raising this as an issue on the Weaviate GitHub repo? It might be RAFT-related and worth the team’s attention:

Best regards,
Mohamed Shahin
Weaviate Support Engineer
(Ireland, UTC±00:00/+01:00)

dhanshew72 · June 16, 2025, 6:11pm

Adding to this thread, I’ve had issues with the founding node crashing on AWS ECS because of this constant (AWS ECS is slow to deploy nodes):

github.com/weaviate/weaviate

usecases/nodes/handler.go

054778707


      
          	"github.com/weaviate/weaviate/usecases/auth/authorization/rbac/rbacconf"
          
          	"github.com/weaviate/weaviate/entities/verbosity"
          
          	"github.com/sirupsen/logrus"
          	"github.com/weaviate/weaviate/entities/models"
          	"github.com/weaviate/weaviate/usecases/auth/authorization"
          	schemaUC "github.com/weaviate/weaviate/usecases/schema"
          )
          
          const GetNodeStatusTimeout = 30 * time.Second
          
          type db interface {
          	GetNodeStatus(ctx context.Context, className, shardName, verbosity string) ([]*models.NodeStatus, error)
          	GetNodeStatistics(ctx context.Context) ([]*models.Statistics, error)
          }
          
          type Manager struct {
          	logger        logrus.FieldLogger
          	authorizer    authorization.Authorizer
          	db            db

Founding nodes that take longer than 30 seconds to restart end up spinning up without other nodes joining than just hang until restarting the rest of the nodes. I fought against this a little by writing a custom restart job, but can’t handle unexpected crashes of the node without manual intervention.

Is there a plan to configure this value?

Dominik_Doberski · June 18, 2025, 9:46am

Thanks for the replies. I created an issue Founding node is unable to rejoin cluster on ECS · Issue #8423 · weaviate/weaviate · GitHub

MohamedBadawi · June 24, 2025, 2:45pm

Hey @Dominik_Doberski,

Thanks for reporting the issue.

I’m curious—what value does CLUSTER_JOIN have for the node after it restarts? From the snippet you shared, it looks like CLUSTER_JOIN might be empty. One possible reason could be the logic behind isFirstNode.

If you get a chance, could you debug the node’s environment variables and check what value is being set for CLUSTER_JOIN? The behavior you’re seeing does seem consistent with it being empty.

Thanks!

...(isFirstNode
  ? []
  : [
      {
        name: 'CLUSTER_JOIN',
        value: `node1.weaviate.local:${WeaviateService.BASE_GOSSIP_PORT}`,
      },
    ]),
],

Topic		Replies	Views
Unable to restart my weaviate container Support	8	1599	May 20, 2024
Weaviate Cluster Setup with Docker on Different Servers Failing General bug , integration , developer-experience , python	13	958	July 4, 2025
Issue: Weaviate Cluster Setup with Docker on Different Servers Failing Support	5	634	December 3, 2024
Production Cluster weaviate multinode without K(S Support	9	427	April 22, 2025
Issue with multi-node setup Support	1	286	April 20, 2024

Should founding node be able to rejoin cluster in multinode setup?

Related topics