[Question] synchronized:false but nodes can see each other

Support
1

Hi there!
I’ve been trying to set up 3 weaviate servers in a cluster for High Availability, using Google Cloud.

I’m nearly done with the configuration, but when I review on v1/cluster/statistics, it displays “synchronized”: false at the bottom, despite all nodes being Healthy, with one marked as a Leader and the other 2 as followers

I’ve reviewed ports/firewalls to make sure communication can happen on ports:
7100, 7101, 8080, 8300, 8301, 50051

I’m using the same 7100/7101 ports on all 3 servers, not sure if that’s an issue, but the fact that they seem to be talking to each other (even if they aren’t synching yet) leads me to believe that’s not a major issue.

I’m running 1.26.4,

with the following env vars:

CLUSTER_GOSSIP_BIND_PORT 7100
CLUSTER_DATA_BIND_PORT 7101
RAFT_JOIN main-server-name.us-central1-a.c.google-project.internal,alt-one.us-west1-b.c.google-project.internal,alt-two.us-east1-b.c.google-project.internal
RAFT_BOOTSTRAP_EXPECT 3
CLUSTER_JOIN main-server-name.us-central1-a.c.google-project.internal:7100
RAFT_ENABLE_FQDN_RESOLVER true

The CLUSTER_JOIN var isn’t set on the main server, as per the documentation.

Help is very much appreciated. Thanks!

Edit:

They’re up and running now. I ended up destroying all 3 hard drives and letting weaviate recreate the /var/lib/weaviate data, and it seems to be fine now.

1 Like
2

hi @Juan_Rodriguez !!

Welcome to our community :hugs:

Glad you found a solution.

Not sure you are using our helm chart, but we strongly suggest using it as it’s the best way to have the same environment as our users have and that our team expects so we can help.

Let me know if there is any other issues we can help with!

Happy building with Weaviate :slight_smile: