Replication setup on AWS EC2 with Docker container

Support
1

Description

I would like to bring your notice and get help to resolve the replication setup issue on Docker container, I have setup the 3 nodes on AWS EC2 machine and installed the docker on top of it
My configuration docker compose files like below I just given node3 config file.

weaviate-node-3:
init: true
command:
- --host
- 0.0.0.0
- --port
- ‘8080’
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.28.4
ports:
- 8080:8080
- 50051:50051
restart: on-failure:0
volumes:
- /data:/data
environment:
LOG_LEVEL: ‘debug’
QUERY_DEFAULTS_LIMIT: 25
PERSISTENCE_DATA_PATH: ‘/data’
ENABLE_MODULES: ‘backup-s3,backup-filesystem’
BACKUP_S3_BUCKET: ‘your-s3-bucket-name’
BACKUP_S3_PATH: ‘weaviate-backups’
BACKUP_FILESYSTEM_PATH: ‘/data/Backups’
AWS_ACCESS_KEY_ID: ‘your-aws-access-key-id’
AWS_SECRET_ACCESS_KEY: ‘your-aws-secret-access-key’
AWS_REGION: ‘your-aws-region’
ENABLE_API_BASED_MODULES: ‘true’
CLUSTER_HOSTNAME: ‘node3’
CLUSTER_GOSSIP_BIND_PORT: ‘7104’
CLUSTER_DATA_BIND_PORT: ‘7105’
CLUSTER_JOIN: ‘node1IP:7100’
RAFT_JOIN: ‘10.X.XX.XX:7100,10.X.XX.XX:7100,10.X.XX.XX:7100’
RAFT_BOOTSTRAP_EXPECT: 3
AUTHENTICATION_APIKEY_ENABLED: ‘true’
AUTHENTICATION_APIKEY_ALLOWED_KEYS: ‘’
AUTHENTICATION_APIKEY_USERS: ‘user’
When I am trying to start the weaviate service all 3 nodes one by one
I am seeing the below errors.
{“action”:“memberlist_init”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“error”:“1 error occurred:\n\t* Failed to join node1:7100: dial tcp node1:7100: connect: connection refused\n\n”,“level”:“error”,“msg”:“memberlist join not successful”,“remote_hostname”:[“node1:7100”],“time”:“2025-02-24T10:42:19Z”}
{“action”:“bootstrap”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“join_list”:{“node1”:7100,“node2”:7100,“node3”:7100},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2025-02-24T10:32:45Z”}

Server Setup Information

  • Weaviate Server Version: 1.28.4
  • Deployment Method:
  • Multi Node? Number of Running Nodes: 3
  • Client Language and Version: python V4
  • Multitenancy?:

Any additional Information

{“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“module offload-s3 is enabled”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“cluster_api_startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“msg”:“serving cluster api on port 7101”,“port”:7101,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“msg”:“start initializing modules”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“reranker-voyageai”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“reranker-cohere”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“generative-aws”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-aws”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“multi2vec-cohere”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“multi2vec-voyageai”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-jinaai”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“generative-octoai”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-octoai”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“multi2vec-google”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-huggingface”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“reranker-jinaai”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“generative-openai”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-google”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-openai”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“generative-friendliai”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“generative-databricks”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“generative-anyscale”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“backup-s3”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-databricks”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-voyageai”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-mistral”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-weaviate”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“generative-cohere”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“generative-anthropic”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“backup-filesystem”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-cohere”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“multi2vec-jinaai”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“generative-mistral”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“generative-google”,“msg”:“initialized module”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“multi2vec-jinaai”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-aws”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“multi2vec-cohere”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“multi2vec-voyageai”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-jinaai”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-octoai”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“multi2vec-google”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-huggingface”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-google”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-openai”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-databricks”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-voyageai”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-mistral”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-weaviate”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“module”:“text2vec-cohere”,“msg”:“initialized module extension”,“time”:“2025-02-24T10:32:00Z”}
{“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“warning”,“msg”:“Multiple vector spaces are present, GraphQL Explore and REST API list objects endpoint module include params has been disabled as a result.”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“msg”:“finished initializing modules”,“time”:“2025-02-24T10:32:00Z”}
{“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“open cluster service”,“servers”:{“node1”:7100,“node2”:7100,“node3”:7100},“time”:“2025-02-24T10:32:00Z”}
{“address”:“172.18.0.2:8301”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“starting cloud rpc server …”,“time”:“2025-02-24T10:32:00Z”}
{“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“starting raft sub-system …”,“time”:“2025-02-24T10:32:00Z”}
{“address”:“172.18.0.2:8300”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“tcp transport”,“tcpMaxPool”:3,“tcpTimeout”:10000000000,“time”:“2025-02-24T10:32:00Z”}
{“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“loading local db”,“time”:“2025-02-24T10:32:00Z”}
{“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“local DB successfully loaded”,“time”:“2025-02-24T10:32:00Z”}
{“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“schema manager loaded”,“n”:0,“time”:“2025-02-24T10:32:00Z”}
{“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“metadata_only_voters”:false,“msg”:“construct a new raft node”,“name”:“node1”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“raft”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“index”:0,“level”:“info”,“msg”:“initial configuration”,“servers”:“[]”,“time”:“2025-02-24T10:32:00Z”}
{“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“last_snapshot_index”:0,“last_store_applied_index_on_start”:0,“level”:“info”,“msg”:“raft node constructed”,“raft_applied_index”:0,“raft_last_index”:0,“time”:“2025-02-24T10:32:00Z”}
{“action”:“raft”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“follower”:{},“leader-address”:“”,“leader-id”:“”,“level”:“info”,“msg”:“entering follower state”,“time”:“2025-02-24T10:32:00Z”}
{“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“hasState”:false,“level”:“info”,“msg”:“raft init”,“time”:“2025-02-24T10:32:00Z”}
{“action”:“bootstrap”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“join_list”:{“node1”:7100,“node2”:7100,“node3”:7100},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2025-02-24T10:32:01Z”}
{“action”:“raft”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“warning”,“msg”:“no known peers, aborting election”,“time”:“2025-02-24T10:32:02Z”}
{“action”:“inverted filter2search migration”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“msg”:“migration skip flag set, skipping migration”,“time”:“2025-02-24T10:32:02Z”}
{“action”:“inverted filter2search migration”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“msg”:“starting switching fallback mode”,“time”:“2025-02-24T10:32:02Z”}
{“action”:“inverted filter2search migration”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“debug”,“msg”:“no missing filterable indexes, fallback mode skipped”,“time”:“2025-02-24T10:32:02Z”}
{“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“configured versions”,“server_version”:“1.28.4”,“time”:“2025-02-24T10:32:02Z”,“version”:“1.28.4”}
{“action”:“grpc_startup”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“grpc server listening at [::]:50051”,“time”:“2025-02-24T10:32:02Z”}
{“action”:“restapi_management”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“Serving weaviate at http://[::]:8080”,“time”:“2025-02-24T10:32:02Z”,“version”:“1.28.4”}
{“action”:“telemetry_push”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“level”:“info”,“msg”:“telemetry started”,“payload”:“\u0026{MachineID:1e257d0b-a36e-47fd-a45a-1910b1882915 Type:INIT Version:1.28.4 NumObjects:0 OS:linux Arch:amd64 UsedModules:}”,“time”:“2025-02-24T10:32:03Z”}
{“action”:“bootstrap”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“join_list”:{“node1”:7100,“node2”:7100,“node3”:7100},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2025-02-24T10:32:03Z”}
{“action”:“bootstrap”,“build_git_commit”:“d4c811b”,“build_go_version”:“go1.22.11”,“build_image_tag”:“v1.28.4”,“build_wv_version”:“1.28.4”,“join_list”:{“node1”:7100,“node2”:7100,“node3”:7100},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2025-02-24T10:32:05Z”}

2

Hello @DudaNogueira :

I still haven’t received any response. While attempting to configure with 3 nodes replication, I observed that all nodes were displaying the same error. Do you have any suggestions on this?

Log:

{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“expect”:3,“got”:{“weaviate-node-3”:“172.18.0.2:8300”},“level”:“debug”,“msg”:“number of candidates lower than bootstrap expect param, stopping notify”,“time”:“2025-02-25T11:27:39Z”}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:39Z”}
{“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:40Z”}
{“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.18.0.2:8300”,“status”:8,“time”:“2025-02-25T11:27:40Z”}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“error”:“could not join a cluster from [172.18.0.2:8300]”,“level”:“warning”,“msg”:“failed to join cluster”,“servers”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:40Z”,“voter”:true}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“expect”:3,“got”:{“weaviate-node-3”:“172.18.0.2:8300”},“level”:“debug”,“msg”:“number of candidates lower than bootstrap expect param, stopping notify”,“time”:“2025-02-25T11:27:40Z”}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:40Z”}
{“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:41Z”}
{“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.18.0.2:8300”,“status”:8,“time”:“2025-02-25T11:27:41Z”}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“error”:“could not join a cluster from [172.18.0.2:8300]”,“level”:“warning”,“msg”:“failed to join cluster”,“servers”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:41Z”,“voter”:true}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“expect”:3,“got”:{“weaviate-node-3”:“172.18.0.2:8300”},“level”:“debug”,“msg”:“number of candidates lower than bootstrap expect param, stopping notify”,“time”:“2025-02-25T11:27:41Z”}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:41Z”}
{“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:43Z”}
{“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.18.0.2:8300”,“status”:8,“time”:“2025-02-25T11:27:43Z”}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“error”:“could not join a cluster from [172.18.0.2:8300]”,“level”:“warning”,“msg”:“failed to join cluster”,“servers”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:43Z”,“voter”:true}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“expect”:3,“got”:{“weaviate-node-3”:“172.18.0.2:8300”},“level”:“debug”,“msg”:“number of candidates lower than bootstrap expect param, stopping notify”,“time”:“2025-02-25T11:27:43Z”}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:43Z”}
{“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:44Z”}
{“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.18.0.2:8300”,“status”:8,“time”:“2025-02-25T11:27:44Z”}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“error”:“could not join a cluster from [172.18.0.2:8300]”,“level”:“warning”,“msg”:“failed to join cluster”,“servers”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:44Z”,“voter”:true}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“expect”:3,“got”:{“weaviate-node-3”:“172.18.0.2:8300”},“level”:“debug”,“msg”:“number of candidates lower than bootstrap expect param, stopping notify”,“time”:“2025-02-25T11:27:44Z”}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:44Z”}
{“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:45Z”}
{“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“172.18.0.2:8300”,“status”:8,“time”:“2025-02-25T11:27:45Z”}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“error”:“could not join a cluster from [172.18.0.2:8300]”,“level”:“warning”,“msg”:“failed to join cluster”,“servers”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:45Z”,“voter”:true}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“expect”:3,“got”:{“weaviate-node-3”:“172.18.0.2:8300”},“level”:“debug”,“msg”:“number of candidates lower than bootstrap expect param, stopping notify”,“time”:“2025-02-25T11:27:45Z”}
{“action”:“bootstrap”,“build_git_commit”:“35d800d”,“build_go_version”:“go1.22.12”,“build_image_tag”:“v1.29.0”,“build_wv_version”:“1.29.0”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.18.0.2:8300”],“time”:“2025-02-25T11:27:45Z”}

3

hi @RamuA !!

Sorry for the late response here.

It seems that nodes are not communicating to each other.

I have seen some issues like this, but while using docker swarm.

I will need to escalate this to our team.

Right now, our recommended way to run multiple node cluster is using Kubernetes:

I will try to experiment with this and docker swarm later this week.

Thanks!