Docker Compose with multi node deployment issue

Support
1

Description

Hi, I am using docker compose to deploy Weaviate on two machines. The first machine’s docker compose yml file is:

services:
  weaviate-node-1:
    init: true
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: cr.weaviate.io/semitechnologies/weaviate:1.24.2
    ports:
    - 8080:8080
    - 7100:7100
    - 7101:7101
    - 50051:50051
    restart: on-failure:0
    volumes:
      - ./data-node-1:/var/lib/weaviate
    environment:
      LOG_LEVEL: 'debug'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      CLUSTER_HOSTNAME: 'node1'
      CLUSTER_GOSSIP_BIND_PORT: '7100'
      CLUSTER_DATA_BIND_PORT: '7101'
    networks:
      - weaviate
networks:
  weaviate:
    attachable: true

The second machine docker compose file is:

services:
  weaviate-node-2:
    init: true
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8081'
    - --scheme
    - http
    image: cr.weaviate.io/semitechnologies/weaviate:1.24.2
    ports:
    - 8081:8080
    - 7102:7102
    - 7103:7103
    - 50052:50051
    restart: on-failure:0
    volumes:
      - ./data-node-2:/var/lib/weaviate
    environment:
      LOG_LEVEL: 'debug'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      CLUSTER_HOSTNAME: 'node2'
      CLUSTER_GOSSIP_BIND_PORT: '7102'
      CLUSTER_DATA_BIND_PORT: '7103'
      CLUSTER_JOIN: '10.12.226.225:7100'
    networks:
      - weaviate
networks:
  weaviate:
    attachable: true

after I ran docker compose up and I got the error as shown in the following logs.
The log from node 1:

[+] Running 2/1
 ✔ Network weaviate_testing_weaviate             Created                                                                                                    0.2s 
 ✔ Container weaviate_testing-weaviate-node-1-1  Created                                                                                                    0.0s 
Attaching to weaviate-node-1-1
weaviate-node-1-1  | {"action":"startup","level":"debug","msg":"created startup context, nothing done so far","startup_time_left":"59m59.997886965s","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"startup","level":"debug","msg":"config loaded","startup_time_left":"59m59.997070446s","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"startup","level":"debug","msg":"configured OIDC and anonymous access client","startup_time_left":"59m59.997044326s","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"startup","level":"debug","msg":"initialized schema","startup_time_left":"59m59.997008365s","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"startup","level":"debug","msg":"startup routine complete","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"startup","level":"debug","msg":"start registering modules","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"startup","level":"debug","msg":"completed registering modules","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"startup_cluster_schema_sync","level":"debug","msg":"Only node in the cluster at this point. No schema sync necessary.","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"inverted filter2search migration","level":"debug","msg":"migration skip flag set, skipping migration","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"cluster_api_startup","level":"debug","msg":"serving cluster api on port 7101","port":7101,"time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"inverted filter2search migration","level":"debug","msg":"starting switching fallback mode","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"inverted filter2search migration","level":"debug","msg":"no missing filterable indexes, fallback mode skipped","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"startup","level":"debug","msg":"start initializing modules","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"startup","level":"debug","msg":"finished initializing modules","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"graphql_rebuild","level":"debug","msg":"rebuilding the graphql schema","schema":{"Objects":{"classes":[]}},"time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-03-15T05:43:29Z"}
weaviate-node-1-1  | {"action":"telemetry_push","level":"info","msg":"telemetry started","payload":"\u0026{MachineID:cd3080db-13b9-4357-8015-9fefbebf3de3 Type:INIT Version:1.24.2 Modules: NumObjects:0 OS:linux Arch:amd64}","time":"2024-03-15T05:43:30Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Stream connection from=10.12.226.153:60154","time":"2024-03-15T05:44:08Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Failed UDP ping: node2 (timeout reached)","time":"2024-03-15T05:44:10Z"}
weaviate-node-1-1  | {"level":"info","msg":" memberlist: Suspect node2 has failed, no acks received","time":"2024-03-15T05:44:11Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Failed UDP ping: node2 (timeout reached)","time":"2024-03-15T05:44:12Z"}
weaviate-node-1-1  | {"level":"info","msg":" memberlist: Suspect node2 has failed, no acks received","time":"2024-03-15T05:44:14Z"}
weaviate-node-1-1  | {"level":"info","msg":" memberlist: Marking node2 as failed, suspect timeout reached (0 peer confirmations)","time":"2024-03-15T05:44:15Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Failed UDP ping: node2 (timeout reached)","time":"2024-03-15T05:44:15Z"}
weaviate-node-1-1  | {"level":"info","msg":" memberlist: Suspect node2 has failed, no acks received","time":"2024-03-15T05:44:18Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Stream connection from=10.12.226.153:53962","time":"2024-03-15T05:44:40Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Stream connection from=10.12.226.153:34830","time":"2024-03-15T05:45:12Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Failed UDP ping: node2 (timeout reached)","time":"2024-03-15T05:45:14Z"}
weaviate-node-1-1  | {"level":"info","msg":" memberlist: Suspect node2 has failed, no acks received","time":"2024-03-15T05:45:18Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Failed UDP ping: node2 (timeout reached)","time":"2024-03-15T05:45:19Z"}
weaviate-node-1-1  | {"level":"info","msg":" memberlist: Marking node2 as failed, suspect timeout reached (0 peer confirmations)","time":"2024-03-15T05:45:22Z"}
weaviate-node-1-1  | {"level":"info","msg":" memberlist: Suspect node2 has failed, no acks received","time":"2024-03-15T05:45:24Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Stream connection from=10.12.226.153:32960","time":"2024-03-15T05:45:44Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Stream connection from=10.12.226.153:56040","time":"2024-03-15T05:46:16Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Failed UDP ping: node2 (timeout reached)","time":"2024-03-15T05:46:17Z"}
weaviate-node-1-1  | {"level":"info","msg":" memberlist: Suspect node2 has failed, no acks received","time":"2024-03-15T05:46:23Z"}
weaviate-node-1-1  | {"level":"debug","msg":" memberlist: Failed UDP ping: node2 (timeout reached)","time":"2024-03-15T05:46:24Z"}

The log from node 2:

[+] Running 2/1
 ✔ Network weaviate_testing_weaviate             Created                                                                                                    0.2s 
 ✔ Container weaviate_testing-weaviate-node-2-1  Created                                                                                                    0.0s 
Attaching to weaviate-node-2-1
weaviate-node-2-1  | {"action":"startup","level":"debug","msg":"created startup context, nothing done so far","startup_time_left":"59m59.997878536s","time":"2024-03-15T05:44:08Z"}
weaviate-node-2-1  | {"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-03-15T05:44:08Z"}
weaviate-node-2-1  | {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-03-15T05:44:08Z"}
weaviate-node-2-1  | {"action":"startup","level":"debug","msg":"config loaded","startup_time_left":"59m59.997113417s","time":"2024-03-15T05:44:08Z"}
weaviate-node-2-1  | {"action":"startup","level":"debug","msg":"configured OIDC and anonymous access client","startup_time_left":"59m59.997089937s","time":"2024-03-15T05:44:08Z"}
weaviate-node-2-1  | {"action":"startup","level":"debug","msg":"initialized schema","startup_time_left":"59m59.997072807s","time":"2024-03-15T05:44:08Z"}
weaviate-node-2-1  | {"level":"debug","msg":" memberlist: Initiating push/pull sync with:  10.12.226.225:7100","time":"2024-03-15T05:44:08Z"}
weaviate-node-2-1  | {"action":"startup","level":"debug","msg":"startup routine complete","time":"2024-03-15T05:44:08Z"}
weaviate-node-2-1  | {"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-03-15T05:44:08Z"}
weaviate-node-2-1  | {"action":"startup","level":"debug","msg":"start registering modules","time":"2024-03-15T05:44:08Z"}
weaviate-node-2-1  | {"action":"startup","level":"debug","msg":"completed registering modules","time":"2024-03-15T05:44:08Z"}
weaviate-node-2-1  | {"level":"debug","msg":" memberlist: Failed UDP ping: node1 (timeout reached)","time":"2024-03-15T05:44:09Z"}
weaviate-node-2-1  | {"level":"info","msg":" memberlist: Suspect node1 has failed, no acks received","time":"2024-03-15T05:44:10Z"}
weaviate-node-2-1  | {"level":"debug","msg":" memberlist: Failed UDP ping: node1 (timeout reached)","time":"2024-03-15T05:44:11Z"}
weaviate-node-2-1  | {"level":"info","msg":" memberlist: Suspect node1 has failed, no acks received","time":"2024-03-15T05:44:13Z"}
weaviate-node-2-1  | {"level":"debug","msg":" memberlist: Failed UDP ping: node1 (timeout reached)","time":"2024-03-15T05:44:13Z"}
weaviate-node-2-1  | {"level":"info","msg":" memberlist: Marking node1 as failed, suspect timeout reached (0 peer confirmations)","time":"2024-03-15T05:44:14Z"}
weaviate-node-2-1  | {"level":"info","msg":" memberlist: Suspect node1 has failed, no acks received","time":"2024-03-15T05:44:16Z"}
weaviate-node-2-1  | {"action":"startup","error":"could not load or initialize schema: sync schema with other nodes in the cluster: read schema: open transaction: broadcast open transaction: host \"172.31.0.2:7101\": send http request: Post \"http://172.31.0.2:7101/schema/transactions/\": context deadline exceeded","level":"fatal","msg":"could not initialize schema manager","time":"2024-03-15T05:44:38Z"}
weaviate-node-2-1 exited with code 0

And I have tried your suggestion here Can only Docker Compose be used for multi node deployment?, but still got same issues.

I have tested the communication between the two nodes and they can connect with each other through TCP and UDP.

Do you have any clue why this is happening and suggestion on how to resolve this? Thanks very much!

2

Hi @fenglizzz !! Welcome to our community :hugs:

It’s not a good idea to reference the CLUSTER_JOIN by IPs (or any container to container connection inside docker/k8s to be fair), as those will eventually change down the road, or maybe even after a reboot.

Here are some steps to get this party going. :partying_face:

First, let’s create a docker attachable network:

docker network create --attachable weaviate

Now, using the latest docker compose, and using this doc to adjust it to a multi node setup, a came up with those two files:

docker-compose-0.yaml

---
version: '3.4'
services:
  weaviate-0:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: cr.weaviate.io/semitechnologies/weaviate:1.24.3
    ports:
    - 8080:8080
    - 50051:50051
    volumes:
    - ./data-node0:/var/lib/weaviate
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      ENABLE_MODULES: 'text2vec-cohere,text2vec-huggingface,text2vec-palm,text2vec-openai,generative-openai,generative-cohere,generative-palm,ref2vec-centroid,reranker-cohere,qna-openai'
      CLUSTER_HOSTNAME: 'node0'
      CLUSTER_GOSSIP_BIND_PORT: '7100'
      CLUSTER_DATA_BIND_PORT: '7101'      
    networks:
      - weaviate

networks:
  weaviate:
    attachable: true
...

let’s first run it and check everything

docker compose -f docker-compose-0.yaml

and check if our founder node is up:

❯ curl localhost:8080/v1/nodes
{"nodes":[{"batchStats":{"queueLength":0,"ratePerSecond":0},"gitHash":"7e5d3aa","name":"node0","shards":null,"status":"HEALTHY","version":"1.24.3"}]}

now, let’s spin up the second node, node1 :slight_smile:

here is the docker-compose-1.yaml

---
version: '3.4'
services:
  weaviate-1:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: cr.weaviate.io/semitechnologies/weaviate:1.24.3
    ports:
    - 8081:8080
    - 50052:50051
    volumes:
    - ./data-node1:/var/lib/weaviate
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      ENABLE_MODULES: 'text2vec-cohere,text2vec-huggingface,text2vec-palm,text2vec-openai,generative-openai,generative-cohere,generative-palm,ref2vec-centroid,reranker-cohere,qna-openai'
      CLUSTER_HOSTNAME: 'node1'
      CLUSTER_GOSSIP_BIND_PORT: '7102'
      CLUSTER_DATA_BIND_PORT: '7103'
      CLUSTER_JOIN: 'weaviate-0:7100'
    networks:
      - weaviate

networks:
  weaviate:
    attachable: true
...

The only differences:

  • Service name
  • Port mappings (so they don’t conflict)
  • Folder for mounting data
  • CLUSTER_HOSTNAME
  • Node1 has CLUSTER_JOIN pointing to the reachable hostname of Node0

Now, you can spin the second node:

docker compose -f docker-compose-0.yaml up -d

And now check the status of your cluster:

❯ curl localhost:8080/v1/nodes
{
   "nodes":[
      {
         "batchStats":{
            "queueLength":0,
            "ratePerSecond":0
         },
         "gitHash":"7e5d3aa",
         "name":"node0",
         "shards":null,
         "status":"HEALTHY",
         "version":"1.24.3"
      },
      {
         "batchStats":{
            "queueLength":0,
            "ratePerSecond":0
         },
         "gitHash":"7e5d3aa",
         "name":"node1",
         "shards":null,
         "status":"HEALTHY",
         "version":"1.24.3"
      }
   ]
}

Let me know if this helps!

And don’t forget to check out events page!

3

HI thanks very much for the detailed steps. I followed exactly as what you said here. But when I run on the second node. There is one error with"action":“cluster_attempt_join”,“error”:“lookup weaviate-0 on 127.0.0.11:53: no such host”,“level”:“warning”,“msg”:"specified hostname to join cluster cannot be resolved. The log on the second node is shown as following.

{"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-03-18T01:33:51Z"}
{"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-03-18T01:33:51Z"}
{"action":"cluster_attempt_join","error":"lookup weaviate-0 on 127.0.0.11:53: no such host","level":"warning","msg":"specified hostname to join cluster cannot be resolved. This is fineif this is the first node of a new cluster, but problematic otherwise.","remote_hostname":"weaviate-0:7100","time":"2024-03-18T01:33:51Z"}
{"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-03-18T01:33:51Z"}
{"level":"warning","msg":"Multiple vector spaces are present, GraphQL Explore and REST API list objects endpoint module include params has been disabled as a result.","time":"2024-03-18T01:33:51Z"}
{"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-03-18T01:33:51Z"}
{"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-03-18T01:33:51Z"}
{"action":"telemetry_push","level":"info","msg":"telemetry started","payload":"\u0026{MachineID:fbfc895b-e0b2-4d44-b191-dfb5965fbdea Type:INIT Version:1.24.3 Modules:generative-cohere,generative-openai,generative-palm,qna-openai,ref2vec-centroid,reranker-cohere,text2vec-cohere,text2vec-huggingface,text2vec-openai,text2vec-palm NumObjects:0 OS:linux Arch:amd64}","time":"2024-03-18T01:33:52Z"}
4

since these are running on to different nodes. How does it know which node that weaviate-0 is running on if we just use the ‘weaviate-0’ instead of IP please?

5

Hi @fenglizzz !

it will infer from the service name.

Every container in a docker network will use it’s service name as the hostname, as well as projectname-container-[1-NN]

project name, if not defined, will be the folder where it runs.

so for example, if you run those docker compose in a folder called feng, the second hostname you could use are:

  • feng-weaviate-0-1
  • feng-weaviate-1-1

Before running those, can completely remove that stack?

you can do it with:
WARNING: this command removes everything from the stack, including docker volumes. As this docker compose volumes are binded to a folder, it will not delete it.

docker compose down -v

Let me know if this works.

THanks!

6

HI thanks very much for the response. I am running under the folder weaviate_testing. The first node is started as:

[+] Running 2/1
 ✔ Network weaviate_testing_weaviate        Created                                                                                                         0.2s 
 ✔ Container weaviate_testing-weaviate-0-1  Created                                                                                                         0.0s 
Attaching to weaviate-0-1
weaviate-0-1  | {"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-03-18T13:54:12Z"}
weaviate-0-1  | {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-03-18T13:54:12Z"}
weaviate-0-1  | {"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-03-18T13:54:12Z"}
weaviate-0-1  | {"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-03-18T13:54:12Z"}
weaviate-0-1  | {"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-03-18T13:54:12Z"}
weaviate-0-1  | {"action":"telemetry_push","level":"info","msg":"telemetry started","payload":"\u0026{MachineID:e140af08-5a21-45dc-a643-99c87d0865a6 Type:INIT Version:1.24.3 Modules: NumObjects:0 OS:linux Arch:amd64}","time":"2024-03-18T13:54:13Z"}

so the hostname should be ‘weaviate_testing-weaviate-0-1’ right? Then in the second node’s compose file I set

 CLUSTER_JOIN: 'weaviate_testing-weaviate-0-1:7100'

but after run compose up, I still got the following error:

[+] Running 2/1
 ✔ Network weaviate_testing_weaviate        Created                                                                                                         0.2s 
 ✔ Container weaviate_testing-weaviate-1-1  Created                                                                                                         0.0s 
Attaching to weaviate-1-1
weaviate-1-1  | {"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-03-18T13:55:06Z"}
weaviate-1-1  | {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-03-18T13:55:06Z"}
weaviate-1-1  | {"action":"cluster_attempt_join","error":"lookup weaviate_testing-weaviate-0-1 on 127.0.0.11:53: no such host","level":"warning","msg":"specified hostname to join cluster cannot be resolved. This is fineif this is the first node of a new cluster, but problematic otherwise.","remote_hostname":"weaviate_testing-weaviate-0-1:7100","time":"2024-03-18T13:55:06Z"}
weaviate-1-1  | {"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-03-18T13:55:06Z"}
weaviate-1-1  | {"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-03-18T13:55:06Z"}
weaviate-1-1  | {"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-03-18T13:55:06Z"}
weaviate-1-1  | {"action":"telemetry_push","level":"info","msg":"telemetry started","payload":"\u0026{MachineID:e43c6368-fdfb-4006-bebb-61d907771ccd Type:INIT Version:1.24.3 Modules: NumObjects:0 OS:linux Arch:amd64}","time":"2024-03-18T13:55:07Z"}
7

After spinning the first node, can you read the node status?

curl localhost:8080/v1/nodes

one thing to try is to clearing everything (including the network) and start over:

docker compose -f docker-compose-0.yaml -f docker-compose-1.yaml down -v
docker network rm weaviate
rm -rf data-node0 data-node1

Now, create again

docker network create --attachable weaviate
docker compose -f docker-compose-0.yaml -f docker-compose-1.yaml up -d

This is how you can check the connectivity from one node to another:

docker compose -f docker-compose-0.yaml exec -ti weaviate-0 wget -qO - http://weaviate-1:8080/v1/nodes

You are running a wget from within the container weaviate-o, pointing to weaviate-1.

Let me know if this helps, and feel free to ping me on our lack if you need further assistance.

THanks!

8

@DudaNogueira,

If I’m not wrong, you are sharing the steps to spawn multiple weaviate nodes in one and the same physical machines, attach them to a common docker network and establish the cluster network.

But the issue lies, say, in spawning three weaviate nodes in three physical machines (one node per machine) and then trying to establish the cluster network. Thats where we face the following issue:

{“action”:“cluster_attempt_join”,“error”:“lookup weaviate_testing-weaviate-0-1 on 127.0.0.11:53: no such host”,“level”:“warning”,“msg”:“specified hostname to join cluster cannot be resolved. This is fine if this is the first node of a new cluster, but problematic otherwise.”,“remote_hostname”:“weaviate_testing-weaviate-0-1:7100”,“time”:“2024-03-18T13:55:06Z”}

Requesting your assistance in this regard.

9

Hi @Ashok_Raja ! Welcome to our community :hugs:

Oh… That’s indeed different.
Sorry @fenglizzz !! I totally missed the “two machines” on the beginning :see_no_evil:

That’s definitely some connectivity issues between this machines.

And yes, on that case the IP address will be needed. Sorry again @fenglizzz :laughing:

one thing you could do to test the connectivity:

docker compose exec -ti weaviate sh
apk add busybox-extras
telnet ip-of-the-other-machine:7100

Let me know if this helps.

10

Hello, I’m currently facing the same issue. Have you managed to resolve it