Docker Compose with multi node deployment issue

Support
2

Hi @fenglizzz !! Welcome to our community :hugs:

It’s not a good idea to reference the CLUSTER_JOIN by IPs (or any container to container connection inside docker/k8s to be fair), as those will eventually change down the road, or maybe even after a reboot.

Here are some steps to get this party going. :partying_face:

First, let’s create a docker attachable network:

docker network create --attachable weaviate

Now, using the latest docker compose, and using this doc to adjust it to a multi node setup, a came up with those two files:

docker-compose-0.yaml

---
version: '3.4'
services:
  weaviate-0:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: cr.weaviate.io/semitechnologies/weaviate:1.24.3
    ports:
    - 8080:8080
    - 50051:50051
    volumes:
    - ./data-node0:/var/lib/weaviate
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      ENABLE_MODULES: 'text2vec-cohere,text2vec-huggingface,text2vec-palm,text2vec-openai,generative-openai,generative-cohere,generative-palm,ref2vec-centroid,reranker-cohere,qna-openai'
      CLUSTER_HOSTNAME: 'node0'
      CLUSTER_GOSSIP_BIND_PORT: '7100'
      CLUSTER_DATA_BIND_PORT: '7101'      
    networks:
      - weaviate

networks:
  weaviate:
    attachable: true
...

let’s first run it and check everything

docker compose -f docker-compose-0.yaml

and check if our founder node is up:

❯ curl localhost:8080/v1/nodes
{"nodes":[{"batchStats":{"queueLength":0,"ratePerSecond":0},"gitHash":"7e5d3aa","name":"node0","shards":null,"status":"HEALTHY","version":"1.24.3"}]}

now, let’s spin up the second node, node1 :slight_smile:

here is the docker-compose-1.yaml

---
version: '3.4'
services:
  weaviate-1:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: cr.weaviate.io/semitechnologies/weaviate:1.24.3
    ports:
    - 8081:8080
    - 50052:50051
    volumes:
    - ./data-node1:/var/lib/weaviate
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      ENABLE_MODULES: 'text2vec-cohere,text2vec-huggingface,text2vec-palm,text2vec-openai,generative-openai,generative-cohere,generative-palm,ref2vec-centroid,reranker-cohere,qna-openai'
      CLUSTER_HOSTNAME: 'node1'
      CLUSTER_GOSSIP_BIND_PORT: '7102'
      CLUSTER_DATA_BIND_PORT: '7103'
      CLUSTER_JOIN: 'weaviate-0:7100'
    networks:
      - weaviate

networks:
  weaviate:
    attachable: true
...

The only differences:

  • Service name
  • Port mappings (so they don’t conflict)
  • Folder for mounting data
  • CLUSTER_HOSTNAME
  • Node1 has CLUSTER_JOIN pointing to the reachable hostname of Node0

Now, you can spin the second node:

docker compose -f docker-compose-0.yaml up -d

And now check the status of your cluster:

❯ curl localhost:8080/v1/nodes
{
   "nodes":[
      {
         "batchStats":{
            "queueLength":0,
            "ratePerSecond":0
         },
         "gitHash":"7e5d3aa",
         "name":"node0",
         "shards":null,
         "status":"HEALTHY",
         "version":"1.24.3"
      },
      {
         "batchStats":{
            "queueLength":0,
            "ratePerSecond":0
         },
         "gitHash":"7e5d3aa",
         "name":"node1",
         "shards":null,
         "status":"HEALTHY",
         "version":"1.24.3"
      }
   ]
}

Let me know if this helps!

And don’t forget to check out events page!