Single Weaviate unable to connect

Support
1

Description

I have weaviate running in an VM but i need another weaviate for Production.Created snapsot and spiined a VM after spiing containers are getting spinned but my old data is present but when i am querying from post i am getting 404 and when i hit http://:8080/v1/.well-known/ready i am getting 503 below is the log error

Server Setup Information

  • Weaviate Server Version:1.26.1
  • Deployment Method: Docker-compose.yml
  • Multi Node? Number of Running Nodes: Single node
  • Client Language and Version:
  • Multitenancy?:
    Below is my Compose file
version: '3.4'

services:
  weaviate:
    image: cr.weaviate.io/semitechnologies/weaviate:1.26.1
    command:
      - --host
      - 0.0.0.0
      - --port
      - '8080'
      - --scheme
      - http
    environment:
      LOG_LEVEL: trace
      QUERY_SLOW_LOG_ENABLED: 'true'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
    ports:
      - 8080:8080
      - 50051:50051
    volumes:
      - weaviate_data:/var/lib/weaviate
    restart: on-failure:0

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus:/etc/prometheus/
      - ./data/prometheus:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - 9090:9090

  grafana:
    image: grafana/grafana
    ports:
      - 3000:3000
    volumes:
      - ./grafana/grafana.ini:/etc/grafana/grafana.ini
      - ./grafana/datasource.yml:/etc/grafana/provisioning/datasources/prometheus.yml
      - ./grafana/dashboard_provider.yml:/etc/grafana/provisioning/dashboards/dashboards.yml
      - ./grafana/dashboards:/var/lib/grafana/dashboards
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

volumes:
  weaviate_data:

Any additional Information

{“action”:“bootstrap”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:16Z”}
{“level”:“warning”,“msg”:“raft heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election”,“time”:“2024-11-15T11:30:16Z”}
{“action”:“inverted filter2search migration”,“level”:“debug”,“msg”:“migration skip flag set, skipping migration”,“time”:“2024-11-15T11:30:16Z”}
{“action”:“inverted filter2search migration”,“level”:“debug”,“msg”:“starting switching fallback mode”,“time”:“2024-11-15T11:30:16Z”}
{“action”:“inverted filter2search migration”,“level”:“debug”,“msg”:“no missing filterable indexes, fallback mode skipped”,“time”:“2024-11-15T11:30:16Z”}
{“docker_image_tag”:“1.26.1”,“level”:“info”,“msg”:“configured versions”,“server_version”:“1.26.1”,“time”:“2024-11-15T11:30:16Z”}
{“action”:“grpc_startup”,“level”:“info”,“msg”:“grpc server listening at [::]:50051”,“time”:“2024-11-15T11:30:16Z”}
{“action”:“restapi_management”,“docker_image_tag”:“1.26.1”,“level”:“info”,“msg”:“Serving weaviate at http://[::]:8080”,“time”:“2024-11-15T11:30:16Z”}
{“action”:“telemetry_push”,“level”:“info”,“msg”:“telemetry started”,“payload”:“\u0026{MachineID:cbf6157a-efb6-46aa-b51b-1cb2af875756 Type:INIT Version:1.26.1 NumObjects:0 OS:linux Arch:amd64 UsedModules:}”,“time”:“2024-11-15T11:30:16Z”}
{“action”:“bootstrap”,“error”:“could not join a cluster from [172.21.0.4:8300]”,“level”:“warning”,“msg”:“failed to join cluster, will notify next if voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:17Z”,“voter”:true}
{“action”:“bootstrap”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:17Z”}
{“action”:“bootstrap”,“error”:“could not join a cluster from [172.21.0.4:8300]”,“level”:“warning”,“msg”:“failed to join cluster, will notify next if voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:19Z”,“voter”:true}
{“action”:“bootstrap”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:19Z”}
{“action”:“bootstrap”,“error”:“could not join a cluster from [172.21.0.4:8300]”,“level”:“warning”,“msg”:“failed to join cluster, will notify next if voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:21Z”,“voter”:true}
{“action”:“bootstrap”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:21Z”}
{“action”:“bootstrap”,“error”:“could not join a cluster from [172.21.0.4:8300]”,“level”:“warning”,“msg”:“failed to join cluster, will notify next if voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:22Z”,“voter”:true}
{“action”:“bootstrap”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:22Z”}
{“action”:“bootstrap”,“error”:“could not join a cluster from [172.21.0.4:8300]”,“level”:“warning”,“msg”:“failed to join cluster, will notify next if voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:24Z”,“voter”:true}
{“action”:“bootstrap”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:24Z”}
{“action”:“bootstrap”,“error”:“could not join a cluster from [172.21.0.4:8300]”,“level”:“warning”,“msg”:“failed to join cluster, will notify next if voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:26Z”,“voter”:true}
{“action”:“bootstrap”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:26Z”}
{“action”:“bootstrap”,“error”:“could not join a cluster from [172.21.0.4:8300]”,“level”:“warning”,“msg”:“failed to join cluster, will notify next if voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:27Z”,“voter”:true}
{“action”:“bootstrap”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:27Z”}
{“action”:“bootstrap”,“error”:“could not join a cluster from [172.21.0.4:8300]”,“level”:“warning”,“msg”:“failed to join cluster, will notify next if voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:29Z”,“voter”:true}
{“action”:“bootstrap”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:29Z”}
{“action”:“bootstrap”,“error”:“could not join a cluster from [172.21.0.4:8300]”,“level”:“warning”,“msg”:“failed to join cluster, will notify next if voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:31Z”,“voter”:true}
{“action”:“bootstrap”,“level”:“info”,“msg”:“notified peers this node is ready to join as voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:31Z”}
{“action”:“restapi_request”,“level”:“debug”,“method”:“GET”,“msg”:“received HTTP request”,“time”:“2024-11-15T11:30:32Z”,“url”:{“Scheme”:“”,“Opaque”:“”,“User”:null,“Host”:“”,“Path”:“/metrics”,“RawPath”:“”,“OmitHost”:false,“ForceQuery”:false,“RawQuery”:“”,“Fragment”:“”,“RawFragment”:“”}}
{“action”:“bootstrap”,“error”:“could not join a cluster from [172.21.0.4:8300]”,“level”:“warning”,“msg”:“failed to join cluster, will notify next if voter”,“servers”:[“172.21.0.4:8300”],“time”:“2024-11-15T11:30:32Z”,“voter”:true}

2

hi @gadey_karthik !!

Not sure I understood the sequence you described.

But consider that you can run two weaviates at the same VM (running on different ports) or you can run two separate VMs, each one with different IPs.

The logs you shared indicate that this node is trying to join a non existent cluster :thinking:

Do you see the same output if using 1.26.latest or 1.27.latest?