Facing cluster formation between pods of running weaviate using kubeadm on EC2

@DudaNogueira,
I have deployed Weaviate using Kubeadm on a cluster hosted on an EC2 instance. The setup consists of 1 master node and 1 worker node. On the worker node, I deployed 2 pods using a StatefulSet deployment. Both pods are running and accessible, but they are not forming a cluster. Additionally, I want to replicate data across the pods. Is it possible to achieve this with a StatefulSet deployment, where each pod has its own storage?

Regarding the cluster formation issue, I’m unsure what the problem might be. Below are the StatefulSet and Service files I used for this deployment.

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: weaviate
namespace: weaviate-v3
spec:
serviceName: weaviate-headless
replicas: 2
selector:
matchLabels:
app: weaviate
template:
metadata:
labels:
app: weaviate
spec:
containers:
- name: weaviate
image: cr.weaviate.io/semitechnologies/weaviate:1.28.2
ports:
- containerPort: 8080
name: http
- containerPort: 50051
name: grpc
- containerPort: 7946
name: gossip
- containerPort: 7947
name: data
command:
- /bin/weaviate
args:
- --host
- “0.0.0.0”
- --port
- “8080”
- --scheme
- “http”
env:
- name: QUERY_DEFAULTS_LIMIT
value: “25”
- name: AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED
value: “true”
- name: PERSISTENCE_DATA_PATH
value: “/var/lib/weaviate”
- name: DEFAULT_VECTORIZER_MODULE
value: “none”
- name: ENABLE_MODULES
value: “”
- name: CLUSTER_HOSTNAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: CLUSTER_GOSSIP_PEERS
value: “weaviate-0.weaviate-headless.weaviate-v3.svc.cluster.local,weaviate-1.weaviate-headless.weaviate-v3.svc.cluster.local”
- name: CLUSTER_JOIN
value: “true”
- name: RAFT_JOIN
value: “true”
- name: CLUSTER_GOSSIP_BIND_PORT
value: “7946”
- name: CLUSTER_GOSSIP_ADVERTISE_PORT
value: “7946”
- name: CLUSTER_DATA_BIND_PORT
value: “7947”
- name: CLUSTER_DATA_ADVERTISE_PORT
value: “7947”
volumeMounts:
- name: data
mountPath: /var/lib/weaviate
resources:
requests:
memory: “512Mi”
cpu: “200m”
limits:
memory: “1Gi”
cpu: “500m”
volumeClaimTemplates:

  • metadata:
    name: data
    spec:
    accessModes: [ “ReadWriteOnce” ]
    storageClassName: ebs-gp3
    resources:
    requests:
    storage: 2Gi

apiVersion: v1
kind: Service
metadata:
name: weaviate-headless
namespace: weaviate-v3
spec:
clusterIP: None
selector:
app: weaviate
ports:
- port: 8080
targetPort: 8080
name: http
- port: 50051
targetPort: 50051
name: grpc
- port: 8300
targetPort: 8300
name: raft
- port: 7946
targetPort: 7946
name: gossip
- port: 7947
targetPort: 7947
name: data

Can you tell me where I am going wrong or the approach I am taking is wrong?

I am also facing same issue

Hello @VINCENT_NADAR, @akshay_madiwale,

Welcome to our community, it’s great to have you here!

I’m not sure if you’ve managed to resolve the issue yet, but I haven’t personally tested this setup as you are trying to achieve. However, based on our documentation:

Those steps are essential for multinode:

  1. Configure one node as a “founding” member
  2. Set CLUSTER_JOIN for the other nodes in the cluster
  3. Define CLUSTER_GOSSIP_BIND_PORT and CLUSTER_DATA_BIND_PORT for each node
  4. Set RAFT_JOIN and RAFT_BOOTSTRAP_EXPECT to specify the number of voters
  5. Optionally, use CLUSTER_HOSTNAME to define node names

Example for the founding member:

CLUSTER_JOIN: 'weaviate-node-1:7100'  # This must be the service name of the founding member node.

If you haven’t already, try running through these steps again to ensure everything is set up correctly.

Regards,
Mohamed Shahin
Weaviate Support