How do I scale in/out statefulset in weaviate

Support
1

Description

Hello,

Our weaviate deployment was accidentally deployed to system nodepool before taints were put onto it. So we have replica set to 5 and end up with 2 pods on system nodepool and 3 on user node pool. We tried to put taint on system nodepool and evict weaviate but it will not budge since weaviate-0 pod was scheduled on system nodepool. What are out options of running just 3 replicas on user nodepools?

Server Setup Information

  • Weaviate Server Version:
  • Deployment Method: k8s
  • Multi Node? Number of Running Nodes: 5
  • Client Language and Version:
  • Multitenancy?:

Any additional Information

2

hi @artisticcheese !!

Welcome to our community :hugs: !!

Looks like this question is more about k8s than Weaviate :thinking:

From Weaviate side, as long as each node can communicate with the other, it should be fine.

Also, while moving your pods around, the content of the PERSISTENCE_DATA_PATH must be the same from the nodeX.

So for example, if you set replica to 5, you will have, for eg. weaviate-data-0 all the way to weaviate-4 and each node ant its configuration need to be attached to the corresponding pvc.

Now, if you already have collections with replica factor 5, and want to now use replica facto 3 instead, you will need to migrate that collection to a new collection or cluster, considering that we do not have the dynamic scaling yet.

Let me know if that helps.

Thanks!