Weaviate clusters deployment recommendation

General
1

Hello @DudaNogueira

We would require recommendation on weaviate k8s deployment for our use cases.

If there are 10+ teams using the weaviate cluster, as we don’t have RBAC all the data will be shared by all the teams.

  1. Can we have multiple weaviate clusters because data inserted by team A will not be used by team B ? Which is recommended multiple weaviate clusters for multiple clients or single weaviate cluster for multiple clients ?
  2. Can we install the modules separately than the weaviate cluster installation so that same modules can be reused for all the weaviate clusters ?
  3. If we provision multiple weaviate clusters, then the data is isolated b/w teams and there won’t be any performance issues(noisy neighbour issues :slight_smile: )
  4. I am aware of Resource constraints, with multiple weaviate cluster we would require more resources, but it can be designed as per the team’s requirement.
    For eg : team A use case is to store 50GB data only - then we will provision a small size weaviate cluster and if team B use case is to store 500GB, we will provide large size weaviate cluster.

Regards,
Adithya

2

hi @adithya.ch !!

1 - This will depend on your usage. But if you want total isolation between teams, considering we do not have RBAC as of now, multiple weaviate clusters will be the go to solution.

2 - No, the modules are bundled inside Weaviate. You can have a same model that is used by multiple Weaviate servers modules. So for example, you could have text2vec-transformers containers running that could serve multiple weaviate servers.

3 - That’s right, considering they are on different machines. If two or more servers are allocated at the same machine, it’s advised to limit the resource usage of a Server, so it doesn’t interfere with each other.

4 - Sound like a plan! Right now our team is working on delivering RAFT and some other improvements on cluster resilience. We do have RBAC feature request listed in our roadmap. I will make sure to advocate this internally so it gets into the planned phase!!

Let me know if this clarifies.

Thanks!

3

Thank you for the details @DudaNogueira

  1. you could have text2vec-transformers containers running that could serve multiple weaviate servers.

If let’s say i am runnign weaviate-cluster1 and weaviate-cluster2 , can they share same module ? Can you share some example how to use it ? text2vec-transformers module have to be enabled on both cluster configuration right which means multiple pods will be provided for both the clusters ?

Regards,
Adithya