I like the idea of isolating data, but each tenant is treated as a shard which becomes difficult if a single tenants data starts growing significantly because it’s isolated to a single node. This makes it difficult to horizontally scale a cluster with multi-tenancy as I’m capped to resources on a node. The only workaround I’m aware of is to swap to filtering to operate the same way, but requires reloading all the data.
Is sharding for tenants in the pipeline or should I look towards swapping towards filtering instead of tenants?
Hey @dhanshew72,
You are correct in your understanding, in Weaviate’s current multi-tenancy implementation, each tenant is isolated to a single shard, and that shard resides on a single node. This means that if a tenant’s data grows significantly, it is limited by the resources of that node, and Weaviate does not currently support distributing a single tenant’s data across multiple nodes (i.e., sharding within a tenant).
I would add this as feature request - go to our github and submit a feature request with that please:
Best regards,
Mohamed Shahin
Weaviate Support Engineer
(Ireland, GMT/UTC timezone)
