Multi-tenancy collection with different embedding models?

Support
1

Description

Currently our customers can have multiple collections with vectors created by different embedding models (e.g. “text-embedding-ada-002” or “text-embedding-3-large”). We tried to replace those collections with a single multi-tenancy collection and did not consider the differences in vector dimensions. When trying to migrate the old data to the new collection we then got errors like “new node has a vector with length 3072. Existing nodes have vectors with length 1536: vector dimensions do not match the index dimensions”. I guess there is no way to get this working in a single collection and we would need to have 1 multi-tenancy collection per embedding model. Would that be the best way to solve this?

Server Setup Information

  • Weaviate Server Version: 1.32.2
  • Deployment Method: Kubernetes/Docker
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: Python 3.10
  • Multitenancy?: Yes

Any additional Information

2

hi @torbenw !

Indeed, you cannot have one collection and each tenant choose their own embedding model.

All tenants must have the same schema and vector configuration. This means, that all properties and named vectors and their models must be exactly the same for each tenant.

So in your scenario, having one collection per model can be a workaround.

Let me know if this clarifies!

Thanks!

3

You could also use two named vectors: Collection definition | Weaviate Documentation

1 Like
4

Name vectors sound like a good idea. Is there a way to dynamically add new named vector configs?

5

Hi!

Since 1.31 you can add new named vectors to already created collections:

1 Like