Using Multiple Nodes with Tenancy

Support
1

Description

How do I know my setup is using multiple nodes with tenancy or how would I verify this state? I’ve been running a 2 node cluster playing around with the setup by loading data. However, I’m not what I would think would be the right things to ensure that this setup is working and wanted some guidance.

When I hit the endpoint /v1/nodes/<MY_COLLECTION_NAME>

{"nodes":[{"batchStats":{"queueLength":0,"ratePerSecond":3},"gitHash":"1ea5766","name":"node0","shards":null,"status":"HEALTHY","version":"1.25.7"},{"batchStats":{"queueLength":0,"ratePerSecond":0},"gitHash":"1ea5766","name":"node1","shards":null,"status":"HEALTHY","version":"1.25.7"}]}

I have 2 tenants loaded up at the moment so I’d expected to see at least a value for shards for node0. The only logs I see that confirms the nodes are connected is debug logs:

"msg":" memberlist: Initiating push/pull sync with: node0

These logs are just repeating itself.

However, I’m not clear if these nodes are being used to store tenants individually or if I need to change my setup in some way to search across multiple ones. I also am not near to filling up a single node. The only thing I’ve seen is when I was developing this I would get error messages if my node1 wasn’t reachable.

Any advice on this? I just want to prove that multiple nodes will be used loading data for multiple tenants.

Server Setup Information

  • Weaviate Server Version: 1.25.7
  • Deployment Method: Docker on AWS ECS
  • Multi Node? Number of Running Nodes: 2
  • Client Language and Version: 4.6.5
  • Multitenancy?: Yes.
2

hi @dhanshew72 !!

Sorry for the delay here.

Missed this one :frowning:

Were you able to figure this out?

Weaviate should distribute the tenants across different available nodes.

Here is a test I did using latest 1.26.1 version with 3 nodes:

from weaviate import classes as wvc
client.collections.delete("MyMTCollection")
collection = client.collections.create(
    "MyMTCollection",
    multi_tenancy_config=wvc.config.Configure.multi_tenancy(enabled=True, auto_tenant_activation=True, auto_tenant_creation=True),
    vectorizer_config=wvc.config.Configure.Vectorizer.none()
)

now let’s check we have 0 shards yet:

for node in client.cluster.nodes(output="verbose"):
    print(node.name, len(node.shards))

outputs:

weaviate-0 0
weaviate-1 0
weaviate-2 0

Now let’s add 100 tenants with some sample data

for i in range(100):
    tenant_data = f"T{i}"
    collection.with_tenant(tenant_data).data.insert({"text": tenant_data})

This is how it is distributed (it varies for every run) after adding the tenants and content:

weaviate-0 40
weaviate-1 28
weaviate-2 32

Let me know if that helps.

THanks!

3

Apologies, I should have reported back. I got this working now.

1 Like