Cannot Add Node to Cluster on AWS ECS

Support
1

Description

I’m attempting to add an extra node onto an ECS cluster using Weaviate and I keep getting the following error message:

received invalid msgType (72), expected pushPullMsg (6) 
memberlist join not successful

I’m trying to connect via a load balancer if that changes anything.

Any ideas on what might be causing this?

Server Setup Information

  • Weaviate Server Version: 1.24.13
  • Deployment Method:Docker on ECS
  • Multi Node? Number of Running Nodes: Attempting to do 2
  • Client Language and Version: 4.6.1
  • Multitenancy?: Yes

Any additional Information

I’m using AWS ECS, so I don’t have the luxury of doing service to service connection like in Kubernetes. Should I be using something like service discovery for this?

Any ideas/thoughts would be appreciated.

2

hi @dhanshew72 !!

Can you get more logs on this for context?

Also, you should know that from 1.24 to 1.25 we have some important changes for multi node setups.

Considering that if not a multi tenant use case, adding a new node to your cluster will not be enough: you would need to reindex your collection to a new one with the desired replica factor.

As you have a multi tenant, adding a new node will get new tenants with the node is ready.

Can you replicate this on a non ECS deployment?

Thanks!

3

Do you have a doc on multi-tenant with multi-node setups? How many would I need to support this? FYI: I’m doing a POC so I’m not locked into migrating data quite yet.

4

Hi!

For multi node, the preferred way is deploying thru our helm chart in k8s:

Now, both single and multi node can handle multi tenants.

Here is the doc for multi tenant operation:

And here a recipe on how to play around with MT:

Let me know if this helps.

Thanks!

5

This helped, I’m using Langchain so I dug in to figure out how to apply in on that end.

My main question, can you shard an individual tenant?

6

Hi!

It is already sharded. All tenants have it’s own shard.

from the docs:

Multi-tenancy provides data isolation. Each tenant is stored on a separate shard. Data stored in one tenant is not visible to another tenant. If your application serves many different users, multi-tenancy keeps their data private and makes database operations more efficient.

:wink:

7

My apologies, to clarify:
Can I split a tenant into 2 shards for a possible setup? Or is it better to use something like replication?

8

Found a solution to this. Using service discovery, you still have to open the ports and attach a security group to get this error to go away.

Basically, any port you have configured in environment CLUSTER_GOSSIP_BIND_PORT, CLUSTER_DATA_BIND_PORT needs to be opened in the security group.

1 Like
9

Nice!

Thanks for sharing.