I had couple of questions regarding using k8s with weaviate.
What are the key differences between Weaviate’s native orchestration algorithm and using Kubernetes for orchestration, and where exactly do these differences manifest? Specifically, given that Kubernetes offers superior container scheduling, resource management, failure recovery, and dynamic node scaling capabilities, how does Weaviate’s built-in orchestration compare in these areas, and what are the trade-offs between using each approach?
What is the difference between Load balancing and service discovery capabilities in weaviate vs using k8s in weaviate ?
Those are different stacks. Weaviate will manage the orchestration (or better, the node communication consistency and routing) on a data level, while k8s will take care of nodes and pods running.
Let’s say you have a k8s cluster and you are running 3 weaviate instances. Once k8s has the 3 Weaviate instances up and running, each weaviate instance will then communicate between each other, and using a raft consensus algorithm, elect a leader that will coordinate all comms.
That’s correct. From an “internal perspective” there is a raft election, and your Weaviate pods will decide together who is the leader that will coordinate its operations (replication, deletion, consistency, etc)
From K8s perspective, you will balance the load to all available your pods thru a k8s service. This will spread the incoming load to all, allowing your cluster to have not only more resilience, but also handling bigger loads.
Here we have a nice conceptual docs that will help you on that, covering also the consistency level of yours queries:
