Hi! sorry for the radio silence on this. We dug back into the problem and identified the root issue relating to the interactions between:
- Version of Kubernetes we were running
- The
musl
DNS resolver used in the Alpine base image of weaviate - The lack of FQDNs used in the weaviate helm chart for inter-node communication.
I’ve created a github issue here to discuss possible fixes: Intra-cluster hostnames need to be FQDNs for musl to resolve DNS in all configurations · Issue #175 · weaviate/weaviate-helm · GitHub