Cluster nodes not aware of eachother

Hi! sorry for the radio silence on this. We dug back into the problem and identified the root issue relating to the interactions between:

  • Version of Kubernetes we were running
  • The musl DNS resolver used in the Alpine base image of weaviate
  • The lack of FQDNs used in the weaviate helm chart for inter-node communication.

I’ve created a github issue here to discuss possible fixes: Intra-cluster hostnames need to be FQDNs for musl to resolve DNS in all configurations · Issue #175 · weaviate/weaviate-helm · GitHub