There is now what looks like a different issue, caused by the same process of replacing one pod out of the cluster.
Still same setup: 3 replicas, collection has a replication factor of 3 and async replication enabled. When pod weaviate-1 got replaced, it now has this message in logs:
{"action":"async_replication","build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","class_name":"AndriiTest","hashbeat_iteration":51,"level":"warning","msg":"hashbeat iteration failed: collecting differences: \"10.0.198.209:7001\": status code: 401, error: ","shard_name":"554JMVbS6e2m","time":"2024-09-05T17:25:51Z"}
When I try to call v1/cluster/statistics endpoint I am getting this error back:
{
"error": [
{
"message": "node: weaviate-1: unexpected status code 401 ()"
}
]
}
Restarting the ReplicaSet helps, but it seems to not be able to recover on its own