# Whole cluster hangups on pod termination

**URL:** <https://forum.weaviate.io/t/whole-cluster-hangups-on-pod-termination/4197>\
**Category:** Support\
**Tags:** bug\
**Created:** [September 18, 2024, 9:14pm UTC](https://forum.weaviate.io/t/whole-cluster-hangups-on-pod-termination/4197 "2024-09-18T21:14:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![wvuser](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/wvuser/32/2371_2.png) [@wvuser](https://forum.weaviate.io/u/wvuser)\
**Post date:** [September 18, 2024, 9:14pm UTC](https://forum.weaviate.io/t/whole-cluster-hangups-on-pod-termination/4197/1 "2024-09-18T21:14:21Z")

</div>

### Description

When trying to remove/restart any pod, whole cluster hangups - requests “fall off” by timeout (60sec).  
API’s “ **/v1/nodes?output=verbose** ” requests freeze too.  
Pods forcelly terminated by k8s after 10min waiting (_ **terminationGracePeriodSeconds: 600** _).  
There were no such errors in versions 1.23/24/25.

### Server Setup Information

- Weaviate Server Version: 1.26.4
- Deployment Method: k8s
- Multi Node? Number of Running Nodes: yes, 3
- Client Language and Version: Python3, WeaviateClient3.26
- Multitenancy?: no

### Any additional Information

Logs (internal ips also masked):  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-1 10.xxx.aaa.bbb:7000",“time”:“2024-09-17T09:20:07Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.xxx.aaa.bbb:44766",“time”:“2024-09-17T09:20:17Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-2 10.xxx.ccc.ddd:7000",“time”:“2024-09-17T09:20:37Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.xxx.aaa.bbb:60348",“time”:“2024-09-17T09:20:47Z”}_  
_{“action”:“restapi\_management”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“docker\_image\_tag”:“1.26.4”,“level”:“info”,“msg”:"Shutting down… ",“time”:“2024-09-17T09:20:49Z”}_  
_{“action”:“restapi\_management”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“docker\_image\_tag”:“1.26.4”,“level”:“info”,“msg”:“Stopped serving weaviate at http://[::]:8080”,“time”:“2024-09-17T09:20:49Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“info”,“msg”:“closing raft FSM store …”,“time”:“2024-09-17T09:20:49Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“info”,“msg”:“shutting down raft sub-system …”,“time”:“2024-09-17T09:20:49Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“info”,“msg”:“closing raft-net …”,“time”:“2024-09-17T09:20:49Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“info”,“msg”:“closing log store …”,“time”:“2024-09-17T09:20:49Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“info”,“msg”:“closing data store …”,“time”:“2024-09-17T09:20:49Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“info”,“msg”:“closing loaded database …”,“time”:“2024-09-17T09:20:49Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“component”:“index\_queue”,“level”:“debug”,“msg”:“index queue closed”,“shard\_id”:“myclass\_nwkAu2cZVXzg”,“time”:“2024-09-17T09:20:49Z”}_  
_{“action”:“hnsw\_delete\_vector\_cache”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:“deleting full vector cache”,“time”:“2024-09-17T09:20:49Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“component”:“index\_queue”,“level”:“debug”,“msg”:“index queue closed”,“shard\_id”:“myclass\_uhCGgHQxXawb”,“time”:“2024-09-17T09:20:49Z”}_  
_{“action”:“hnsw\_delete\_vector\_cache”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:“deleting full vector cache”,“time”:“2024-09-17T09:20:49Z”}_  
_ **{“action”:“raft-net”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“error”:“transport shutdown”,“level”:“error”,“msg”:“raft-net failed to decode incoming command”,“time”:“2024-09-17T09:20:49Z”}** _  
_ **{“action”:“raft-net”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“error”:“transport shutdown”,“level”:“error”,“msg”:“raft-net failed to decode incoming command”,“time”:“2024-09-17T09:20:49Z”}** _  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-1 10.xxx.aaa.bbb:7000",“time”:“2024-09-17T09:21:07Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-1 10.xxx.aaa.bbb:7000",“time”:“2024-09-17T09:21:37Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.xxx.aaa.bbb:48332",“time”:“2024-09-17T09:21:47Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-2 10.xxx.ccc.ddd:7000",“time”:“2024-09-17T09:22:07Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.xxx.aaa.bbb:41448",“time”:“2024-09-17T09:22:17Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.xxx.ccc.ddd:41304",“time”:“2024-09-17T09:22:34Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-2 10.xxx.ccc.ddd:7000",“time”:“2024-09-17T09:22:37Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-2 10.xxx.ccc.ddd:7000",“time”:“2024-09-17T09:23:07Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.xxx.aaa.bbb:37156",“time”:“2024-09-17T09:23:17Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.xxx.ccc.ddd:36930",“time”:“2024-09-17T09:23:34Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-1 10.xxx.aaa.bbb:7000",“time”:“2024-09-17T09:23:37Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.xxx.ccc.ddd:42362",“time”:“2024-09-17T09:24:04Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-1 10.xxx.aaa.bbb:7000",“time”:“2024-09-17T09:24:07Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-1 10.xxx.aaa.bbb:7000",“time”:“2024-09-17T09:24:37Z”}_

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [September 18, 2024, 9:56pm UTC](https://forum.weaviate.io/t/whole-cluster-hangups-on-pod-termination/4197/2 "2024-09-18T21:56:09Z")

</div>

hi @wvuser !!

Do you know if this will also happen on a clean, fresh cluster or only with this specific one?

Also, this this log goes in loop for the entire `terminationGracePeriodSeconds`?

---

<div class="post-metadata">

**Author:** ![wvuser](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/wvuser/32/2371_2.png) [@wvuser](https://forum.weaviate.io/u/wvuser)\
**Post date:** [September 18, 2024, 10:52pm UTC](https://forum.weaviate.io/t/whole-cluster-hangups-on-pod-termination/4197/3 "2024-09-18T22:52:23Z")

</div>

> [@DudaNogueira](#):
>
> if this will also happen on a clean, fresh cluster

Cluster, schema and data-import are new (fresh). Not upgraded from previous version.

> [@DudaNogueira](#):
>
> this this log goes in loop for the entire `terminationGracePeriodSeconds`?

Yes, after “raft-net” error records, “synch” log records are cyclic…

---

<div class="post-metadata">

**Author:** ![wvuser](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/wvuser/32/2371_2.png) [@wvuser](https://forum.weaviate.io/u/wvuser)\
**Post date:** [September 19, 2024, 12:39pm UTC](https://forum.weaviate.io/t/whole-cluster-hangups-on-pod-termination/4197/4 "2024-09-19T12:39:41Z")

</div>

Hello, @DudaNogueira!

Some detailed information about cluster/weaviate configuration:

- [cluster stats](https://drive.google.com/file/d/1BYmdeVSAkJHtS1pZahb9Y_U_79Fg0qSC/view?usp=sharing)
- [env vars](https://drive.google.com/file/d/1VT-n9oypVmS5iCedClkL6rzvYTxN14KM/view?usp=sharing)
- [nodes info](https://drive.google.com/file/d/1hSbLgwoW49FYyQevY0MhNmZdeVFNpors/view?usp=sharing)
- [schema info](https://drive.google.com/file/d/1fyHJcJnqHo5A_IEKAemzVdIrSBD9L1Jb/view?usp=sharing)

Аccording to our observations and experiments… problem in ‘async replication’ processes (or other internal grpc communications?).  
When pods hangups on terminate, synchronization logs look like this:

_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-1 10.xxx.aaa.bbb:7000",“time”:“2024-09-17T09:21:07Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-1 10.xxx.aaa.bbb:7000",“time”:“2024-09-17T09:21:37Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.xxx.aaa.bbb:48332",“time”:“2024-09-17T09:21:47Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-2 10.xxx.ccc.ddd:7000",“time”:“2024-09-17T09:22:07Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.xxx.aaa.bbb:41448",“time”:“2024-09-17T09:22:17Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.xxx.ccc.ddd:41304",“time”:“2024-09-17T09:22:34Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-2 10.xxx.ccc.ddd:7000",“time”:“2024-09-17T09:22:37Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-2 10.xxx.ccc.ddd:7000",“time”:“2024-09-17T09:23:07Z”}_

But, if logs look like below - pods can normally/fast restarts without freezing:

_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.aaa.bbb.ccc:59830",“time”:“2024-09-19T11:09:27Z”}_  
_**{“action”:“async\_replication”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“class\_name”:“MyClass”,“diff\_calculation\_took”:“26.801µs”,“hashbeat\_iteration”:107,“hosts”:[“10.aaa.ddd.eee:7001”,“10.aaa.fff.ggg:7001”,“10.aaa.bbb.ccc:7001”],“level”:“info”,“local\_objects”:0,“msg”:“hashbeat iteration successfully completed”,“object\_progation\_took”:“0s”,“objects\_propagated”:0,“remote\_objects”:0,“shard\_name”:“2q4qSTnnbzwk”,“time”:“2024-09-19T11:09:34Z”}**_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-1 10.aaa.bbb.ccc:7000",“time”:“2024-09-19T11:09:38Z”}_  
_**{“action”:“async\_replication”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“class\_name”:“MyClass”,“diff\_calculation\_took”:“24.166µs”,“hashbeat\_iteration”:107,“hosts”:[“10.aaa.fff.ggg:7001”,“10.aaa.ddd.eee:7001”,“10.aaa.bbb.ccc:7001”],“level”:“info”,“local\_objects”:0,“msg”:“hashbeat iteration successfully completed”,“object\_progation\_took”:“0s”,“objects\_propagated”:0,“remote\_objects”:0,“shard\_name”:“s5zw0VLZnClB”,“time”:“2024-09-19T11:09:39Z”}**_  
_{“action”:“async\_replication”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“class\_name”:“MyClass”,“diff\_calculation\_took”:“23.906µs”,“hashbeat\_iteration”:107,“hosts”:[“10.aaa.fff.ggg:7001”,“10.aaa.ddd.eee:7001”,“10.aaa.bbb.ccc:7001”],“level”:“info”,“local\_objects”:0,“msg”:“hashbeat iteration successfully completed”,“object\_progation\_took”:“0s”,“objects\_propagated”:0,“remote\_objects”:0,“shard\_name”:“qUdhGTwM36c5”,“time”:“2024-09-19T11:09:42Z”}_  
_{“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“level”:“debug”,“msg”:" memberlist: Stream connection from=10.aaa.fff.ggg:39338",“time”:“2024-09-19T11:09:50Z”}_  
_{“action”:“async\_replication”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“class\_name”:“MyClass”,“diff\_calculation\_took”:“24.117µs”,“hashbeat\_iteration”:108,“hosts”:[“10.aaa.bbb.ccc:7001”,“10.aaa.ddd.eee:7001”,“10.aaa.fff.ggg:7001”],“level”:“info”,“local\_objects”:0,“msg”:“hashbeat iteration successfully completed”,“object\_progation\_took”:“0s”,“objects\_propagated”:0,“remote\_objects”:0,“shard\_name”:“2q4qSTnnbzwk”,“time”:“2024-09-19T11:09:54Z”}_  
_{“action”:“async\_replication”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“class\_name”:“MyClass”,“diff\_calculation\_took”:“22.584µs”,“hashbeat\_iteration”:108,“hosts”:[“10.aaa.ddd.eee:7001”,“10.aaa.fff.ggg:7001”,“10.aaa.bbb.ccc:7001”],“level”:“info”,“local\_objects”:0,“msg”:“hashbeat iteration successfully completed”,“object\_progation\_took”:“0s”,“objects\_propagated”:0,“remote\_objects”:0,“shard\_name”:“s5zw0VLZnClB”,“time”:“2024-09-19T11:09:59Z”}_  
_{“action”:“async\_replication”,“build\_git\_commit”:“584532a”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.4”,“build\_wv\_version”:“1.26.4”,“class\_name”:“MyClass”,“diff\_calculation\_took”:“25.238µs”,“hashbeat\_iteration”:108,“hosts”:[“10.aaa.fff.ggg:7001”,“10.aaa.bbb.ccc:7001”,“10.aaa.ddd.eee:7001”],“level”:“info”,“local\_objects”:0,“msg”:“hashbeat iteration successfully completed”,“object\_progation\_took”:“0s”,“objects\_propagated”:0,“remote\_objects”:0,“shard\_name”:“qUdhGTwM36c5”,“time”:“2024-09-19T11:10:02Z”}_

Also noticed, that the “async replication” sometimes stops working… ( **/schema** replies that: _“asyncEnabled”: true_), but only re-update this param to ‘true’ again, helps activate async replication).