# Rolling Update Not Working

**URL:** <https://forum.weaviate.io/t/rolling-update-not-working/1492>\
**Category:** Support\
**Created:** [February 15, 2024, 2:06am UTC](https://forum.weaviate.io/t/rolling-update-not-working/1492 "2024-02-15T02:06:13Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Landon\_Edwards](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/landon_edwards/32/649_2.png) [@Landon\_Edwards](https://forum.weaviate.io/u/Landon_Edwards)\
**Post date:** [February 15, 2024, 2:06am UTC](https://forum.weaviate.io/t/rolling-update-not-working/1492/1 "2024-02-15T02:06:13Z")

</div>

Hi I am experiencing issues when re-deploing a multi-replica setup on K8.  
What I am finding is if I do a full delete of all created k8 object and then deploy everything again it launches as expected however when attempting to do a rolling update over the existing pods the first new replica to be added experiences the following errors repeatedly:

```auto
{"action":"startup", "error":"could not load or initialize schema: sync schema with other nodes in the cluster: read schema: open transaction: broadcast open transaction: host "xxx.xx.xx.xxx:7001": unexpected status code 401 ()", "level":"fatal", "msg":"could not initialize schema manager"}

```

The rolling update then just get stuck in a crashback loop while this repeatedly happens.

Any help that can be offered would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [February 15, 2024, 2:37pm UTC](https://forum.weaviate.io/t/rolling-update-not-working/1492/2 "2024-02-15T14:37:35Z")

</div>

Hi!

Just to confirm, you are using our helm chart, right?

> **[GitHub - weaviate/weaviate-helm: Helm charts to deploy Weaviate to k8s](https://github.com/weaviate/weaviate-helm)**
>
> Helm charts to deploy Weaviate to k8s. Contribute to weaviate/weaviate-helm development by creating an account on GitHub.

It seems some connectivity issue between pods 🤔

---

<div class="post-metadata">

**Author:** ![Landon\_Edwards](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/landon_edwards/32/649_2.png) [@Landon\_Edwards](https://forum.weaviate.io/u/Landon_Edwards)\
**Post date:** [February 15, 2024, 5:26pm UTC](https://forum.weaviate.io/t/rolling-update-not-working/1492/3 "2024-02-15T17:26:20Z")

</div>

Yes using your helm templates

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [February 15, 2024, 5:53pm UTC](https://forum.weaviate.io/t/rolling-update-not-working/1492/4 "2024-02-15T17:53:18Z")

</div>

Can you detail the steps? I would like to reproduce this.

Thanks!

---

<div class="post-metadata">

**Author:** ![walter](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/walter/32/1008_2.png) [@walter](https://forum.weaviate.io/u/walter)\
**Post date:** [February 20, 2024, 11:12am UTC](https://forum.weaviate.io/t/rolling-update-not-working/1492/5 "2024-02-20T11:12:54Z")

</div>

I’m using the helm chart and observing the same issue, I use ArgoCD to manage my weaviate deployment, after one pod gets replaced, the others would report that the pod is not available but will be always trying to reach it in its old IP address, somehow the old pods will never figure out what the new pod IP address is, I don’t know which resource is the responsible of updating the IP is.

This behaviour however doesn’t occur if one of the pods is deleted, when the pod is deleted the rest of the pods will notice that it went away and after some time the new one is created they will stop reporting the error.

---

<div class="post-metadata">

**Author:** ![MohamedBadawi](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/mohamedbadawi/32/1015_2.png) [@MohamedBadawi](https://forum.weaviate.io/u/MohamedBadawi)\
**Post date:** [February 21, 2024, 2:53pm UTC](https://forum.weaviate.io/t/rolling-update-not-working/1492/6 "2024-02-21T14:53:50Z")

</div>

@walter , @Landon_Edwards, Thanks for reporting. Could you provide us the logs from the nodes, from start till reporting that error ? that would help us to detect the exact problem.

---

<div class="post-metadata">

**Author:** ![joris](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/joris/32/129_2.png) [@joris](https://forum.weaviate.io/u/joris)\
**Post date:** [March 21, 2024, 3:04pm UTC](https://forum.weaviate.io/t/rolling-update-not-working/1492/7 "2024-03-21T15:04:53Z")

</div>

Hi @MohamedBadawi ,

We are experiencing the same issue. These are our logs:

```auto
{"action":"startup","level":"debug","msg":"created startup context, nothing done so far","startup_time_left":"59m59.998652288s","time":"2024-03-21T15:00:49Z"}
{"action":"config_load","config_file_path":"/weaviate-config/conf.yaml","level":"info","msg":"Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.","time":"2024-03-21T15:00:49Z"}
{"deprecation":{"apiType":"Configuration","id":"config-files","locations":["--config-file=\"\""],"mitigation":"Configure Weaviate using environment variables.","msg":"use of deprecated command line argument --config-file","sinceTime":"2020-09-08T09:46:00.000Z","sinceVersion":"0.22.16","status":"deprecated"},"level":"warning","msg":"use of deprecated command line argument --config-file","time":"2024-03-21T15:00:49Z"}
{"action":"startup","default_vectorizer_module":"text2vec-huggingface","level":"info","msg":"the default vectorizer modules is set to \"text2vec-huggingface\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-03-21T15:00:49Z"}
{"action":"startup","auto_schema_enabled":false,"level":"info","msg":"auto schema enabled setting is set to \"false\"","time":"2024-03-21T15:00:49Z"}
{"action":"startup","level":"debug","msg":"config loaded","startup_time_left":"59m59.997784462s","time":"2024-03-21T15:00:49Z"}
{"action":"startup","level":"debug","msg":"configured OIDC and anonymous access client","startup_time_left":"59m59.997761381s","time":"2024-03-21T15:00:49Z"}
{"action":"startup","level":"debug","msg":"initialized schema","startup_time_left":"59m59.997737462s","time":"2024-03-21T15:00:49Z"}
{"level":"debug","msg":" memberlist: Initiating push/pull sync with: 10.42.11.190:7000","time":"2024-03-21T15:00:49Z"}
{"level":"debug","msg":" memberlist: Initiating push/pull sync with: 10.42.12.82:7000","time":"2024-03-21T15:00:49Z"}
{"level":"debug","msg":" memberlist: Stream connection from=10.42.12.82:47438","time":"2024-03-21T15:00:49Z"}
{"level":"debug","msg":" memberlist: Initiating push/pull sync with: 10.42.12.157:7000","time":"2024-03-21T15:00:49Z"}
{"action":"startup","level":"debug","msg":"startup routine complete","time":"2024-03-21T15:00:49Z"}
{"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-03-21T15:00:49Z"}
{"action":"startup","level":"debug","msg":"start registering modules","time":"2024-03-21T15:00:49Z"}
{"action":"startup","level":"debug","module":"reranker-cohere","msg":"enabled module","time":"2024-03-21T15:00:49Z"}
{"action":"startup","level":"debug","module":"text2vec-openai","msg":"enabled module","time":"2024-03-21T15:00:49Z"}
{"action":"startup","level":"debug","module":"text2vec-huggingface","msg":"enabled module","time":"2024-03-21T15:00:49Z"}
{"action":"startup","level":"debug","module":"text2vec-cohere","msg":"enabled module","time":"2024-03-21T15:00:49Z"}
{"action":"startup","level":"debug","msg":"completed registering modules","time":"2024-03-21T15:00:49Z"}
{"level":"info","msg":"async indexing enabled","time":"2024-03-21T15:00:49Z"}
{"action":"broadcast_abort_transaction","error":"host \"10.42.12.157:7001\": unexpected status code 401: ","id":"ce132362-4a30-4383-adb2-44092fa40d11","level":"error","msg":"broadcast tx abort failed","time":"2024-03-21T15:00:49Z"}
{"action":"startup","error":"could not load or initialize schema: sync schema with other nodes in the cluster: read schema: open transaction: broadcast open transaction: host \"10.42.12.157:7001\": unexpected status code 401 ()","level":"fatal","msg":"could not initialize schema manager","time":"2024-03-21T15:00:49Z"}

```

---

<div class="post-metadata">

**Author:** ![MohamedBadawi](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/mohamedbadawi/32/1015_2.png) [@MohamedBadawi](https://forum.weaviate.io/u/MohamedBadawi)\
**Post date:** [April 4, 2024, 2:51pm UTC](https://forum.weaviate.io/t/rolling-update-not-working/1492/8 "2024-04-04T14:51:32Z")

</div>

Thanks @joris for reporting, as mentioned in thread it’s connectivity issue, see [here](https://forum.weaviate.io/t/rolling-update-not-working/1492/2). can you details the steps which made ended up to this situation

---

<div class="post-metadata">

**Author:** ![joris](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/joris/32/129_2.png) [@joris](https://forum.weaviate.io/u/joris)\
**Post date:** [April 8, 2024, 7:15am UTC](https://forum.weaviate.io/t/rolling-update-not-working/1492/9 "2024-04-08T07:15:31Z")

</div>

Hi @MohamedBadawi,

The connectivity issue happens when you have rolling updates enabled and one of the shards has a different version than the other.

---

<div class="post-metadata">

**Author:** ![adithya.ch](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/adithya.ch/32/615_2.png) [@adithya.ch](https://forum.weaviate.io/u/adithya.ch)\
**Post date:** [April 29, 2024, 7:04pm UTC](https://forum.weaviate.io/t/rolling-update-not-working/1492/10 "2024-04-29T19:04:09Z")

</div>

Hi @joris , Were you able to fix this issue ? We are also seeing the same issue.  
It would be of great help if you can share the fix details.

Regards,  
Adithya