# Cluster inconsistency and cpu usage

**URL:** <https://forum.weaviate.io/t/cluster-inconsistency-and-cpu-usage/822>\
**Category:** Support\
**Created:** [October 13, 2023, 12:09pm UTC](https://forum.weaviate.io/t/cluster-inconsistency-and-cpu-usage/822 "2023-10-13T12:09:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![KCog](https://avatars.discourse-cdn.com/v4/letter/k/439d5e/32.png) [@KCog](https://forum.weaviate.io/u/KCog)\
**Post date:** [October 13, 2023, 12:09pm UTC](https://forum.weaviate.io/t/cluster-inconsistency-and-cpu-usage/822/1 "2023-10-13T12:09:36Z")

</div>

Hi, we ran into problems with a cluster with 3 replicas.

The 3d replica failed with the following error logs:

```auto
{"action":"startup_cluster_schema_sync","diff":["local has 322 classes, but cluster has 321 classes","class C_651fdeebf176f871a80562de_enUS exists in local, but not in cluster"],"level":"error","msg":"mismatch between local schema and remote (other nodes consensus) schema","time":"2023-10-06T10:47:47Z"}
{"action":"startup","error":"could not load or initialize schema: sync schema with other nodes in the cluster: corrupt cluster: other nodes have consensus on schema, but local node has a different (non-null) schema","level":"fatal","msg":"could not initialize schema manager","time":"2023-10-06T10:47:47Z"}

```

Scaling down to 2 and scaling up again didn’t resolve the issue. We ended up deleting the volume of this 3d replica. Everything seemed to be fine after.

Related or not, sometime later we had the following problem in this cluster: the first 2 replicas have a constant cpu usage (and cpu throttling) of 100%. The 3d replica has very low cpu usage. Updates and search still work.

In all 3 replicas we see a lot of “context cancelled” errors, e.g.:

```auto
2023-10-10T09:35:08+02:00 {"level":"error","msg":"\"10.7.64.14:7001\": connect: Patch \"http://10.7.64.14:7001/replicas/indices/C_6524ea3823742848c7547e53_enUS/shards/TIhpJZ5a7rXp/objects/85478e9d-3a5a-4a16-ab27-c6c0535b5fa1?request_id=weaviate-1-02-18b1882d2b5-164\": context canceled","op":"broadcast","time":"2023-10-10T07:35:08Z"}
2023-10-10T09:57:44+02:00 {"level":"error","msg":"\"10.7.64.14:7001\": connect: Post \"http://10.7.64.14:7001/replicas/indices/C_6524f7c7e1f5116d8f8ff154_enUS/shards/FYmnc47ubeYR/objects?request_id=weaviate-1-01-18b18978205-1\": context canceled","op":"broadcast","time":"2023-10-10T07:57:44Z"}

```

Any advice on how we can solve these problems?

We noticed a misconfiguration on our end: we use helm chart v16.1.0, but image tag 1.19.0.  
Not sure if the issues are caused by this.

Thanks!

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [October 17, 2023, 1:21pm UTC](https://forum.weaviate.io/t/cluster-inconsistency-and-cpu-usage/822/2 "2023-10-17T13:21:05Z")

</div>

Hi @KCog ! Welcome to our community! 🤗

I am not sure, but scaling down may lead to different other issues. We have a nice repo that tries to replicate those scenarios:

> **[GitHub - weaviate/weaviate-chaos-engineering: Chaos-Engineering-Style CI...](https://github.com/weaviate/weaviate-chaos-engineering)**
>
> Chaos-Engineering-Style CI Pipelines to make sure Weaviate handles whatever the real world throws at it. - GitHub - weaviate/weaviate-chaos-engineering: Chaos-Engineering-Style CI Pipelines to make...

I have asked internally about what one should do when faced with class mismatch and hope to get back here with more info when possible.

Thanks!