# Catastrophic performance drop when upscaling a cluster - while pod not in READY state (loading data/index)

**URL:** <https://forum.weaviate.io/t/catastrophic-performance-drop-when-upscaling-a-cluster-while-pod-not-in-ready-state-loading-data-index/8861>\
**Category:** Support\
**Tags:** technical\
**Created:** [November 29, 2024, 5:32am UTC](https://forum.weaviate.io/t/catastrophic-performance-drop-when-upscaling-a-cluster-while-pod-not-in-ready-state-loading-data-index/8861 "2024-11-29T05:32:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![wvuser](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/wvuser/32/2371_2.png) [@wvuser](https://forum.weaviate.io/u/wvuser)\
**Post date:** [November 29, 2024, 5:32am UTC](https://forum.weaviate.io/t/catastrophic-performance-drop-when-upscaling-a-cluster-while-pod-not-in-ready-state-loading-data-index/8861/1 "2024-11-29T05:32:57Z")

</div>

### Description

We have a K8s cluster with 3 servers and 3 pods (1 pod per server).  
Weaviate has 20mln objects (2 named vectors per object: 1024 and 768).  
All requests (set of CRUD+vectors-search operations) executed with QUORUM concistency level.  
With 3 active (READY) pods we have a total performance with ~380RPS.  
When downscaling cluster to 2 pods (“weaviate-2” goes away), cluster’s performance returns to same.  
**Upscaling cluster back to 3 pods (“weaviate-2” returns) make performance dropping to ~160RPS… on period while pod loading data/index (not in READY state).**  
Can’t see any CRUD/search requests in weaviate-2’s logs.  
When “weaviate-2’s” loading finished (became READY), performance returns to normal ~380RPS.

 ![downscale_to_2](https://canada1.discourse-cdn.com/flex027/uploads/weaviate/original/2X/1/1d38742dbe967c1b3f844c72225b0d915b103835.png)

### Server Setup Information

- Weaviate Server Version: 1.25.25
- Deployment Method: k8s
- Multi Node? Number of Running Nodes: yes, 3 nodes, repl.factor=3
- Client Language and Version: Python-3, weaviate-client-4.6.2
- Multitenancy?: no

### Any additional Information

DISABLE\_LAZY\_LOAD\_SHARDS=true  
HNSW\_STARTUP\_WAIT\_FOR\_VECTOR\_CACHE=true

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [November 29, 2024, 10:11am UTC](https://forum.weaviate.io/t/catastrophic-performance-drop-when-upscaling-a-cluster-while-pod-not-in-ready-state-loading-data-index/8861/2 "2024-11-29T10:11:38Z")

</div>

hi @wvuser !!

A lot has changed since 1.25, so if possible, I would suggest you to upgrade to latest version.

Can you see any logs while doing those changes?

---

<div class="post-metadata">

**Author:** ![wvuser](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/wvuser/32/2371_2.png) [@wvuser](https://forum.weaviate.io/u/wvuser)\
**Post date:** [December 19, 2024, 9:42am UTC](https://forum.weaviate.io/t/catastrophic-performance-drop-when-upscaling-a-cluster-while-pod-not-in-ready-state-loading-data-index/8861/3 "2024-12-19T09:42:53Z")

</div>

Hi @DudaNogueira !  
Sorry for the long absence…  
What LOG\_LEVEL minimum value do you recommend/require?  
The logs (from all PODs) will be large because with a ‘light load’ the performance drop is not so visible.
