High CPU Usage if number of shards increases

General
1

if number of shards/tenants increases , cpu usage also increases even in idle and very less number of objects in each shards. As far as tracked profile using go profiler it seems CycleManager package plays a significant role in consuming cpu .

image
image2892×150 52.9 KB

profile svg

image
image1336×1348 265 KB

Environment
Weaviate version - 1.28.11
cpu configured - 6 core per node.

Avg Shards per node
Node1 - 1500 shards
Node2 - 1800 shards
Node3 - 3000 shards

is this the default CPU behavior for this amount of shards?

2

Hi @Dharanish,

I understand your concern about the CPU usage seem odd for small shards count.

You’re currently on version 1.28.11. There have been several performance optimizations and bug fixes. Before dive deeper into this, I’d upgrade to the latest version (1.31.0) first then test. This will help to see if this is something need to be looked at or addressed.

Could you test on 1.31.0 and let me know if you see the same behavior?

Also, here’s our resource planning documentation which have helpful information:

Best regards,
Mohamed Shahin
Weaviate Support Engineer
(Ireland, UTC±00:00/+01:00)