High resource usage in Flat Index

Support
1

Team, I have created 10,000 tenants and imported 100 data for each tenant. My configured GOMAXPROCS and GOMEMLIMIT values as per the setup needs.

image

The above is the resource consumption for the data, and it seems very high for the flat index in terms of both CPU and memory.

Weaviate version: 1.23.11
Python client: 4.5.7

Vector Dimension: 384
Replica : 2
No. of properties: 4
Total number of records : 1000000.

2

hi @Dharanish !

I would know what would be the usage on this scenario :thinking:

I am curious: Have you benchmarked this with other databases?

Thanks!

3

Sorry for the delayed response
No, we are primarily using Weaviate for production; there is a new usecase and we don’t evaluate with other DBs yet.

4

Oh, ok.

So what are the allocated values for memory and cpu, and what are the values of GOMEMLIMIT and GOMAXPROCS?

Not sure you have read thru this nice blog article about GOMEMLIMIT:

Because you are using a flat index, your heap will not be an issue here, (you are storing on disk, not on memory), but setting those variables can force garbage collection kick in quicker and free up memory.

Let me know if that helps!

THanks!

5

@DudaNogueira , I’ve read that article before and set values for those variables based on our resource needs.

I’ve found the catch that this resource consumption is due to PROMETHEUS_MONITORING_ENABLED. After disabled, we get

image
image2686×124 101 KB

It seems prometheus metrics collection plays a major role in resource consumption. We found this using go profiling tool But don’t know why the Prometheus client used in Weaviate consumes that much amount of resources.

6

Nice catch!!

Do you see the same behavior in latest versions?

A lot has happened since 1.23… :grimacing: