Weaviate resource usage

Support
1

Description

Weaviate using almost 100% of memory even after setting the LIMIT_RESOURCES to true

Server Setup Information

  • Weaviate Server Version: 1.23.3
  • Deployment Method: k8s
  • Multi Node? Number of Running Nodes: no
  • Client Language and Version: python

Any additional Information

We are using Weaviate for indexing objects with the OpenAI model text-adda-002, which produces vectors with 1536 dimensions each.

We have multiple classes in our dataset. One of these classes, which is the most resource-intensive, is configured with vectorCacheMaxObjects set to 10,000. This class has approximately 1.2 million vector indices, as indicated by the Prometheus vector_index_size monitoring metrics. This configuration ensures that it doesn’t consume all available memory, as the maximum cache objects are limited to 10,000. The remaining classes each have around 1 million vector indices and are not configured with vectorCacheMaxObjects. The instance has a total of 24 GB of RAM available.

However, when observing the memory usage graph, we see a gradual increase in memory consumption until it reaches 100%, eventually leading to an out-of-memory (OOM) event that causes Weaviate to restart.

i am unable to find any cause for this

also i am adding a screenshot for my vector index size count changes as i doubt delete operations or ingestion operations can increase the resources :

image(1)
image(1)1920×444 39.2 KB

2

hi @darshilshahquantive !

Welcome to our community! :hugs:

As this was also asked in slack, here is the recommendations from @antas-marcin :hugs:

Please set GOMEMLIMIT to about 90% of your available memory and (let’s say GOMEMLIMIT=21GiB ) and maybe set GOGC to 50 which should trigger Go GC more frequently

and

after setting those values, please observe how your setup behaves, maybe those values will work for your setup or maybe they would need further tuning

Thanks!!