Over Memory consumption of Weaviate

General
1

I’ve been engaged in a project for the past few months, and recently, I’ve come across a memory consumption issue. Currently, out of my total 47 GB of memory, one Weaviate instance is utilizing 13.1 GB, which seems higher than expected.

on_docker_Stats
on_docker_Stats913×51 4.42 KB

Upon consulting the Weaviate documentation, I discovered a method to calculate memory usage. The precise formula is as follows:

Number of objects * ( (Length of single vector * Float value precision in bytes) + (MaxConnections * Memory used by each vector[no of connections]))

Using this formula:
2750000 * ((768 * 4) + (64 * 10)) = 9.50 GB

I would appreciate any insights into potential reasons for memory consumption beyond what this calculation suggests.

2

as answered in StackOverflow

Hi! Duda from Weaviate here :slight_smile:

This is the page where you probably got this information: Resource Planning | Weaviate - vector database

This is a rough estimative and it may vary with other factors, for example while indexing, it will consume more memory, or depending on some Index configurations, etc.

Also the maxConnections estimate doesn’t account for garbage collection.

Let me know if that helps!