[Clarification on resource planning]

Reading this resource planning doc and this post, the calculation seems to be the total storage size needed for a example dataset instead of storing index in memory.

To clarify, based on the calculation, does it mean the complete vector is stored in memory as index (together with other overhead such max maxConnection)?

If that is the case, what data are stored in the storage, are most data in memory and storage overlap?


That’s the total memory used. The HNSW will be stored in memory.

Those calculations in resource planning doc will show the RAM memory to have those vectors in READY shards :slight_smile:

If your class is compressed, for example, it will load the compressed HNSW in memory (reduced memory usage), and it will now store both the original vectors and the compressed/trained ones in disk.

Depending on the configurations of your index, you can expect the storage size to be between 0.1 and 0.6 of the memory size. Because of the different configurations that will impact on the graph, it’s hard to have an assertive formula.

Again, thanks for those awesome questions :wink:

1 Like