Very High Memory usage even after low vector_cache_max_objects

Support
1

Description

Hi,
I have 60 million objects that have been imported, and the memory usage was high during the import and I thought it was just high usage bc of the import.
Now, even after I set the vector_cache_max_objects to 1,000,000 to limit the object cache usage, it still requires 130gbs of memory and would go oom in 30 min, it would be a little bit better if I set gogc lower than 20, but it would still go oom anyway.
Why the usage is still high though?

Here is my docker yml file:

version: '3.4'
services:
  weaviate:
    image: cr.weaviate.io/semitechnologies/weaviate:1.25.2
    ports:
    - 8080:8080
    - 50051:50051
    restart: always
    volumes:
    - weaviate_data:/weaviate/data
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      LIMIT_RESOURCES: 'true'
      # GOGC: 15
      GOMAXPROCS: 20
      PERSISTENCE_DATA_PATH: '/weaviate/data'
      DEFAULT_VECTORIZER_MODULE: 'none'
      CLUSTER_HOSTNAME: 'node1'
      ASYNC_INDEXING: 'true'
      PROMETHEUS_MONITORING_ENABLED: 'true'
      TOMBSTONE_DELETION_MAX_PER_CYCLE: '10000000'
      TOMBSTONE_DELETION_MIN_PER_CYCLE: '1000000'
      AUTOSCHEMA_ENABLED: 'true'

and here is my docker log that clearly shows I have set vector_cache_max_objects right

2024-06-18 12:54:21 {"action":"hnsw_vector_cache_prefill","count":67698902,"index_id":"main","level":"info","limit":1000000,"msg":"prefilled vector cache","time":"2024-06-18T04:54:21Z","took":13709}

I also tried to use pq since memory usage was too high
but got this

2024-06-18 12:54:27 {"action":"lsm_compaction","class":"Case","index":"case","level":"warning","msg":"compaction halted due to shard READONLY status","path":"/weaviate/data/case/K47eMCytsotX/lsm","shard":"K47eMCytsotX","time":"2024-06-18T04:54:27Z"}

Server Setup Information

  • Weaviate Server Version:
  • Deployment Method: Docker
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version:
  • Multitenancy?: no

Any additional Information

2

And here is the list all of my collection

{'Case': _CollectionConfig(name='Case', description=None, generative_config=None, inverted_index_config=_InvertedIndexConfig(bm25=_BM25Config(b=0.75, k1=1.2), cleanup_interval_seconds=60, index_null_state=False, index_property_length=False, index_timestamps=False, stopwords=_StopwordsConfig(preset=<StopwordsPreset.EN: 'en'>, additions=None, removals=None)), multi_tenancy_config=_MultiTenancyConfig(enabled=False, auto_tenant_creation=False, auto_tenant_activation=False), properties=[_Property(name='case', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none'), _Property(name='publication_date', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none'), _Property(name='type', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none'), _Property(name='case_title', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none'), _Property(name='law', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none'), _Property(name='procedure', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none'), _Property(name='party', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none'), _Property(name='case_name', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none'), _Property(name='doc_id', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.UUID: 'uuid'>, index_filterable=True, index_searchable=False, nested_properties=None, tokenization=None, vectorizer_config=None, vectorizer='none'), _Property(name='location', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none'), _Property(name='reason', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none'), _Property(name='court', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none'), _Property(name='case_date', description="This property was generated by Weaviate's auto-schema feature on Wed Jun 12 11:20:15 2024", data_type=<DataType.TEXT: 'text'>, index_filterable=True, index_searchable=True, nested_properties=None, tokenization=<Tokenization.WORD: 'word'>, vectorizer_config=None, vectorizer='none')], references=[], replication_config=_ReplicationConfig(factor=1), reranker_config=None, sharding_config=_ShardingConfig(virtual_per_physical=128, desired_count=1, actual_count=1, desired_virtual_count=128, actual_virtual_count=128, key='_id', strategy='hash', function='murmur3'), vector_index_config=_VectorIndexConfigHNSW(quantizer=_PQConfig(internal_bit_compression=False, segments=96, centroids=256, training_limit=100000, encoder=_PQEncoderConfig(type_=<PQEncoderType.KMEANS: 'kmeans'>, distribution=<PQEncoderDistribution.LOG_NORMAL: 'log-normal'>)), cleanup_interval_seconds=300, distance_metric=<VectorDistances.COSINE: 'cosine'>, dynamic_ef_min=100, dynamic_ef_max=500, dynamic_ef_factor=8, ef=-1, ef_construction=128, flat_search_cutoff=40000, max_connections=64, skip=False, vector_cache_max_objects=1000000), vector_index_type=<VectorIndexType.HNSW: 'hnsw'>, vectorizer_config=None, vectorizer=<Vectorizers.NONE: 'none'>, vector_config=None)}
3

Could it be the vectors are still inserting?
I just checked the dashboard, and even though there is no object importing, the vector index stats section still shows the “vectors inserted” number increasing over time.

4

hi @Frank_Gao !

Have you seen this page?

You can estimate the resource usage. Depending on the number of dimensions used.

For example, for objects with 512 dimensions, something around 228.6 gigabytes is expected.

From your logs, seems like you had reached the READONLY threshold, so Weaviate set your shards to READONLY. check here for more on that

You may need more disk space so it can keep the compaction going on.

Let me know if this helps.

Thanks!

5

Hi,
Thank you so much for responding.
The memory is still as high as before ( 130 GB) even after I set the [vector_cache_max_objects] to 10k or even 1k only.
My embedding has 768 dimensions. By the calculations, it shouldn’t take this high memory, right?

6

hi thank you for your response its help me alot

1 Like
7

for 60 million objects:

with 768 dimensions, you can expect something around 5.72 GB per million

so: the estimation would be 343.2 gigabytes
note: this is already with a rule of thumbs for 2x

However, changing that parameter should take some effect.

Have you tried restarting the cluster so it forces a reload of that collection?

8

hi @Talal_dev !! Welcome to our community!

Glad you liked!

9

Also, compression reduces memory footprint a lot.

1 Like