Memory Pressure in Single-Instance Weaviate Under Continuous Write/Deletion Load

Support
,
1

Description

We deployed a single-instance Weaviate in Kubernetes with memory requests and limits both set to 3 GiB. After Weaviate starts up, its memory usage quickly rises to around 2.64 GiB. This isn’t an issue under light load, but when we send continuous requests (e.g., adding or deleting objects) about 300K objects, the Weaviate Python client starts throwing “out of memory” errors.
Below is the relevant portion of our Kubernetes configuration:

    Limits:
      cpu:     3
      memory:  3000Mi
    Requests:
      cpu:      500m
      memory:   3000Mi
    Environment:
      CLUSTER_GOSSIP_BIND_PORT:                 7000
      CLUSTER_DATA_BIND_PORT:                   7001
      GOGC:                                     100
      PROMETHEUS_MONITORING_ENABLED:            false
      GOMEMLIMIT:                               2500MiB
      QUERY_MAXIMUM_RESULTS:                    100000
      TRACK_VECTOR_DIMENSIONS:                  false
      REINDEX_VECTOR_DIMENSIONS_AT_STARTUP:     false
      STANDALONE_MODE:                          true
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED:  true
      CLUSTER_HOSTNAME:                         8b4f4cd0f4c5
      PERSISTENCE_DATA_PATH:                    /var/lib/weaviate
      BACKUP_FILESYSTEM_PATH:                   /var/lib/backup
      ENABLE_MODULES:                           backup-filesystem
      DISABLE_GRAPHQL:                          true
    Mounts:
      /var/lib/weaviate from weaviate-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sjqt6 (ro)
      /weaviate-config from weaviate-config (rw)

Error from the Python client:

Query call with protocol GRPC delete failed with message <AioRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "batch delete: cannot process batch delete object: not enough memory"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:2, grpc_message:"batch delete: cannot process batch delete object: not enough memory"}"
>.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/weaviate/connect/v4.py", line 1219, in grpc_batch_delete
    return await self.grpc_stub.BatchDelete(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/grpc/aio/_call.py", line 327, in __await__
    raise _create_rpc_error(
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "batch delete: cannot process batch delete object: not enough memory"

Corresponding error from the Weaviate server logs:

{"build_git_commit":"6f11fca","build_go_version":"go1.24.5","build_image_tag":"v1.30.13","build_wv_version":"1.30.13","error":"not enough memory","level":"error","msg":"memory pressure: cannot process batch delete object","time":"2025-11-02T00:21:44Z"}
{"build_git_commit":"6f11fca","build_go_version":"go1.24.5","build_image_tag":"v1.30.13","build_wv_version":"1.30.13","error":"not enough memory","level":"error","msg":"memory pressure: cannot process batch delete object","time":"2025-11-02T00:21:44Z"}

We’re trying to understand why Weaviate is hitting memory limits despite having 3 GiB allocated, especially since the GOMEMLIMIT is set to 2500 MiB . Any insights would be greatly appreciated!

Server Setup Information

  • Weaviate Server Version: 1.30.13
  • Deployment Method: k8s
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: python 3.11
  • Multitenancy?: No

Any additional Information

  1. When I restarted the weaviate pod, it raises “not enough memory” immediately
{"action":"lsm_compaction","build_git_commit":"6f11fca","build_go_version":"go1.24.5","build_image_tag":"v1.30.13","build_wv_version":"1.30.13","class":"PtAi_rag_39fa06cf_4178_e8bc_8172_9869630559c1_knowledge_68d5216dcff624278e17e236","error":"not enough memory","event":"compaction_skipped_oom","index":"ptai_rag_39fa06cf_4178_e8bc_8172_9869630559c1_knowledge_68d5216dcff624278e17e236","level":"warning","msg":"skipping compaction due to memory pressure","path":"/var/lib/weaviate/ptai_rag_39fa06cf_4178_e8bc_8172_9869630559c1_knowledge_68d5216dcff624278e17e236/x22wtE6RzHmd/lsm/objects","shard":"x22wtE6RzHmd","time":"2025-11-02T03:18:27Z"}
  1. When I delete the class “PtAi_rag_39fa06cf_4178_e8bc_8172_9869630559c1_knowledge_68d5216dcff624278e17e236” which has 300K objects，and restart the pod, the memory is only 519M.
2

I’ve read the article, and based on my use case, storing all the vectors in memory would require approximately 1.89 GB—calculated as:

(300000 * ((1536 * 4) + (64 * 10)))/1024/1024/1024≈1.89 GB

Given that I also have several other classes in memory, it now makes sense why the total memory usage could easily exceed 3 GB. Does my understanding sound correct?

My main question now is:

according to the article, during querying, it should be possible to keep only a subset of vectors in memory while storing the rest on disk. I understand that I can set a large vectorCacheMaxObjects during import for optimal performance and then reduce it after the import is complete.

My question is: how exactly do I configure this setting after import? Is it done via an environment variable, or do I need to update the collection’s schema (e.g., through a REST API call or change code)? What’s the correct way to apply this change post-import?