Disk usage for weaviate and tombstone cleanup

Support
1

I have a test database on a local machine for experiments.
With the following settings:

    environment:
      - LOG_LEVEL=info
      - LIMIT_RESOURCES=true
      - CLUSTER_HOSTNAME=weaviate
      - CLUSTER_GOSSIP_BIND_PORT=7100
      - CLUSTER_DATA_BIND_PORT=7101
      - IMAGE_INFERENCE_API=http://i2v:8082
      - AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true
      - PERSISTENCE_DATA_PATH=/var/lib/weaviate
      - DEFAULT_VECTORIZER_MODULE=img2vec-neural
      - ENABLE_MODULES=img2vec-neural
      - ASYNC_INDEXING=true
      - PROMETHEUS_MONITORING_ENABLED=true
      - TOMBSTONE_DELETION_MIN_PER_CYCLE=30000
      - TOMBSTONE_DELETION_MAX_PER_CYCLE=300000
      - QUERY_DEFAULTS_LIMIT=40
      - PERSISTENCE_LSM_MAX_SEGMENT_SIZE=50GB

I have about 4 million vectors written, the Estimated Sizes LSM stores value shows 772 GB usage and it almost matches the real disk usage value.

I did a large deletion of vectors, I deleted about 600 thousand vectors, but the size of LSM stores did not change at all, and the occupied disk space did not change either.
Although during the deletion there were messages in the log like:

{"action":"tombstone_cleanup_complete","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","class":"Test","duration":45083524945,"level":"info","msg":"class Test: shard 4FOgY9AQitCU: completed tombstone cleanup in 45.083524945s","shard":"4FOgY9AQitCU","time":"2025-01-23T15:21:18Z","tombstones_in_cycle":162375,"tombstones_total":162375}
{"action":"hnsw_condensing_complete","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","level":"info","msg":"completed hnsw condensing","time":"2025-01-23T15:21:18Z"}

It was expected that after deletion the disk space would be freed.
Сan you find out more about how exactly the deletion happens and why the disk space never decreases?

Server Setup Information

  • Weaviate Server Version: 1.28.2
  • Deployment Method: docker
  • Multi Node? Number of Running Nodes: One node
  • Client Language and Version: python, 4.9.2
  • Multitenancy?: no
2

hi @ilsg !!

That’s strange. The size should reduce after a the tombstone clean up.

Can you confirm that after some time the disk size still has not decreased?

Thanks!

3

Yes, I deleted several hundred thousand more vectors to make the cleanup happen, but after that the size of the data on the disk did not decrease either.
Data in the log after cleaning in info mode:

{"action":"tombstone_cleanup_complete","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","class":"Test","duration":44520095647,"level":"info","msg":"class Test: shard 4FOgY9AQitCU: completed tombstone cleanup in 44.520095647s","shard":"4FOgY9AQitCU","time":"2025-01-26T16:10:47Z","tombstones_in_cycle":134207,"tombstones_total":134207}
{"action":"hnsw_condensing_complete","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","level":"info","msg":"completed hnsw condensing","time":"2025-01-26T16:10:48Z"}
Second tombstone_cleanup
{"action":"hnsw_condensing_complete","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","level":"info","msg":"completed hnsw condensing","time":"2025-01-26T16:16:17Z"}
{"action":"tombstone_cleanup_complete","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","class":"Test","duration":30373802263,"level":"info","msg":"class Test: shard 4FOgY9AQitCU: completed tombstone cleanup in 30.373802263s","shard":"4FOgY9AQitCU","time":"2025-01-26T16:16:18Z","tombstones_in_cycle":87434,"tombstones_total":87434}

I also switched the log mode to debug, here are some lines that might be of interest:

{"action":"tombstone_cleanup_progress","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","class":"Test","level":"debug","msg":"class Test: shard 4FOgY9AQitCU: 3000000/4600858 nodes processed","processed_nodes":3000000,"shard":"4FOgY9AQitCU","time":"2025-01-26T16:25:28Z","total_nodes":4600858}
{"action":"attach_tombstone_to_deleted_node","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","level":"debug","msg":"found a deleted node (1833291) without a tombstone, tombstone was added","node_id":1833291,"time":"2025-01-26T16:25:35Z"}
{"action":"lsm_memtable_flush_complete","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","class":"Test","index":"test","level":"debug","msg":"flush and switch took 11.171313ms\n","path":"/var/lib/weaviate/test/4FOgY9AQitCU/lsm/property_post_id","shard":"4FOgY9AQitCU","time":"2025-01-26T16:25:42Z","took":11171313}
{"action":"tombstone_cleanup_complete","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","class":"Test","duration":49417121838,"level":"info","msg":"class Test: shard 4FOgY9AQitCU: completed tombstone cleanup in 49.417121838s","shard":"4FOgY9AQitCU","time":"2025-01-26T16:25:49Z","tombstones_in_cycle":152320,"tombstones_total":152320}
{"action":"lsm_precompute_disk_segment_build_bloom_filter_primary","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","class":"Test","index":"test","level":"debug","msg":"building bloom filter took 6.54414ms\n","path":"/var/lib/weaviate/test/4FOgY9AQitCU/lsm/property_uid/segment-1737908620745905701_1737908681565500141.db","shard":"4FOgY9AQitCU","time":"2025-01-26T16:25:49Z","took":6544140}
{"action":"lsm_precompute_disk_segment_build_bloom_filter_primary","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","class":"Test","index":"test","level":"debug","msg":"building bloom filter took 3.035396ms\n","path":"/var/lib/weaviate/test/4FOgY9AQitCU/lsm/property_post_id/segment-1737908559463092956_1737908681523994626.db","shard":"4FOgY9AQitCU","time":"2025-01-26T16:25:52Z","took":3035396}
{"action":"lsm_replace_compacted_segments_blocking","build_git_commit":"5a3991d","build_go_version":"go1.22.10","build_image_tag":"v1.28.2","build_wv_version":"1.28.2","class":"Test","index":"test","level":"debug","msg":"replacing compacted segments took 3.113686ms","path_left":"/var/lib/weaviate/test/4FOgY9AQitCU/lsm/property_uid/segment-1737908399384048762.db","path_right":"/var/lib/weaviate/test/4FOgY9AQitCU/lsm/property_uid/segment-1737908681565500141.db","segment_index":3,"shard":"4FOgY9AQitCU","time":"2025-01-26T16:25:56Z","took":3113686}

This is a big problem for me because I use fast m2 nvme drives for maximum performance and they run out of space because the data size does not shrink after deletion.