# Weaviate using tons of CPU during "tombstone\_cleanup\_begin" **URL:** \ **Category:** Support\ **Tags:** developer-experience, technical\ **Created:** [October 30, 2024, 2:29am UTC](https://forum.weaviate.io/t/weaviate-using-tons-of-cpu-during-tombstone-cleanup-begin/7290 "2024-10-30T02:29:32Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![00.lope.naughts](https://avatars.discourse-cdn.com/v4/letter/0/df705f/32.png) [@00.lope.naughts](https://forum.weaviate.io/u/00.lope.naughts)\ **Post date:** [October 30, 2024, 2:29am UTC](https://forum.weaviate.io/t/weaviate-using-tons-of-cpu-during-tombstone-cleanup-begin/7290/1 "2024-10-30T02:29:32Z")
### Description I am using local docker container weaviate instance on a Mac. I have test run with it for past week and have \>1.2m doc with embedding vectors. I am on “bring my own vector” case. What I noticed is one day, when I started out the weaviate instance, it will use single digit % in cpu, but after roughly 4-5min, it will ramp up to 500% (I have 12 cores). When I looked at the log, I highly suspect it is running “tombstone\_cleanup\_begin” (see below for more details). I don’t notice ever this has caused so much CPU and going at it for so long (still waiting…). Is this normal? I did make a lot of deletions. This is only a dev env, we will eventually expect this to be either on weaviate cloud, or roll our own on GCP. But I would like to understand if what’s causing this and if this could disrupt service in a deployment env. Any advice or debug tip will be appreciated. (we are new and still poking around in dev, and touching a little on scalability matter). ### Server Setup Information - Weaviate Server Version: 1.27.0 - Deployment Method: docker version 4.34.3 (170107) on macOS 14.2.1 (23C71) - Multi Node? Number of Running Nodes: 1 - Client Language and Version: ? - Multitenancy?: No ### Any additional Information > 2024-10-29 21:49:29 {“action”:“lsm\_recover\_from\_active\_wal”,“build\_git\_commit”:“6c571ff”,“build\_go\_version”:“go1.22.8”,“build\_image\_tag”:“”,“build\_wv\_version”:“”,“class”:“Listing\_Text”,“index”:“listing\_text”,“level”:“warning”,“msg”:“empty write-ahead-log found. Did weaviate crash prior to this or the tenant on/loaded from the cloud? Nothing to recover from this file.”,“path”:“/var/lib/weaviate/listing\_text/48GOdFIN20rh/lsm/property\_baths\_searchable/segment-1730088224356409675”,“shard”:“48GOdFIN20rh”,“time”:“2024-10-30T01:49:29Z”} > 2024-10-29 21:49:30 {“action”:“hnsw\_prefill\_cache\_async”,“build\_git\_commit”:“6c571ff”,“build\_go\_version”:“go1.22.8”,“build\_image\_tag”:“”,“build\_wv\_version”:“”,“level”:“info”,“msg”:“not waiting for vector cache prefill, running in background”,“time”:“2024-10-30T01:49:30Z”,“wait\_for\_cache\_prefill”:false} > 2024-10-29 21:49:30 {“build\_git\_commit”:“6c571ff”,“build\_go\_version”:“go1.22.8”,“build\_image\_tag”:“”,“build\_wv\_version”:“”,“level”:“info”,“msg”:“Completed loading shard listing\_text\_48GOdFIN20rh in 1.691448875s”,“time”:“2024-10-30T01:49:30Z”} > 2024-10-29 21:49:33 {“action”:“hnsw\_prefill\_cache\_async”,“build\_git\_commit”:“6c571ff”,“build\_go\_version”:“go1.22.8”,“build\_image\_tag”:“”,“build\_wv\_version”:“”,“level”:“info”,“msg”:“not waiting for vector cache prefill, running in background”,“time”:“2024-10-30T01:49:33Z”,“wait\_for\_cache\_prefill”:false} > 2024-10-29 21:49:33 {“build\_git\_commit”:“6c571ff”,“build\_go\_version”:“go1.22.8”,“build\_image\_tag”:“”,“build\_wv\_version”:“”,“level”:“info”,“msg”:“Completed loading shard listing\_image\_48d5dFITJv7f in 4.504469669s”,“time”:“2024-10-30T01:49:33Z”} > 2024-10-29 21:49:57 {“action”:“hnsw\_vector\_cache\_prefill”,“build\_git\_commit”:“6c571ff”,“build\_go\_version”:“go1.22.8”,“build\_image\_tag”:“”,“build\_wv\_version”:“”,“count”:401613,“index\_id”:“main”,“level”:“info”,“limit”:2000000,“msg”:“prefilled vector cache”,“time”:“2024-10-30T01:49:57Z”,“took”:26793010096} > 2024-10-29 21:50:37 {“action”:“hnsw\_vector\_cache\_prefill”,“build\_git\_commit”:“6c571ff”,“build\_go\_version”:“go1.22.8”,“build\_image\_tag”:“”,“build\_wv\_version”:“”,“count”:1221520,“index\_id”:“main”,“level”:“info”,“limit”:2000000,“msg”:“prefilled vector cache”,“time”:“2024-10-30T01:50:37Z”,“took”:64299448321} > 2024-10-29 21:54:29 {“action”:“tombstone\_cleanup\_begin”,“build\_git\_commit”:“6c571ff”,“build\_go\_version”:“go1.22.8”,“build\_image\_tag”:“”,“build\_wv\_version”:“”,“class”:“Listing\_Image”,“level”:“info”,“msg”:“class Listing\_Image: shard 48d5dFITJv7f: starting tombstone cleanup”,“shard”:“48d5dFITJv7f”,“time”:“2024-10-30T01:54:29Z”,“tombstones\_in\_cycle”:22162,“tombstones\_total”:22162} Focusing on last 2 lines, I think CPU was low up until tombstone\_cleanup\_begin when it ramped to 500% and after \>20min, it is still apparently working on it… since I dont see tombstone\_cleanup\_complete. ---
**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\ **Post date:** [November 1, 2024, 2:38pm UTC](https://forum.weaviate.io/t/weaviate-using-tons-of-cpu-during-tombstone-cleanup-begin/7290/2 "2024-11-01T14:38:09Z")
hi @00.lope.naughts !! You can set the up the aggressiveness of the tombstone cycles: > **[Environment variables | Weaviate](https://weaviate.io/developers/weaviate/config-refs/env-vars)** > > To configure Weaviate in a Docker or a Kubernetes deployment, set these environment variables You will want to tweak the `TOMBSTONE_DELETION_` variables. This blogpost also can help you understanding how Weaviate uses GC or run cycles: > **[GOMEMLIMIT is a game changer for high-memory applications | Weaviate](https://weaviate.io/blog/gomemlimit-a-game-changer-for-high-memory-applications)** > > Go 1.19 introduced GOMEMLIMIT, which completely changes how you can manage memory limits in Go. Learn how it helps Weaviate be more reliable. Let me know if that helps! ---
**Author:** ![00.lope.naughts](https://avatars.discourse-cdn.com/v4/letter/0/df705f/32.png) [@00.lope.naughts](https://forum.weaviate.io/u/00.lope.naughts)\ **Post date:** [November 1, 2024, 5:30pm UTC](https://forum.weaviate.io/t/weaviate-using-tons-of-cpu-during-tombstone-cleanup-begin/7290/3 "2024-11-01T17:30:21Z")
Thanks, I will check out those pages. I looked back at older logs and found the tombstone delete operations took only few sec to a few min even with \>10k in size. So the one that got stuck is not out of the norm in terms of size. what I remembered was I shutdown the container, maybe too soon after a large delete op, or maybe I was unlucky doing so during some preset cycle/schedule. I ended up just exporting all data, delete the instance, and just create a new one. I suspect something is corrupted, and I have these old collections saved on disk. and so far, I haven’t seen this happen again. perhaps, I shouldnt have just stop the weaviate container, but just “pause” it. ---
**Author:** ![00.lope.naughts](https://avatars.discourse-cdn.com/v4/letter/0/df705f/32.png) [@00.lope.naughts](https://forum.weaviate.io/u/00.lope.naughts)\ **Post date:** [November 12, 2024, 5:45pm UTC](https://forum.weaviate.io/t/weaviate-using-tons-of-cpu-during-tombstone-cleanup-begin/7290/4 "2024-11-12T17:45:49Z")
Unfortunately, I see my weaviate container instance stuck at CPU intensive cycle for doing tombstone cleanup again, and this time it isnt due to shutdown corruption. I noted that this round, I have caused it delete over 36k in ~3min span. I will take a look at TOMBSTONE\_DELETION\_\* to see what I can tune. But it does appear this is prune to happen if I try to delete a large # of docs within a short span of time.