# Memory consumption increases after embedding deletion

**URL:** <https://forum.weaviate.io/t/memory-consumption-increases-after-embedding-deletion/4407>\
**Category:** Support\
**Created:** [October 2, 2024, 11:54pm UTC](https://forum.weaviate.io/t/memory-consumption-increases-after-embedding-deletion/4407 "2024-10-02T23:54:43Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![fairymane](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/fairymane/32/1029_2.png) [@fairymane](https://forum.weaviate.io/u/fairymane)\
**Post date:** [October 2, 2024, 11:54pm UTC](https://forum.weaviate.io/t/memory-consumption-increases-after-embedding-deletion/4407/1 "2024-10-02T23:54:43Z")

</div>

### Description

Hi Team, I observed a situation that after deleting a certain amount of embeddings, the memory consumption reversely increases. What could be the reason?  
For more detail, On a weaviate cluster (on K8s), I reduced embeddings from 203M to 195M (use  
`collection.data.delete_many with filter, and set the **consistency_level=ConsistencyLevel.ONE** `  
).  
The memory consumption, instead of decreases, increased from ~514Gb to 621 Gb…

The deletion happens more than 18 hrs ago and no other operation was performed after deletion. The memory consumption stays at the higher level (621 Gb) instead of coming down… Any idea on how this could happen? Is the garbage collection daemon suppose to run in the background?

### Server Setup Information

- Weaviate Server Version: 1.25.0
- Deployment Method: k8s
- Multi Node? Number of Running Nodes: 7
- Client Language and Version: V4
- Multitenancy?: No

### Any additional Information\>

> #set env variable QUERY\_MAXIMUM\_RESULTS: 20000  
> …  
> collection = weaviate\_client.collections.get(collection\_name).with\_consistency\_level(  
> consistency\_level=ConsistencyLevel.ONE  
> )  
> delete\_response = collection.data.delete\_many(where=Filter.by\_property(“log\_name”).equal(log\_name))  
> …

Besides that, while running deletion, I observed lots of error like

> “Query call with protocol  
> GRPC delete failed with message \<AioRpcError of RPC  
> that terminated with:  
> status = StatusCode.UNKNOWN  
> details = “batch delete: cannot find  
> objects: find matching doc ids in shard  
> “05i1dRqNYiPq”: class Data\_discovery\_production has  
> no physical shard “05i1dRqNYiPq”: failed to execute  
> query: leader not found””

or

> "Query call with protocol  
> GRPC delete failed with message \<AioRpcError of RPC  
> that terminated with:  
> status = StatusCode.DEADLINE\_EXCEEDED  
> details = “Deadline Exceeded”  
> debug\_error\_string = “UNKNOWN:Error  
> received from peer”

This is the timeout config I set when initialize the Weaviate client

```auto
additional_config=AdditionalConfig(
                timeout=Timeout(init=180, query=360, insert=360),
            ),

```

What could be the reason? I use retry with exponential back-off that the deletion could eventually succeed but still see the memory increase issue as described above.

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [October 3, 2024, 4:38pm UTC](https://forum.weaviate.io/t/memory-consumption-increases-after-embedding-deletion/4407/2 "2024-10-03T16:38:30Z")

</div>

Hi!

There was a lot of changes after 1.25.X regarding this.

So my advice would be upgrading.

On top of that, it is interesting to have visibility on tombstones, and change the related environment variables for it:

> **[Environment variables | Weaviate](https://weaviate.io/developers/weaviate/config-refs/env-vars)**
>
> To configure Weaviate in a Docker or a Kubernetes deployment, set these environment variables

Let me know if this helps.

Thanks!