Memory consumption increases after embedding deletion

Support
1

Description

Hi Team, I observed a situation that after deleting a certain amount of embeddings, the memory consumption reversely increases. What could be the reason?
For more detail, On a weaviate cluster (on K8s), I reduced embeddings from 203M to 195M (use
collection.data.delete_many with filter, and set the **consistency_level=ConsistencyLevel.ONE**
).
The memory consumption, instead of decreases, increased from ~514Gb to 621 Gb…

The deletion happens more than 18 hrs ago and no other operation was performed after deletion. The memory consumption stays at the higher level (621 Gb) instead of coming down… Any idea on how this could happen? Is the garbage collection daemon suppose to run in the background?

Server Setup Information

  • Weaviate Server Version: 1.25.0
  • Deployment Method: k8s
  • Multi Node? Number of Running Nodes: 7
  • Client Language and Version: V4
  • Multitenancy?: No

Any additional Information>

#set env variable QUERY_MAXIMUM_RESULTS: 20000
…
collection = weaviate_client.collections.get(collection_name).with_consistency_level(
consistency_level=ConsistencyLevel.ONE
)
delete_response = collection.data.delete_many(where=Filter.by_property(“log_name”).equal(log_name))
…

Besides that, while running deletion, I observed lots of error like

“Query call with protocol
GRPC delete failed with message <AioRpcError of RPC
that terminated with:
status = StatusCode.UNKNOWN
details = “batch delete: cannot find
objects: find matching doc ids in shard
“05i1dRqNYiPq”: class Data_discovery_production has
no physical shard “05i1dRqNYiPq”: failed to execute
query: leader not found””

or

"Query call with protocol
GRPC delete failed with message <AioRpcError of RPC
that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = “Deadline Exceeded”
debug_error_string = “UNKNOWN:Error
received from peer”

This is the timeout config I set when initialize the Weaviate client

additional_config=AdditionalConfig(
                timeout=Timeout(init=180, query=360, insert=360),
            ),

What could be the reason? I use retry with exponential back-off that the deletion could eventually succeed but still see the memory increase issue as described above.

2

Hi!

There was a lot of changes after 1.25.X regarding this.

So my advice would be upgrading.

On top of that, it is interesting to have visibility on tombstones, and change the related environment variables for it:

Let me know if this helps.

Thanks!