Adding removing objects with the same vectors

[Lucas_Sajdak]

we are removing and adding thousands of documents on daily basis. It often happens that inserted documents have the same vector value like already existing records (could it be a problem that we insert identical embedding several times into HNSW?). Then we remove old set of documents entirely (but we can see that they are not removed immediately, we can search for them).

Additionanly we can observe that from 3 replicas we are using all of them reports different storage used: 77, 81, 86%. Is there a way we can try to “sync” it somehow? Regenerating all those data might be quite costly. Could you provide us a hint how to fix replicas’ state?

[Lucas_Sajdak]

not sure why it was split into another topic. It was originally part of the Constant storage drain without increasing number of vectors
We are working with the same cluster.

[antas-marcin]

@Lucas_Sajdak

Additionanly we can observe that from 3 replicas we are using all of them reports different storage used: 77, 81, 86%. Is there a way we can try to “sync” it somehow?

I think that there’s no possibility now to do this operation, those should now be compacted independently.