Inconsistency between object store and index store

When a new object is inserted to a shard, it’s first get inserted to objectStore LSM, then reverted index, then vector index. It is possible that in some rare cases that the object store insert succeeded, but the index store inserts failed, that would cause discrepancy between index store and object store?

From user point of view, it’s possible that an insert operation failed, but they still see the object when they list all objects in the collection?

Hi @litlig !

This is possible, and our team just found some issue that happens on some rare ocasions. Check discussion here:

1 Like

thanks @DudaNogueira , however I’m not quite sure if the linked index corruption issue is the caused by this.

I took a more detailed look into the implementation though, it looks like object store and index store have their own WAL, so if the insert fails to get to index store’s WAL, it’ll never be indexed? so the discrepancy can get larger over time? I’m wondering if there is any mechanism to bring them to sync?

Hi @litlig, using async indexing, the vector index may eventually catch up. But there are chances of generating discrepancies.
The issue @DudaNogueira referenced while not specifically tackling this part it’s one of the first tasks towards solving this potential situations.

1 Like