Alternate to Reindexing of existing records

Support
1

As per the Weaviate documentation, a newly added property to a Collection/Schema doesn’t allow re-indexing of the data that is inserted in newly added properties of already existing objects.

image
image1065×519 45.3 KB

As per my understanding, updating the newly added properties of existing objects in Weaviate won’t index the data.
If we use Replace for this scenario. Does it index the data?

Server Setup Information

  • Weaviate Server Version: 1.25.7
  • Deployment Method:Docker
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: Python
  • Multitenancy?: Yes

@DudaNogueira

2

Hi @saurbhhsharrma,

The Replace operation allows for replacing the entire object but does not trigger re-indexing for new properties added after the initial import. Therefore, using Replace will not help in indexing the newly added properties of existing objects.

Currently, the only way to ensure the new property is indexed for existing data is to delete the collection and re-import it with the new property. This will build the entire vector space properly for the collection.

We are working on a re-indexing API for future releases to address this.

Have a lovely week!

3

Thanks @Mohamed_Shahin for the quick response.
I understand that Weaviate is working on Re-indexing API. However, deleting the collection and re-import it with the new property will be quite expensive operation in all aspects.

Assuming that we have objects on a single shard, can we delete the shard and re-import the data to re-index the newly added fields.

Additionally, would you be able to share the roadmap for the re-indexing API? I understand this might be a common issue, and any information you can provide would be greatly appreciated.

4

@saurbhhsharrma I am not sure if deleting the shard would work! however I am digging a bit into if there is a way around until the feature is implemented! Also seeking to know when we plan to release such feature. I will let you know once I hear something from our engineers.

5

Thanks alot @Mohamed_Shahin. It’ll be a great help for us.

6

Hey @saurbhhsharrma,

I hope you had a good weekend!

We’ve this open request Reindex API (Inverted Index) · Issue #2359 · weaviate/weaviate · GitHub
give it a thumb up as we are increasing its priority on our roadmap.

1 Like
7

Thanks @Mohamed_Shahin for sharing the roadmap. It will be a great help.

1 Like