Feature suggestion: status messages during large bulk imports or cluster status update in console

General
1

Today I did my first successful bulk import using the importMany() method. It worked fine, but my Node.js script went “radio silent” during the entire call. Thanks fully it was only about 5 minutes, but still, during that time I worried that my script had crashed.

I would like to see a bulk import call that either opened a websocket that delivered status updates of the “imported X out of a total of Y” messages, or perhaps some kind of promise based callback method or whatever works easiest for your devs to create.

If that’s just too big of an ask and the following idea would be easier, an activity update in the WCS console in the cluster detail area would be fine too. Just something, somewhere that we can check to make sure a monolithic bulk import hasn’t just died or got “stuck” would be a life saver.

I know this is a tricky and awkward thing to implement, but it’s really important. I only did about 8000 objects today, but in the future when it reaches tens of thousands or more, it will be a critical workflow item.

2

Nice!

There is a feature “being cooked” that I believe will help a lot here.

Consider that now we have (in Beta) the ASYNC_INDEXING. This will asynchronously build your vector index.

For for example, whenever you ingest data - and bring along your your vectors - Weaviate will write all data in a “queue” and then process in asynchronously.

The new feature is around ASYNC_VECTORIZING. So, as you can imagine, even if you rely on Weaviate to do the vectorization for you, you just throw your data IN, and Weaviate will take care of not only vectorizing it, but also indexing.

Right now, the best way to keep track of the imports, is by controlling it a import time, or querying the collection from a different client instance.

And glad to hear you are making progress! Whenever you need help, we are here to help you succeed ! :slight_smile:

Thanks for the suggestions!

1 Like
3

Oh, that’s great. Any ETA on when it will be available?

4

Unfortunately no. :frowning:

The ASYNC_VECTORIZING is still in the draft phase.

1 Like