Upload without indexing

msj242 · October 22, 2023, 5:25am

Is there a way to upload tons of data, then ask weaviate to build its index once all embeddings and data are uploaded hopefully increasing the speed of upload dramatically (my uploads are seeing 5*slower as the database grows to 1million 384d vectors for my setup).

DudaNogueira · October 23, 2023, 4:54pm

Hi!

We have been cooking some new features to address this. The idea is to have async index building. This, on top of GRPC will deliver improved import times.

Check it out:

github.com/weaviate/weaviate

Faster Imports and better import DX: Async vector index building

opened 09:43AM - 27 Aug 23 UTC

etiennedi

backlog

At the moment, a batch request is fully synchronous. This means it will only com…plete once everything is fully indexed. Instead, it should be possible to import async where only the object storage (and possibly the inverted index) is written synchronously. Vector indexing happens async in more organized batches. This should lead to: - Better CPU utilization by making sure the import queue does not run dry - Easier to avoid lock contention as some processes can happen in larger batches - Avoid import vs delete races and contention, i.e. the whole delete re-built logic can be rewritten To avoid the negative user impact of the index not being available immediately, we need to make sure that: - If feasible, a search while importing makes use of the partially built index and in addition brute-force objects which are not yet indexed - It is easily visible to the user whether an index is not fully built yet and when it turns ready. Clients should offer methods to wait until all is ready.

Let me know if this helps

msj242 · October 23, 2023, 6:16pm

Ah yes - this is what I am looking for. For massive data entry it would be extremely useful. Even as a flag for on/off building the index so setup is quick even if later additions are slow. But the solution suggested (ie: async) works too - more sophisticated.

Any clue timeline for feature? Like 1 month or 1-2 years?

DudaNogueira · October 23, 2023, 6:56pm

This feature should land soon.

Check the PR here: async: Introduce async indexing by asdine · Pull Request #3424 · weaviate/weaviate · GitHub

so you can keep track of it

msj242 · November 1, 2023, 3:59am

Thanks - looks like it has been merged and hopefully part of next release for anyone searching for this.

It states you need to use ASYNC_INDEXING=true, out of curiosity where would this flag be put? Is it in the docker container config file?

DudaNogueira · November 1, 2023, 6:28pm

That’s right!

You must have it set as your environment variable and then restart Weaviate.

More info: Release v1.22.0 - Async Indexing, Nested Object Support, gRPC API Support, Schema Repair, OIDC Group Auth, and many other improvements & fixes · weaviate/weaviate · GitHub

Samuel_Jude_S · May 11, 2024, 4:51am

Hi Duda
Can you let me know when does the ASYNC start to index then? I was able to import the data
Thanks!

DudaNogueira · May 12, 2024, 1:43pm

hi Samuel!

AFAIK, it will balance between the ingest operations and the index operations. The idea is to avoid having the load of ingesting and indexing running at the same time and not performing well on both. So I believe it will control the server load and keep the index queue grow when the ingestion is in a higher rate.

There is more info here too, as well on the aforementioned PR:

github.com/weaviate/weaviate

Faster Imports and better import DX: Async vector index building

opened 09:43AM - 27 Aug 23 UTC

closed 08:12PM - 06 May 24 UTC

etiennedi

planned-1.23

At the moment, a batch request is fully synchronous. This means it will only com…plete once everything is fully indexed. Instead, it should be possible to import async where only the object storage (and possibly the inverted index) is written synchronously. Vector indexing happens async in more organized batches. This should lead to: - Better CPU utilization by making sure the import queue does not run dry - Easier to avoid lock contention as some processes can happen in larger batches - Avoid import vs delete races and contention, i.e. the whole delete re-built logic can be rewritten To avoid the negative user impact of the index not being available immediately, we need to make sure that: - If feasible, a search while importing makes use of the partially built index and in addition brute-force objects which are not yet indexed - It is easily visible to the user whether an index is not fully built yet and when it turns ready. Clients should offer methods to wait until all is ready.

Topic		Replies	Views
Optimizing Object Import Performance in Large Weaviate Classes with HSNW Indexing Support developer-experience	1	430	February 9, 2024
Optimizing Weaviate Class Indexing for Efficient Bulk Uploads and Hybrid Searches Support	3	325	February 7, 2024
Data upload speed is low Support	4	278	January 18, 2024
Parallel Batch Operations and Consistency Level Support developer-experience	2	443	January 30, 2024
Indexing embeddings taking too long. What am I doing wrong? Support	4	1336	September 27, 2024

Upload without indexing

Related topics