Vector indexing latency and search inconsistency during high-volume batch imports

Hey everyone, I have been encountering an unexpected latency spike while batch importing text embeddings into our self-hosted Weaviate cluster. We are currently migrating user interaction logs and structured profiles from our instaapro module, where each data object contains short descriptive text along with several custom metadata tags. The initial chunks ingest smoothly, but once the collection exceeds a few hundred thousand objects, memory allocation on our HNSW index climbs sharply and standard queries start timing out intermittently.

In addition to the throughput bottleneck, hybrid search returns inconsistent distance metrics between freshly updated records and older existing shards. I have already verified our batch concurrency limits and memory thresholds on the node, but I am wondering if dynamic index construction during concurrent reads might be the root cause, or if tweaking the vector cache allocation and efConstruction parameters would be the recommended fix here. Any insight into stabilizing ingestion for this type of workload would be greatly appreciated.

Hi @michael_turner

Welcome to our forum, and it’s nice to have you here :tada:

From what you describe, this looks like a resource or configuration issue rather than a single root cause.

It’s expected for memory to climb during ingest, since HNSW keeps the graph and the vectors in memory. What you need to stabilise it is compression, so my recommendation is RQ-8. See here: Rotational Quantization (RQ) | Weaviate Documentation

The timeouts are most likely caused by pressure on the memory. I would need more logs and details of your cluster to know for sure, but make sure ASYNC_INDEXING=true is set, set GOMEMLIMIT as well, and use smaller batches (or better use this server-side batching).

For the inconsistent distances, Hybrid search scores are normalised per query, so they can’t be compared across queries. BM25 is also scored per shard. On top of that, with ASYNC_INDEXING=true there is a short delay before new or updated objects show up in vector search while they are still in the indexing queue.

It’s hard to narrow down the root cause yet, as there are many areas involved in the same situation. Please share more about:

Weaviate version, cluster nodes, object count, vector dimensions, number of shards, whether compression is on, memory limit and GOMEMLIMIT, batch size, and any OOM or timeout lines from the logs. All of that is important.

Feel free to use these open-source tools (community tools) if you’d like a UI to help:

Kind regards,

Mohamed Shahin
Weaviate Admin
Weaviate & LinkedIn

Michael’s breakdown covers the likely memory angle well, so I’d focus first on the distance inconsistency. It’s worth confirming whether the older shards and freshly updated records land in different shards, since Hybrid scores are normalized per query and BM25 is scored per shard, which can make cross-shard comparisons misleading. With ASYNC_INDEXING enabled there’s also a lag before queued objects appear in vector search, so timing your checks after the queue drains would give cleaner results. Smaller server-side batches plus GOMEMLIMIT should reduce the timeout pressure while you isolate which factor dominates.