Hey everyone, I have been encountering an unexpected latency spike while batch importing text embeddings into our self-hosted Weaviate cluster. We are currently migrating user interaction logs and structured profiles from our instaapro module, where each data object contains short descriptive text along with several custom metadata tags. The initial chunks ingest smoothly, but once the collection exceeds a few hundred thousand objects, memory allocation on our HNSW index climbs sharply and standard queries start timing out intermittently.
In addition to the throughput bottleneck, hybrid search returns inconsistent distance metrics between freshly updated records and older existing shards. I have already verified our batch concurrency limits and memory thresholds on the node, but I am wondering if dynamic index construction during concurrent reads might be the root cause, or if tweaking the vector cache allocation and efConstruction parameters would be the recommended fix here. Any insight into stabilizing ingestion for this type of workload would be greatly appreciated.