Assistance Needed to Improve Weaviate's Vector Search Performance

General
1

Hello Everyone :hugs:,

Weaviate’s potent vector search skills are being used in a project I’m working on, but I could appreciate some help with some performance problems I’ve been having.

Project Synopsis:

I’m using Weaviate for my project to manage a sizable textual document dataset. Every document is vectorized with a transformer model that has been trained beforehand, and I use Weaviate’s vector search feature to find the most pertinent documents depending on user queries. There are presently about a million documents in the collection, and I predict that number will increase rapidly in the near future.

Problems Seen:

Search Latency: I’ve discovered that as the number of documents in the dataset grows, search latency increases noticeably. Although the initial searches were rather quick, some are now taking several seconds to provide results due to the magnitude of the collection.

Memory Usage: It appears that my Weaviate instance is using a lot of memory, and it seems to be getting worse as more content are added. This is starting to worry me, especially with my plans to expand the dataset.

Indexing Time: Adding new documents to the datasets is also taking a long time, which is affecting how effective my programme is as a whole.

What I’ve Attempting:

Parameter tuning: I’ve tried adjusting the size for the HNSW index and the amount of shards in the Weaviate set up, among other things, but the results have only been mediocre.

Batching: Although the implementation of batching for document indexing made a minor improvement, overall performance remains subpar.

Resource Allocation: The Weaviate instance was given more CPU and RAM, however the performance improvements did not make the extra expenses worthwhile.

Help Request:

I’m searching the community for any best practices or recommendations that could help with these problems. In particular:

  • Are there any setups or settings that are suggested to help reduce memory usage and search latency? :thinking:
  • Has anyone grown Weaviate to manage millions of documents successfully? If yes, what tactics worked well? :thinking:
  • Exist any potentially more effective methods for vector search or indexing? :thinking:

I also followed this :point_right: https://forum.weaviate.io/t/why-my-weaviate-vector-search-performance-is-low/mlops

Thank you :pray: in advance.