Assistance Needed to Improve Weaviate's Vector Search Performance

General
1

Hello Everyone :hugs:,

Weaviate’s potent vector search skills are being used in a project I’m working on, but I could appreciate some help with some performance problems I’ve been having.

Project Synopsis:

I’m using Weaviate for my project to manage a sizable textual document dataset. Every document is vectorized with a transformer model that has been trained beforehand, and I use Weaviate’s vector search feature to find the most pertinent documents depending on user queries. There are presently about a million documents in the collection, and I predict that number will increase rapidly in the near future.

Problems Seen:

Search Latency: I’ve discovered that as the number of documents in the dataset grows, search latency increases noticeably. Although the initial searches were rather quick, some are now taking several seconds to provide results due to the magnitude of the collection.

Memory Usage: It appears that my Weaviate instance is using a lot of memory, and it seems to be getting worse as more content are added. This is starting to worry me, especially with my plans to expand the dataset.

Indexing Time: Adding new documents to the datasets is also taking a long time, which is affecting how effective my programme is as a whole.

What I’ve Attempting:

Parameter tuning: I’ve tried adjusting the size for the HNSW index and the amount of shards in the Weaviate set up, among other things, but the results have only been mediocre.

Batching: Although the implementation of batching for document indexing made a minor improvement, overall performance remains subpar.

Resource Allocation: The Weaviate instance was given more CPU and RAM, however the performance improvements did not make the extra expenses worthwhile.

Help Request:

I’m searching the community for any best practices or recommendations that could help with these problems. In particular:

  • Are there any setups or settings that are suggested to help reduce memory usage and search latency? :thinking:
  • Has anyone grown Weaviate to manage millions of documents successfully? If yes, what tactics worked well? :thinking:
  • Exist any potentially more effective methods for vector search or indexing? :thinking:

I also followed this :point_right: https://forum.weaviate.io/t/why-my-weaviate-vector-search-performance-is-low/mlops

Thank you :pray: in advance.

2

hi,
I am using hybrid search of the Weaviate where I have kept value of alpha = 1 means it will do vector search.

But I am not getting accurate results for that.

I am using openai’s ada model to generate an embeddings.

I am using structure data in which there are two languages that is English and Danish.

example response:
query: I want results with value less than 10mm.
response will give values with less than 10mm also more than 10mm

example2:
query: I want results with value not less than 10mm.
here it gives values less than 10mm, it is not able to understand the meaning of not.

What can be the solution to improve the results of vector search.

3

Hi @Rohini_vaidya !

It is best to create a new thread for your question as they doesn’t seem related :thinking:

Is 10mm a property value?

Doing a hybrid query to filter a data set will not properly work, as Weaviate will get the closer vector for the semantic search, and then all objects that has the provided keywords, then fuse the distance and keyword.

We have developer a Query Agent, that could help here, as it will generate a Weaviate query based on your initial query:

Let me know if thish elps!

hi @Jackzz !! Sorry for the delay here, missed this one :frowning:

We have customers hosting billions of objects :slight_smile:

For anyone finding this thread:

For bigger deployments, you certainly will want to spread the load using a multi node deployment.