Advice Needed on Optimizing Vector Search in Weaviate

Support
1

Hey everyone!

I am fairly new to Weaviate and have been working on a project that involves using vector search to improve the accuracy of some recommendation algorithms. So far, it’s been pretty exciting, but I feel like there’s still a lot more potential that I could unlock with Weaviate.

Right now, I’m looking for some advice on a few things:

  1. Index configurations: Are there any specific settings you recommend to optimize performance when dealing with large datasets?
  2. Vector dimensions: Is there an ideal range of vector dimensions I should aim for, or does it really depend on the type of data I’m working with?
  3. Hybrid search: I’ve read a bit about hybrid search (vector + keyword) but haven’t tried it out yet. Has anyone had success with this, and if so, how did you set it up?

When I was searching about this, I came across these resources/articles 🔍 Seeking Assistance with Weaviate Vector Search UiPath Tutorial, however, they are quite good, but I wanted to learn more form community members.

Would love to hear your thoughts, suggestions, or even experiences with similar projects.

Appreciate any guidance you can share.

Thanks in advance

2

hi @halcyoncv !!

Glad you are enjoying your Weaviate journey :slight_smile:

  1. Regarding index configuration, some options that can help you on having a more performant vector index are ef efConstruction and maxConnections, as stated here: Vector indexes | Weaviate

  2. For the right embedded model and vector dimension, it will indeed depend on your data. For example, if you need multi language support, you will need to use a model that has support for the target languages. If your chunks are larger, a high dimension vector can capture more meaning, and give you better results. So before deciding on the model, it is interesting to do some tests/benchmarks.

  3. Once you have data indexed, you can start doing hybrid queries right away. Then you can start fine tuning it, specifying different alfa or different weights for specific properties.

Let me know if this helps.

Thanks!