# Semantic search is not as good as I expected

**URL:** <https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963>\
**Category:** Support\
**Tags:** developer-experience\
**Created:** [July 7, 2024, 10:38am UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963 "2024-07-07T10:38:27Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![uma\_shankar](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/uma_shankar/32/1791_2.png) [@uma\_shankar](https://forum.weaviate.io/u/uma_shankar)\
**Post date:** [July 7, 2024, 10:38am UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/1 "2024-07-07T10:38:27Z")

</div>

I just started with verba and I am using ADAEmbedder

Query: “find me renovated apartment for less than 1 million”

The search does not care about 1 million at all. Also it does not consider “refurbished apartments” (as refurbished is a synonym of renovated).

How can I make this simple search work perfect ? Should I use some other Embedder or should I make a custom embedder for myself ?

I do not know about these embedders at all, as I am very new to LLMs

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [July 8, 2024, 12:06pm UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/2 "2024-07-08T12:06:24Z")

</div>

hi @uma_shankar !!

the results will depend on the vectorizer used, the content you have indexed (how it was chunked, some metadata, etc) and the query.

I am not sure that using a “natural language” filtering (less than 1 million) will indeed get you the results you expect. 🤔

You will, however, get the closest object from your query based on the vector distance.

Let me know if this helps!

Thanks!

---

<div class="post-metadata">

**Author:** ![uma\_shankar](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/uma_shankar/32/1791_2.png) [@uma\_shankar](https://forum.weaviate.io/u/uma_shankar)\
**Post date:** [July 8, 2024, 12:14pm UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/3 "2024-07-08T12:14:53Z")

</div>

Thanks I understand.

I do not know which Vectorizer is best for my case “property recommendation”.

Should I do any thing special regarding “metadata” during ingestion ?  
Or I just have to add “metadata” during the query ?

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [July 8, 2024, 6:26pm UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/4 "2024-07-08T18:26:05Z")

</div>

hi!

The vectorizer will depend on some experimentation, really. if it must be multilingual, the size of dimensions, etc.

Regarding metadata, consider for example 20 chunks. 10 for each document. You will want to store at least the main title of that document with each chunk.

Otherwise, for each chunk, you don’t add the meaning of those words per chunk.

With that said, and considering metadata, you want to make sure that each chunk has not only it’s content, but some metadata that can add context to what that chunk is related to.

Let me know if this helps.

If you want to learn more on chunking, check this docs:

> **[A brief introduction to chunking | Weaviate - Vector Database](https://weaviate.io/developers/academy/py/standalone/chunking/introduction)**
>
> \<!-- import imageUrl from '../../tmpimages/academyplaceholder.jpg';

Thanks!

---

<div class="post-metadata">

**Author:** ![Ghattas\_Salloum](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/ghattas_salloum/32/1863_2.png) [@Ghattas\_Salloum](https://forum.weaviate.io/u/Ghattas_Salloum)\
**Post date:** [July 10, 2024, 10:28am UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/5 "2024-07-10T10:28:37Z")

</div>

Hi,  
i already have experience with that same situation, after lots of testing i find out that vector search is not the best for property recommendations at all, what you will need to do is create set of features like price and size and locations and so one then use gpt-4 to extract the features from the query as json then use that json to search properties in sql database that will give way better results that using vector database  
I have already done that and thr difference is huge,  
the vector database is better of unstructured data like huge amounts text which i use weaviate for and it works really well

Hope that helps

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [July 11, 2024, 8:26pm UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/6 "2024-07-11T20:26:59Z")

</div>

thanks for sharing, @Ghattas_Salloum !!

---

<div class="post-metadata">

**Author:** ![Ghattas\_Salloum](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/ghattas_salloum/32/1863_2.png) [@Ghattas\_Salloum](https://forum.weaviate.io/u/Ghattas_Salloum)\
**Post date:** [July 12, 2024, 8:21am UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/7 "2024-07-12T08:21:19Z")

</div>

Sure any time

have a great one

---

<div class="post-metadata">

**Author:** ![uma\_shankar](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/uma_shankar/32/1791_2.png) [@uma\_shankar](https://forum.weaviate.io/u/uma_shankar)\
**Post date:** [July 15, 2024, 8:41am UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/8 "2024-07-15T08:41:28Z")

</div>

I did some more work on this. It looks like @Ghattas_Salloum is right. I used metadata(gpt extracted) search for filtering price, nr\_bedrooms, etc… I then relied on dense-search to match other criterias like amenities, patio, etc…

But the vector-search is not doing its job at all. For instance, if I search for a simple query “close to school”. The apartment which is close to school is getting less score than the other without any mention about school.

This is the only criteria I want to match. But it is not working. I wonder why.

---

<div class="post-metadata">

**Author:** ![Ghattas\_Salloum](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/ghattas_salloum/32/1863_2.png) [@Ghattas\_Salloum](https://forum.weaviate.io/u/Ghattas_Salloum)\
**Post date:** [July 15, 2024, 9:17am UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/9 "2024-07-15T09:17:24Z")

</div>

Hi,  
I think i can help with that too but with a different acpect,

When i add a property info i always add the lat and long coordinates for that property ,

So when i do the search i would use these coordinates with google maps api to check the schools near by or the bus station or restaurants etc…

in that way i don’t really have to depend on search to find that but google maps api can handle that really well.

other advice in your situation i would look at chucking methods used that might have huge impact on the results, i would parse all the data as csv, and each chunk will be a full csv row.

Hope that hepls

---

<div class="post-metadata">

**Author:** ![uma\_shankar](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/uma_shankar/32/1791_2.png) [@uma\_shankar](https://forum.weaviate.io/u/uma_shankar)\
**Post date:** [July 15, 2024, 9:25am UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/10 "2024-07-15T09:25:51Z")

</div>

Should the csv be like

“type: apartment, bedrooms: 2, bathrooms: 2” ?

---

<div class="post-metadata">

**Author:** ![Ghattas\_Salloum](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/ghattas_salloum/32/1863_2.png) [@Ghattas\_Salloum](https://forum.weaviate.io/u/Ghattas_Salloum)\
**Post date:** [July 15, 2024, 9:38am UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/11 "2024-07-15T09:38:43Z")

</div>

yes that would be correct, and you can add any more search criteria in csv,

one more important note in this case you shoud not depend on the search score at all just do the search and pass the results to gpt-4 along with chat history and last query and leave task of chose best match to gpt-4 believe me you will get best results ever.

---

<div class="post-metadata">

**Author:** ![uma\_shankar](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/uma_shankar/32/1791_2.png) [@uma\_shankar](https://forum.weaviate.io/u/uma_shankar)\
**Post date:** [July 16, 2024, 6:02pm UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/12 "2024-07-16T18:02:53Z")

</div>

The CSV solution works perfectly! You’re amazing. 😁

I’m feeling optimistic again about vector search.

---

<div class="post-metadata">

**Author:** ![Ghattas\_Salloum](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/ghattas_salloum/32/1863_2.png) [@Ghattas\_Salloum](https://forum.weaviate.io/u/Ghattas_Salloum)\
**Post date:** [July 17, 2024, 5:06pm UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/13 "2024-07-17T17:06:38Z")

</div>

Nice to hear that worked out for you

Any other help let me know 😀😀

---

<div class="post-metadata">

**Author:** ![uma\_shankar](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/uma_shankar/32/1791_2.png) [@uma\_shankar](https://forum.weaviate.io/u/uma_shankar)\
**Post date:** [July 18, 2024, 10:42am UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/14 "2024-07-18T10:42:11Z")

</div>

@Ghattas_Salloum I am using GPT34o for metadata extraction.

I used openAI embedder (text-embedding-3-small - 1536 dimensions), with good results.

I am thinking of using huggingface sentence transformers for embeddings, to reduce cost.

Will opensource model work as well for my property search ?

I do not know which model to choose either.

 ![image](https://canada1.discourse-cdn.com/flex027/uploads/weaviate/original/2X/0/08ff42040b93d3a67604abd535a5596195170140.png)

---

<div class="post-metadata">

**Author:** ![Ghattas\_Salloum](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/ghattas_salloum/32/1863_2.png) [@Ghattas\_Salloum](https://forum.weaviate.io/u/Ghattas_Salloum)\
**Post date:** [July 18, 2024, 7:09pm UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/15 "2024-07-18T19:09:34Z")

</div>

Well unfortunately i haven’t tested that much with open source so can’t advise on that one

i am sorry.

---

<div class="post-metadata">

**Author:** ![Gene\_Mc](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/gene_mc/32/1095_2.png) [@Gene\_Mc](https://forum.weaviate.io/u/Gene_Mc)\
**Post date:** [July 19, 2024, 5:04pm UTC](https://forum.weaviate.io/t/semantic-search-is-not-as-good-as-i-expected/2963/16 "2024-07-19T17:04:02Z")

</div>

For what it’s worth I am getting acceptable results with multi-qa-MiniLM-L6-cos-v1.  
My use case is primarily technical documentation embedding.

If you are using Docker, Weaviate has a pre-built image for this one. As well as others: [GitHub - weaviate/t2v-transformers-models: This is the repo for the container that holds the models for the text2vec-transformers module](https://github.com/weaviate/t2v-transformers-models?tab=readme-ov-file)