Adding special symbols in the chunks and Does it affect the retrieval to pick up the right chunks?

General
1

I have a markdown file and chunk it using LangchainNodeparser - Recursive splitter with markdown language. It returns the chunk with markdown symbols. Since LLM can natively speak the markdown format, I write directly without post processing them in weaviate. Does this special symbol influence the retrieval score in a negative way?

2

Hi @Sudhakar, Happy Friday :partying_face:

Welcome to our community — it’s great to have you here! :hugs:

From a Weaviate perspective, here’s helpful docs regarding symbols & tokenization and search:

Best regards,

Mohamed Shahin
Weaviate Support Engineer
(Ireland, GMT/UTC timezone)