I have a markdown file and chunk it using LangchainNodeparser - Recursive splitter with markdown language. It returns the chunk with markdown symbols. Since LLM can natively speak the markdown format, I write directly without post processing them in weaviate. Does this special symbol influence the retrieval score in a negative way?
Hi @Sudhakar, Happy Friday
Welcome to our community — it’s great to have you here!
From a Weaviate perspective, here’s helpful docs regarding symbols & tokenization and search:
Best regards,
Mohamed Shahin
Weaviate Support Engineer
(Ireland, GMT/UTC timezone)