Retrieving “Adjacent” Chunks for Better Context

In my RAG application I use Flowise to upsert a text file which is then chunked into documents of fixed chunk size. The embeddings for these chunks are stored in Weaviate. I ask a question via Flowise, which sends the question’s embeddings to Weaviate as a query. Weaviate retrieves the text chunk with the closest vector similarity to my question. All good. But how can I then query Weaviate to also retrieve the two text chunks which are located before and after that chunk in the original file? The idea is to use small chunks in the VDB for precision, but to bring (selectably) wider context back into the LLM for better reasoning.

Hi Mike!

This is not possible. Also, the query should already bring those chunks, if they are relevant enough to your query, considering you have small chunks, you should be able to pass forward to the generation a bigger number of chunks.

I believe you want to tweak the chunking size and the overlap, so it has enough context before and after.

Have you seen this material we have on that subject?

Let me know if this helps.

Hi Duda,

Let me try to explain more clearly. I want to use small chunks because it gives more specific similarity search results – increasing the chunk size is therefore not an option, and adjusting overlap does not solve the problem. Whenever I get a ‘hit’ on cosine similarity for a chunk, I want to retrieve that chunk and also the two chunks which are contiguous to that chunk. So, for example, let’s say I upload a book which is chunked and embedded at paragraph level. When I query the VDB if my query matches a particular paragraph I would ideally like to retrieve the whole page of the book, not just the matching paragraph. But I don’t have the concept of a page, and so as a work-around I want to get the paragraphs immediately before and after the matching paragraph. Let’s says I query the VDB and get a hit on paragraph “B”. I would also want to retrieve paragraphs A and C – even though they may not match at all for cosine similarity - I want the whole block of text A-B-C.

I guess what this comes down to is how Weaviate stores the original text; whether contiguous chunks of text have a numbering system which reflects the order in which they were upserted, so that if the three paragraphs ABC were numbered 123 then paras A and B could perhaps be retrieved as para 2-1=1 and para 2+1=3. Vectara does something similar to this: “x sentences before and x sentences after” a matching sentence.

One thing that could be worth trying, is doing 2 queries.

So first query, will get the topX objects.

When you chunk those, you save the position of the chunk.

Based on the topX chunks, you do a second query to also select the -1 +1 chunk position of the topX chunks.

The downside of this approach is that for generating the answer, you would have to consume the generative api yourself at code level, as you would need to “reaugment” your context window.

Let me know if this helps.

Thanks Duda but I’m not sure I follow. Do you mean that Weaviate does not store the ordinal number of the chunks as they are upserted? Or are you saying that the matching chunk does have a stored ordinal number and that I should therefore:

  1. Query Weavitate and get the chunk that matches by cosine sim;
  2. Query Weaviate for that matching chunk’s ordinal number x; and
  3. Use that ordinal number to query Weaviate for the (possibly non-matching) chunks of ordinal number x-1 and x+1?

(To make things simple let’s just assume that Top_k=1 – only one matching chunk to be retrieved, plus the chunk before and after it)