What is the recommended way to handle paginated results to avoid duplicates?

As per Weaviate docs:

The pagination setup is not stateful. If the database state has changed between retrieving two pages there is no guarantee that your pages cover all results.

I am having an issue right now where, when someone publishes their new Player X then another Player Y ranking could go either higher or lower from the search relevance.

If Player Y (ranked 89 before the update) has already appeared in page 81-90 (currently loaded) and after an update this player went to ranked 92 thus, this player will be part of page 91-100 which will appear like a duplicate record for the user viewing our search page.

It is more noticeable on our side since we are using infinite scroll (business rule).

Is there like a recommended way on how to handle this on the application side? Any suggestions to move forward?

Hi @junbetterway - I’ve moved this to Support as I think that fits better with the topic.

As to your question, I’m not sure what the best solution will be - I will ask around and get back to you.

1 Like