GraphQL sorting issue?

Support
1

What could be the problem here? I have an article index, with a pubDate of datatype “date”, when I do a search with a limit of 10 results, I get a different top 10 then when i do a limit of a 1000 results, but I sort by the same descending date??

When I do a search: POST /v1/graphql/
{“query”:“{ Get {ArticleTest(limit: 10, sort:{order:desc,path:["pubDate"]}, nearText: {concepts: ["brand"], distance: 0.7}) {title,language,edition,author,pubDate,wordCount,body,source,sourceid,_additional {id,certainty}}}}”}

I get a different top 10 then when I do:

{“query”:“{ Get {ArticleTest(limit: 1000, sort:{order:desc,path:["pubDate"]}, nearText: {concepts: ["brand"], distance: 0.7}) {title,language,edition,author,pubDate,wordCount,body,source,sourceid,_additional {id,certainty}}}}”}

I did a full reindex, tried a different sort field (string), same problem…

I use version 1.22.5

2

Hi @edwineve, and welcome!

From what I can see, your search is based on nearText - so that causes Weaviate to grab the (max: n) objects based on vector distance to the query (brand, in this case).

The sorting by date happens after, and separately, to the search. In other words, sorting overrides the natural order of the vector search here.

So what is probably happening is that there are objects in the 990 additional search results, which have an earlier publication date. (Or later? :thinking: I always get ascending & descending dates confused).

Does that help?

Source:

3

hmmmm, that’s not a very intuitive way of quering with a sort, probably done that way because of performance. To me it was not clear reading the documentation that you do a query, get the top N results by certainty and order those by the sorting property… Is that always the case with vector search engines?

4

It’s done that way because vector search has a natural order, which is the degree of similarity.

So all vector searches produce ordered results, as do keyword (BM25) searches. This is different to filters, which only include or exclude results (like WHERE clauses in SQL).

The only way to apply sorting after a keyword or vector search is to override these vector or keyword search orders with the sort. I hope that makes sense!