The problem is that this happens as well for vector & hybrid search, and same data can sometimes work and sometimes not on the same query, so for a tenant with 10k data, I get 6k maximum with hybrid search, and if I removed the collection and insert it again, it can be a different number than 6k (e.g. it was 1.3k before), so I feel like there is a bit of inconsistency here.

I believe this is the case of a higher efConstruction. From here:

HNSW is an approximate nearest neighbour algorithm. This means there is always a chance a vector will not be found in a search even though it would be found via brute force (using where filter).

The solution to this problem is to re-index your data with higher efConstruction parameter (for example efConstruction=512) that way your newly built graph will have higher number of node candidates that will be considered during vector search.