Subject: Unable to Modify Default Query Limits in Weaviate Deployment. Retreving only 100 results

Support
1

I am facing an issue with my Weaviate deployment where I am unable to change the default query limits for search results. Despite adjusting both the QUERY_DEFAULTS_LIMIT and QUERY_MAXIMUM_RESULTS parameters in my configuration, as well as trying different values and even omitting them entirely, the number of search results returned by Weaviate remains fixed at 100.

Below is the relevant excerpt from my Weaviate configuration:

---
version: '3.4'
services:
  weaviate:
    command:
      - --host
      - 0.0.0.0
      - --port
      - '1000'
      - --scheme
      - http
    image: semitechnologies/weaviate:1.22.0
    ports:
      - "1000:1000"
    restart: on-failure:0
    volumes:
      - C:/backups:/tmp/backups
    environment:
      QUERY_DEFAULTS_LIMIT: 10000
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      ENABLE_MODULES: 'backup-filesystem,backup-s3'
      AWS_ACCESS_KEY_ID: ''
      AWS_SECRET_ACCESS_KEY: ''
      AWS_SESSION_TOKEN: ''
      AWS_REGION: 'us-east-1'
      BACKUP_S3_BUCKET: 'bucket-dev'
      BACKUP_S3_PATH: 'weaviate/backups/'
      BACKUP_S3_ENDPOINT: ''
      BACKUP_FILESYSTEM_PATH: '/tmp/backups'
      CLUSTER_HOSTNAME: 'node1'
      QUERY_MAXIMUM_RESULTS: 10000
      DISK_USE_READONLY_PERCENTAGE: 100
---

I would greatly appreciate any insights or suggestions on how to resolve this issue and successfully adjust the default query limits in Weaviate.

Thank you in advance for your assistance.

2

Hi @Sandip !

I was not able to reproduce part of this.

setting QUERY_MAXIMUM_RESULTS to 5, and adding 10000 objects:

import weaviate
client = weaviate.connect_to_local()

from weaviate import classes as wvc

collection = client.collections.create(
    name="TestQueryLimit",
    vectorizer_config=None,
    properties=[
        wvc.config.Property(name="text", data_type=wvc.config.DataType.TEXT)
    ]
)

from weaviate.util import generate_uuid5
objects = []
for i in range(10000):
    objects.append(
        wvc.data.DataObject(
            {"text": f"Object {i}"},
            uuid=generate_uuid5(f"item-{i}")
        )
    )

collection.data.insert_many(objects)

Now, whenever I query:

collection.query.bm25(query="Object", limit=3000)

I get the following error:

WeaviateQueryError: Query call with protocol GRPC search failed with message explorer: get class: vector search: invalid pagination params: query maximum results exceeded.

This is expected.

However, for QUERY_DEFAULTS_LIMIT I tried setting it to 25, and when performing a query using bm25, for example:

len(collection.query.bm25(query="Object", limit=None).objects)

it indeed didn’t respect that limit from server, and was only returning 10 objects by default.

the same happens for this query:

len(collection.query.fetch_objects().objects)

I will need to ask this internally and will get back here when I get an answer.

Thanks for reporting!

3

Hi @Sandip !

I found out that that limit, QUERY_DEFAULTS_LIMIT will only take effect at the rest endpoint, so:

http://localhost:8080/v1/objects

I understand that from a user perspective, setting that limit should take effect in all queries: rest, grpc and graphql.

I have raised that internally.

Thanks!

4

Hello!

Thank you for raising that problem internally, I am facing the same issue.

On the other hand, I was wondering if it would be possible to pass filters in the cursor api ?

Also, do you mind updating the doc: until the fix to GRPC i completed?

  • The number of objects you can retrieve is limited. A single query returns up to QUERY_MAXIMUM_RESULTS. If the sum of offset and limit exceeds QUERY_MAXIMUM_RESULTS, Weaviate returns an error. To change the limit, edit the QUERY_MAXIMUM_RESULTS environment variable. If you increase QUERY_MAXIMUM_RESULTS, use the lowest value possible to avoid performance problems.

Thank you!

5

Thanks for spotting this.

Note: Our team is working on clients so all queries get persistent with QUERY_DEFAULTS_LIMIT in server.

A fix should come soon :star_struck:

6

Hi! You cannot pass filter to the cursor api. as it will iterate over all objects.

If you want iterate over filtered objects, you will need to do a search:

The issue is not in GRPC nor Weaviate, but in the client that will be fixed soon.

What exactly would need changing on the doc? :thinking: