Query response time is very slow after several hours of inactivity

Support
1

Description

I’ve noticed that after several hours of inactivity (no queries or imports), the first query against my collection takes a very long time, upwards of 30 seconds. Then the second and third queries are a bit faster. Then, by about the 10th or 15th query, the query time is back to where I expect it to be. What explains this behavior? Is there a cache setting that could be misconfigured?

Information about my collection:

  • 8 million objects
  • vectors are 768 dimensions

Config for my HNSW index:

    vector_index_config=Configure.VectorIndex.hnsw(
        distance_metric=wc.VectorDistances.COSINE,
        ef = -1,
        dynamic_ef_min = 25,
        dynamic_ef_max = 125,
        ef_construction = 512
    )

Note I did not set vectorCacheMaxObjects, so this parameter should have the default value of 1e12.

Server Setup Information

  • Weaviate Server Version: 1.24
  • Deployment Method: docker
  • Multi Node? Number of Running Nodes: 1 node
  • Client Language and Version: Python v4
  • Amount of memory on my node: 120 GB, which should be plenty

Any additional Information

2

hi @cpwalker !

That’s interesting.

Do you have any memory, disk, cpu readings or logs from those events?

That could give us some leads.

3

I just ran a series of test queries just now while running top on the command line so I could watch memory usage. These were the first queries I ran after several hours of inactivity.

Before running any queries against my collection (baseline): 41.3 GB RAM used
available memory: 78 GB RAM

First query attempt:

used memory rises to 55 GB RAM. ~30 seconds pass, then I get this error:

WeaviateQueryError: Query call with protocol GRPC search failed with message Deadline Exceeded.

second query attempt:

used memory rises to 66 GB. ~30 seconds pass again and get this error WeaviateQueryError: Query call with protocol GRPC search failed with message Deadline Exceeded.

third query attempt:

used memory rises to 83.8 GB
queries successfully executes in ~4.25 seconds.

4

For the benefit of others, two things I’ve tried recently and will see if they work:

  1. Increase timeout on queries.
  2. Disabled swap in Linux.
1 Like