Why my Weaviate vector search performance is low?

General
1

Hello, we use Weaviate 1.22.11 to store 3 million vectors in a class, the vectorization is done outside of Weaviate with our trained model. The vector search with nearVector is slow, between 5 and 20 seconds.

Could anyone give me some advice on how to improve search performance? Thanks.

Environment: Weaviate on AWS ECS, one task 4 vCPU, 8 GB RAM
Model: 256 dimensions
Weaviate class (some properties may be null):

{
    "class": "Test",
    "description": "test",
    "vectorIndexType": "hnsw",
    "vectorIndexConfig": {
        "vectorCacheMaxObjects": 24000,
        "ef": -1,
        "efConstruction": 2000,
        "maxConnections": 64
    },
    "vectorizer": "none",
    "properties": [
        {"name": "reference", "dataType": ["text"], "description": "ID", "tokenization": "field", "indexSearchable": true, "indexFilterable": false},
        {"name": "title", "dataType": ["text"], "description": "doc title", "indexSearchable": true, "indexFilterable": false},
        {"name": "display_title", "dataType": ["text"], "description": "display title", "indexSearchable": true, "indexFilterable": false},
        {"name": "book_title", "dataType": ["text"], "description": "doc title", "indexSearchable": true, "indexFilterable": false},        
        {"name": "text", "dataType": ["text"], "description": "text", "indexSearchable": true, "indexFilterable": false},
        {"name": "document_type", "dataType": ["text"], "description": "document type", "tokenization": "field", "indexSearchable": true, "indexFilterable": false},
        {"name": "document_source", "dataType": ["text"], "description": "document source", "tokenization": "field", "indexSearchable": true, "indexFilterable": false},
        {"name": "publisher", "dataType": ["text"], "description": "publisher", "tokenization": "field", "indexSearchable": true, "indexFilterable": false},
        {"name": "type_facet", "dataType": ["text[]"], "description": "type facet", "tokenization": "field", "indexSearchable": true, "indexFilterable": false},
        {"name": "date_facet", "dataType": ["text"], "description": "date facet", "tokenization": "field", "indexSearchable": true, "indexFilterable": false},
        {"name": "sort_date", "dataType": ["text"], "description": "date", "tokenization": "field", "indexSearchable": true, "indexFilterable": false}
    ]
}

Search result example:

{
  "data": {
    "Get": {
      "Test": [
        {
          "_additional": {
            "certainty": 0.8171625733375549,
            "id": "44048721-c4c1-41d9-9e6b-707f74e5ebf8"
          },
          "book_title": "",
          "date_facet": null,
          "display_title": "Display test",
          "document_source": "doc",
          "document_type": "test",
          "publisher": "website",
          "sort_date": "2014-03-25",
          "text": "Why my Weaviate vector search performance is low?",
          "title": "It's a test"
        }
      ]
    }
  }
}

For information, we had a Weaviate 1.19.13 on AWS Kubernetes, 2 nodes 16 vCPU and 32 GB RAM, the vectorIndexConfig is the same, but with differents indexed documents. The vector search was less than 1 second.

2

Hi!

Do you see any outstanding logs?

According to here:

For 3MM of objects with that dimension, 2G of RAM per 1M objects would be needed to run it smoothly, so it’s probably not it.

How is the memory consumption?

Also, it would help to test this in latest versions, if possible reindexing the data to a new cluster.

This would give you access to the python v4 client that leverages GRPC alog a lot of new improvements.

Depending on the usage (high amount of concurrent queries) multiple nodes is one avenue too:

Let me know if this helps.

3

Hi @DudaNogueira, thank you for the response.
I found this error in CloudWatch logs for all queries, but I can’t fix the error:

{
    "action": "cyclemanager",
    "callback_id": "segmentgroup/compaction//var/lib/weaviate/test_EM7lfLePxDvQ_lsm/property_document_type_searchable",
    "callbacks_id": "store/compaction/..",
    "class": "Test",
    "index": "test",
    "level": "error",
    "msg": "callback panic: runtime error: index out of range [-1]",
    "shard": "EM7lfLePxDvQ",
    "time": "2024-03-04T08:23:07Z"
}

Here is an example of query:

echo '{ 
  "query": "{
    Get {
      Test(
        limit: 100
        nearVector: {
          vector: [0.1712646484375,0.1673583984375,...]}
      )
      {
        title text display_title book_title document_type document_source publisher date_facet sort_date
        _additional {
          certainty id
        }
      }
    }
  }"
}' | curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -d @- \
    https://test.com/v1/graphql

Furthemore I think vectorCacheMaxObjects is not enough for 3 MM of objects.

The consumption of CPU and RAM

image
image1036Ă—278 25.6 KB

4

I removed the null properties from the Weaviate class Test, and now I haven’t the error:
callback panic: runtime error: index out of range [-1]

I put “vectorCacheMaxObjects”: 3200000 in the class.
After the import of 2400000 objects, the search is still slow, between 2 and 10 seconds. The consumption of CPU and RAM is near to 100% (the search begins after 16 pm)

image
image1035Ă—277 16 KB

Logs:

{
    "action": "startup",
    "default_vectorizer_module": "none",
    "level": "info",
    "msg": "the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer",
    "time": "2024-03-04T16:25:45Z"
}

{
    "action": "startup",
    "auto_schema_enabled": true,
    "level": "info",
    "msg": "auto schema enabled setting is set to \"true\"",
    "time": "2024-03-04T16:25:45Z"
}

We also have performance problem of importing more than 2MM objects in Weaviate (before 16 pm), with 100% of CPU and RAM.
Before test the latest version, we will add more CPU and RAM.