Creating and deleting class take > 1m and search sometimes only return 204

Support
1

Hello Team,

We currently operate a standalone instance of Weaviate with a total of 4514 classes.

Recently, we’ve encountered two peculiar issues:

  1. The process of creating and deleting a class are taking over 1 minute, which is too long. However, adding and deleting data objects to existing classes still work well.
  2. In the past, querying certain classes resulted in responses, but now the same classes only return 204 responses.
    We experimented to create another class with exactly the same data objects, the new one can get response.

Our Weaviate deployment is on k8s. The CPU limit is set at 16, and the memory limit is configured to 64GB. Neither resource has reached its limit.

All weaviate class configuration is set to default as below:

{
      "class": "xxx",
      "description": "xxx",
      "invertedIndexConfig": {
        "bm25": {
          "b": 0.75,
          "k1": 1.2
        },
        "cleanupIntervalSeconds": 60,
        "stopwords": {
          "additions": null,
          "preset": "en",
          "removals": null
        }
      },
      "properties": [xxx],
      "replicationConfig": {
        "factor": 1
      },
      "shardingConfig": {
        "actualCount": 1,
        "actualVirtualCount": 128,
        "desiredCount": 1,
        "desiredVirtualCount": 128,
        "function": "murmur3",
        "key": "_id",
        "strategy": "hash",
        "virtualPerPhysical": 128
      },
      "vectorIndexConfig": {
        "cleanupIntervalSeconds": 300,
        "distance": "cosine",
        "dynamicEfFactor": 8,
        "dynamicEfMax": 500,
        "dynamicEfMin": 100,
        "ef": -1,
        "efConstruction": 128,
        "flatSearchCutoff": 40000,
        "maxConnections": 64,
        "pq": {
          "bitCompression": false,
          "centroids": 256,
          "enabled": false,
          "encoder": {
            "distribution": "log-normal",
            "type": "kmeans"
          },
          "segments": 0
        },
        "skip": false,
        "vectorCacheMaxObjects": 1000000000000
      },
      "vectorIndexType": "hnsw",
      "vectorizer": "none"
    }

I would appreciate any help

2

Hi Allison and welcome :slight_smile:

That’s interesting. May I ask what the classes are for, and ask whether our multi-tenancy feature might be suitable?

Multi-tenancy is designed for users who might be building apps for many end users (e.g. SaaS use cases).

We’ve seen slowdowns with high number of classes, but MT cases are designed for that. So I wonder if that might be helpful for you.

Here is the launch blog, and also some example snippets.

What do you think? Is that potentially suitable for your use case?

JP

3

Thank you JP, the good news is that after we downgraded from 1.22 to 1.20, creating and deleting classes work well now within 1-2 seconds. I am wondering if there are some changes between these two versions impacting the behavior?

Every class is a collection in our design, and users can upload documents to it to chat with their collection.

We would consider multi tenancy in the backlog :slightly_smiling_face:

4

Thanks Allison for the update (sorry for the late reply, I had been away).

That’s interesting that now it is now faster. Are you still seeing the same speedup? The slowdown in collection (class) operations with the high number of collections should exist across all versions, unfortunately. So I am curious to see if you are still seeing the speedup.

It is also exactly what multi-tenancy was created to address, so I would highly encourage you to migrate to a MT setup. You should see very fast collection operations then. :slight_smile: