Cluster memory usage reaching capacity limits! What does this mean?

Support
1

Description

I see this on my WCS dashboard:

Cluster memory usage reaching capacity limits!

To avoid unwanted spending, clusters are limited in their memory capacity. They are regularly adjusted depending on the load and usage. In order to request a limit increase now, please check the details of the following cluster(s):

• my cluster

To avoid unwanted spending, clusters are limited in their memory capacity. They are regularly adjusted depending on the load and usage. In order to request a limit increase now, please check the details of the following cluster(s):

What does this mean? I feel like some of my data (embeddings) is being cut off (disappearing). Is that what it means? If so, what do I do about it?

Server Setup Information

  • Weaviate Server Version: 1.23.10
  • Deployment Method: WCS
  • Multi Node? Number of Running Nodes: ??? Where do I find that?
  • Client Language and Version:

Any additional Information

1 Like
2

Hi!

This means that your WCS cluster doesn’t have unlimited allocated resources. So whenever needed, more resources will be allocated.

Now, if you have a huge influx of data (usually the first ingestion, for example), it is advised to let us know so we can scale your cluster accordingly and avoid it hitting those limits.

What is probably happenning is that, due to this limit, some of your objects may not be included.

You can catch those errors while batch inserting, by doing some proper error handling:

let me know if this helps :slight_smile:

3

How do I do this? Who do I need to talk to?

4

Hi!

Best place for WCS issues and requests is opening a support ticket by sending an email to support@weaviate.io

Please, specify the cluster endpoint, and the number of objects and dimensions you plan on ingesting.

Thanks!

5

I just got to thinking about this today. I have opened a ticket on it, but thought I would mention it here as well.

It appears that a lot of our content has simply disappeared, and we do not know what is missing at this point.

So, my question is: Is there some sort of log records which would give us some indication of:

a. The date(s) our content was truncated?
b. What was truncated?

Because now, we have to figure out what exactly is missing and how to replace it.

Thanks.

6

Hi!

It doesn’t get truncated. On that situation, the shards enter a READONLY state, and batch imports start returning this.

Two good practices to be on top of that is handling error on imports
and using deterministic ids (if possible), so it doesn’t import a same object twice.

Let me know if that helps.

7

What I know so far is that one of the datasets that I recently embedded is now missing over 50% of it’s content. If I understand you correctly, that would mean that the missing content was never embedded in the first place.

This is the most recent error I have noticed: ERROR: (from Weaviate) - update vector: send POST request: Post “https://api.openai.com/v1/embeddings”: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

And this:

ERROR: [{“message”:“msg:merge and vectorize code:500 err:update vector: connection to: OpenAI API failed with status: 500 error: Timed out generating response. Please try again with a shorter prompt or with max_tokens set to a lower value.”}]

I’m going to go through and test of of my content for re-embedding. Hopefully it’s just this one batch

8

Those messages points to OpenAI taking too long to get your vectors embeddings back.

It also indicate you might have big of chunks, and could maybe reduce each chunk size.