OOM issues when importing data

Support
1

Description

We’re seeing OOM kills (we assume this to be the case) when doing imports while weaviate is using high amounts of memory (in percent of what is available).

We are using LIMIT_RESOURCES, which we thought would alleviate this issue fully, but this seems to not be the case. We are also using flat indexing, which we thought would reduce memory needed by a lot (by 100%?), but this seems to not be the case.

Our use case is where our users continually upload data, so we don’t know how many vectors we will need to support, and will always need to support more.

Below is a screenshot of GCP monitoring, when “read” spikes the server is unresponsive, this only happens under import.

image
image965×250 11.8 KB

Is this expected or are we doing something massively wrong? We want to keep using weaviate but if this is just the nature of the Go garbage collector we might not be able to (I have read all there is on this subject from the docs).

Since the server becomes unresponsive it is not a 100% that it is OOM kills due to slow garbage collection, but that is the only explanation I can find. Please let me know if it could be other things.

Server Setup Information

  • Weaviate Server Version: 1.24.10
  • Deployment Method: docker (docker compose)
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: Python, 4.5.1
2

hi @jontpan !

If you are running a large scale operation, with concurrent writes and reads, you will need to spread the load on multiple nodes.

Are you using tenants?

flat indices will not store vectors in memory, so the source of your OOM may be from the read/write while ingesting?

Let me know about this info so we can help you further.

Thanks!

3

Hi,

Thanks for the reply.

We are not a large scale operation yet, but we will be.

We are using tenants.

What do you mean when you say read/write while ingesting? The last time we got the OOM kill was on our staging environment, where the only read/write was the ingesting. Or do you mean that ingesting does some read/write operations that I am not aware of? Even with flat index?

4

hi @jontpan !

Ingesting does a lot of operations under the hood. For example, if you are not providing the vectors yourself, it will vectorize it for you. Also, it will not only write the objects, but also build the index.

So while you are ingesting, a lot is happening.

Usually, OOMs are a symptom of lack of resources (memory or cpu), as Weaviate will eat up/allocate all resources you feed it.

Also, be aware of the batch size. The best option and safest is to use dynamic batch size, as described here:

I have seen users importing millions of objects with flat indexes (that for using it as source of truth while doing benchmark), so most certainly this is a resource issue.

Let me know if this helps.