Modify helm chart for a 413

Support
1

Description

Hello, we have a weaviate deployment and we’re currently running into a 413

413, with response body: None. Payload Too Large. Try to decrease the batch size or increase the maximum request size on your weaviate server

that said, I was wondering how we can increase the maximum request size on our weaviate server via helm? I saw there’s an env var for increasing the total objects read (QUERY_MAXIMUM_RESULTS) but is there an env var or configuration of some sort via helm that we could consider for altering the maximum requests used for objects written

our weaviate server version is 1.24.8 and we’re using weaviate-client = "^4"

Note: the largest item we had was 0.013 MB in size and the total document’s embeddings were roughly 17 MB in size

2

Hi @mmoya !! Welcome to our community! :hugs:

How are you inserting those objects?

Have you tried reducing the batch size?

This kind of issue will usually surface when the payload per batch is too high the connection will fail.

I have asked internally, but I am not sure on how to increase this on Weaviate side, as it usually will accept any amount of data you throw.

Have you tried using the dynamic size batch?

3

Thank you for the reply. Unfortunately I have tried different batch sizes (I even considered a batch size of 1) and I’m still running into this issue

I’m using docarray to write my objects into weaviate

import weaviate
from docarray import BaseDoc, DocList
from docarray.index import WeaviateDocumentIndex

def index_documents(
    client: weaviate.Client,
    auth_api_key: str,
    objects: list[BaseDoc] | DocList[BaseDoc],
    batch_size: int = 100,
) -> WeaviateDocumentIndex[BaseDoc]:
    """
    Index documents in weaviate
    """

    weaviate_url = client._connection.url
    index = get_weaviate_index(objects[0].__class__, weaviate_url, auth_api_key)
    runtime_config = WeaviateDocumentIndex.RuntimeConfig(
        batch_config={
            "batch_size": batch_size,
            "dynamic": True,
            "num_workers": 2,
            "timeout_retries": 10,
        }
    )
    index.configure(runtime_config=runtime_config)
    index.index(objects)
    return index

we’re also using ada v2 embeddings and I have also tried using "text-embedding-3-small" instead but both embeddings are 1536 in dimensionality. Again, the largest batch I tried writing wasn’t even 1MB in size before throwing a 413
Given that this also gave me a 413 when I considered a batch size of 1, I was wondering if the weaviate server/helm chart has a limitation as far as dimensionality or is there anything else you would suggest as far as altering our helm chart?

4

Hi!

On that case, your content may be too big, even for a single object.

I have not used docarray, but you may need some chunking before ingesting data. :thinking:

6

Hello, how do I increase the maximum request size on your weaviate server as suggested in the logs via the weaviate client?

7

Hi!

I believe that the error message is misleading. The solution is about reducing the batch size, as by default, AFAIK, the server will already operate with the maximum possible data limit to be ingested.

Maybe, another component I can think about is on the reverse proxy in front of Weaviate?

8

I was able to resolve it by modifying our nginx ingress payload size by adding the nginx.ingress.kubernetes.io/proxy-body-size: "200m" annotation

looks like it’s set to 1MB by default

1 Like
9

Awesome!! Thanks for sharing, @mmoya !