Vectorizer Timeout settings and behavior

Support
1

Description

I am connecting with these timeout settings:

client = weaviate.connect_to_local(
    headers=headers,     
    additional_config=AdditionalConfig(
        timeout=Timeout(init=30, query=60, insert=120)  # Values in seconds
    )
)

I am using the dynamic batch functionality which I specifically thought would rate limit itself to avoid any timeout issues, but I am seeing these types of errors at the end of my batch:

Failed to import object with error: %s "ErrorObject(message=‘WeaviateBatchError(\‘Query call with protocol GRPC batch failed with message <AioRpcError of RPC that terminated with:\\n\\tstatus = StatusCode.DEADLINE_EXCEEDED\\n\\tdetails = "Deadline Exceeded"\\n\\tdebug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-07-25T15:35:02.96801-04:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"\\n>.\’)’, object_=_BatchObject(collection=‘Product’, vector=None, uuid=‘190f46bd-9a54-5a77-8463-b4845e0e4384’, properties=
…

I can see that the count of objects doesn’t match the number that went into the batch processor (it’s off by almost half), so I’m guessing there is a timeout going into the vectorizer and so the object fails to insert?

Is there a different timeout I should be setting (infinite?) or some other way I should be throttling or retrying these things?

Do you advise people to infinitely loop over the failed batch objects and retry them until the failed batch array is empty?

2

hi @ctindel !!

If with the default timeout settings you are seeing timeouts and errors in batch imports, the problem is usually on the resource you have allocated or the network connection between those parties.

This message doesn’t indicate the timeout occurred between Weaviate and the inference model api, but from the client to the server.

Do you have any observability in place for this setup? What are the cpu and memory readings?

Let me know if this helps!

Thanks!

3

This is all just running directly on my laptop, python on the macos and weaviate inside of docker.

I did set the insert timeout to 120 seconds as you can see, are you saying its taking longer than 120 seconds to insert even though i’m using the throttled dynamic batch functionality?

When I looked at Activity Monitor, between qemu and LM Studio but the CPU and GPU are pretty busy but the system is still very responsive.

I’ve given colima 4 CPUs and 12 GB RAM, not sure if the weaviate process needs more than to handle a single batch insertion thread?

4

Also are there any examples anywhere of looping over the failed object array and re-inserting them? I am having trouble finding code samples that show how to properly deal with the WeaviateBatchError() object type. Do you just peek into the private object_ property and replay it from there?

5

The indexing part is very CPU bound. So maybe it is having a hard time processing both indexing, the vectorization and answering those query/ingestion requests.

On top of the resources you gave to colima, it is interesting to play around with some resource limiting env vars:

We have some error handling tips you can do while importing, but we indeed can improve the docs on advanced batching tips and tricks.

There is a Spark connector, that can be used to create a more robust pipeline and I recall some other tools too.

For example, some time ago, we had this awesome integration between quix and weaviate:

Let me know if this helps!