Write Timeout for Batch Vectorization in Docker – TLS & Container Restart Issues

[Volvo]

I’m building a chatbot that uses Weaviate and t2v-transformers for vectorizing my knowledge base. My way of sends batches like this:

var batchResponse = await _httpClient.PostAsync($"{_weaviateUrl}/v1/batch/objects", batchContent);

Each batch contains 50 objects, runs with 5 parallel processes, and each object includes about 12 sentences. Most batches succeed, but occasionally I run into this error:

t2v-transformers-1 | INFO: 172.21.0.4:55808 - “POST /vectors HTTP/1.1” 200 OK
weaviate-1 | {“build_git_commit”:“584532a”, … “error”:“write tcp 172.21.0.4:8080->172.21.0.1:33884: i/o timeout”, “hint”:“Either try increasing the server-side timeout using e.g. ‘–write-timeout=600s’ …”}

The error suggests increasing the server-side timeout using --write-timeout=600s. When I added that to my Weaviate Docker configuration, the container started to reload repeatedly with errors about missing TLS certificates. To resolve the TLS issue, I generated certificates with OpenSSL, mapped them into the container, and configured Weaviate to use them.

Here’s my current Docker Compose configuration for Weaviate and t2v-transformers:

weaviate:
  image: cr.weaviate.io/semitechnologies/weaviate:1.26.4
  restart: unless-stopped
  ports:
    - 8080:8080
    - 50051:50051
  volumes:
    - /var/beyond_weaviate_data:/data
    - ./certs:/certs 
  command: [
      "/app/weaviate",
      "--write-timeout=600s",
      "--tls-certificate=/certs/cert.pem", 
      "--tls-key=/certs/key.pem"          
    ]
  environment:
    QUERY_DEFAULTS_LIMIT: 100
    AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
    PERSISTENCE_DATA_PATH: "/data"
    DEFAULT_VECTORIZER_MODULE: text2vec-transformers
    ENABLE_MODULES: text2vec-transformers
    TRANSFORMERS_INFERENCE_API: http://t2v-transformers:8080
    CLUSTER_HOSTNAME: 'node1'
    BATCH_TIMEOUT: 300

t2v-transformers:
  image: cr.weaviate.io/semitechnologies/transformers-inference:sentence-transformers-multi-qa-MiniLM-L6-cos-v1
  restart: unless-stopped
  environment:
    ENABLE_CUDA: 0
    USE_SENTENCE_TRANSFORMERS_MULTI_PROCESS: "true"
    ENABLE_CACHE: "true"

After adding the certificate parameters, the container stopped reloading, but now my application cannot connect. I get connection errors in my .NET client similar to:

at System.Net.Http.HttpConnection.<SendAsync>d__57.MoveNext()
...
at NLP_LLMEngine.Services.WeaviateVectorizationService.<CreateCollectionIfNotExistsAsync>d__11.MoveNext() in ...\WeaviateVectorizationService.cs:line 70

Weaviate logs show that the transformer remote inference service is not ready (connection refused on t2v-transformers:8080). And thats how i get connectivity issues and timeout errors.

My Question

How can I safely add a server-side write timeout (e.g., --write-timeout=600s) for batch vectorization without causing TLS certificate issues or container restarts? And if the tls is needed in my case is there a way to configure Weaviate to use the specified TLS certificate and key without interfering with communication between Weaviate and t2v-transformers?

Any help to handle these issues would be greatly appreciated. I’m fairly new to vector databases and Weaviate, so any help would be great, thanks in advance.

[Mohamed_Shahin]

Hey @Volvo

It’s great to have you in our community!

Have you tried to reduce the parallel processes to 1 or 2, or limit it to 100 objects at a time. It’s also worth checking that your network latency is good.

I’d also suggest ensuring the resources are sufficient in your cluster. Also, try increasing the connection timeout for Weaviate, especially for inserts:

additional_config=AdditionalConfig(
timeout=Timeout(init=30, query=240, insert=240),
)

Local instances | Weaviate

Regarding TLS configuration, please see the following example from my colleague @DudaNogueira regarding SSL/TLS

Does Weaviate support SSL out of the box?

hi @lakshminarayana ! Welcome to our community. Our documentation usually doesn’t cover the SSL/TLS side of the deployment for two main reasons: 1 - Usually, self deployments will not expose Weaviate directly. Their applications will be exposed. but not Weaviate. 2 - Whenever there is a reason to expose Weaviate under a SSL/TLS connection, one can use a variety of reverse proxies, load balancers and so on. I have crafted here a gist on how to deploy Weaviate, with a single node, using docker…

Regards,
Mohamed Shahin
Support Engineer

[Volvo]

Hi @Mohamed_Shahin ,thanks for replying!

I have tried reducing parallelism to 2 and increasing the batch size from 50 to 100, like so:

"WeaviateSettings": {
  "Host": "127.0.0.1",
  "Port": 8080,
  "GrpcPort": 50051,
  "BatchChunkSize": 100,
  "MaxParallelBatches": 2,
  "SentencesPerObject": 12
}

However, I am still facing the same issue with this error in my Weaviate console:

weaviate-1 | {"build_git_commit":"584532a","build_go_version":"go1.21.13","build_image_tag":"1.26.4","build_wv_version":"1.26.4","description":"An I/O timeout occurs when the request takes longer than the specified server-side timeout.","error":"write tcp 172.21.0.5:8080-\u003e172.21.0.1:35834: i/o timeout","hint":"Either try increasing the server-side timeout using e.g. '--write-timeout=600s' as a command line flag when starting Weaviate, or try sending a computationally cheaper request, for example by reducing a batch size, reducing a limit, using less complex filters, etc. Note that this error is only thrown if client-side and server-side timeouts are not in sync, more precisely if the client-side timeout is longer than the server side timeout.","level":"error","method":"POST","msg":"i/o timeout","path":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/v1/batch/objects","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"time":"2025-03-05T09:29:35Z"}

My Setup:

I am using .NET and hosting Weaviate locally on Docker with the following configuration:

weaviate:
  image: cr.weaviate.io/semitechnologies/weaviate:1.26.4
  restart: unless-stopped
  ports:
    - 8080:8080
    - 50051:50051
  volumes:
      - /var/beyond_weaviate_data:/data
  environment:
    QUERY_DEFAULTS_LIMIT: 100
    AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
    PERSISTENCE_DATA_PATH: "/data"
    DEFAULT_VECTORIZER_MODULE: text2vec-transformers
    ENABLE_MODULES: text2vec-transformers
    TRANSFORMERS_INFERENCE_API: http://t2v-transformers:8080
    CLUSTER_HOSTNAME: 'node1'
    BATCH_TIMEOUT: 300

t2v-transformers:
  image: cr.weaviate.io/semitechnologies/transformers-inference:sentence-transformers-multi-qa-MiniLM-L6-cos-v1
  restart: unless-stopped
  environment:
    ENABLE_CUDA: 0  
    USE_SENTENCE_TRANSFORMERS_MULTI_PROCESS: "true"
    ENABLE_CACHE: "true"

What I Tried:

I have attempted to add --write-timeout=600s to my Docker setup. However, after doing so, I am facing another issue: Weaviate keeps reloading.

The error I receive is:

weaviate-1 | {"action":"restapi_management","build_git_commit":"584532a","build_go_version":"go1.21.13","build_image_tag":"1.26.4","build_wv_version":"1.26.4","docker_image_tag":"1.26.4","level":"info","msg":"the required flags `--tls-certificate` and `--tls-key` were not specified","time":"2025-03-05T09:34:07Z"}
weaviate-1 exited with code 0

I’d appreciate any guidance on resolving this issue, because I’m unsure of what am I doing wrong and how should my docker look like

[DudaNogueira]

Hi!

Can you try increasing the value of MODULES_CLIENT_TIMEOUT?

You can get more info on this here:

Environment variables | Weaviate