How to configure Kubernetes yaml files to work behind a proxy

Support
1

Description

If we add https_proxy in kubernetes, node is not starting and no peers found, election aborted. Without proxy we are not able to hit openai services. How to configure deployment.yaml to remedy this. This is a single-node setup.

Server Setup Information

  • Weaviate Server Version: 1.31.5
  • Deployment Method: Kubernetas
  • Multi Node? Number of Running Nodes: Single node
  • Client Language and Version: Python client v4
  • Multitenancy?: No

Any additional Information

containers:
- args:
- --host
- 0.0.0.0
- --port
- “8080”
- --scheme
- http
env:
- name: AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED
value: “true”
- name: CLUSTER_HOSTNAME
value: node1
- name: DEFAULT_VECTORIZER_MODULE
value: none
- name: ENABLE_MODULES
value: text2vec-cohere,text2vec-huggingface,text2vec-palm,text2vec-openai,generative-openai,generative-cohere,generative-palm,ref2vec-centroid,reranker-cohere,qna-openai
- name: PERSISTENCE_DATA_PATH
value: /var/lib/weaviate
- name: BACKUP_FILESYSTEM_PATH
value: /tmp/backups
- name: QUERY_DEFAULTS_LIMIT
value: “25”
- name: https_proxy
value: http://120.0.0.1:8080/
image: semitechnologies/weaviate:1.31.5-68c0927.amd64
imagePullPolicy: IfNotPresent
name: weaviate
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 50051
protocol: TCP
resources: {}

2

Hey @Ashu

Welcome to our community—it’s lovely to have you here :hugs:

I’m not an expert in proxies, but I did look into this, and as probably same as "attempt to join and failed" when using PERSISTENCE_DATA_PATH env with efs storage - #3 by franz_hals by other community member, you can try setting the no_proxy environment variable to exclude local addresses and the pod/container’s own IPs from being proxied. This way, Weaviate’s internal communication won’t go through the proxy, while external requests (like to OpenAI) still will.

Try adding a no_proxy variable alongside your https_proxy. For a single-node setup, you’d typically want to exclude at least localhost, 127.0.0.1, and your pod’s own IP range.

I will also tag my colleague @DudaNogueira who is experienced with proxies in case I missed on any critical point in the above.

Have you tried that already?

Best regards,

Mohamed Shahin
Weaviate Support Engineer
(Ireland, UTC±00:00/+01:00)

3

Hi @Mohamed_Shahin,

Thanks for the suggestion. I have tried adding the no_proxy the above issue (no peers found, election aborted), this seems to be resolved. But getting another error i.e. not part of a stable configuration or a non-voter, not triggering a leader election.

containers:

  • args:
  • –host
  • 0.0.0.0
  • –port
  • “8080”
  • –scheme
  • http
    env:
  • name: AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED
    value: “true”
  • name: CLUSTER_HOSTNAME
    value: node1
  • name: DEFAULT_VECTORIZER_MODULE
    value: none
  • name: ENABLE_MODULES
    value: text2vec-cohere,text2vec-huggingface,text2vec-palm,text2vec-openai,generative-openai,generative-cohere,generative-palm,ref2vec-centroid,reranker-cohere,qna-openai
  • name: PERSISTENCE_DATA_PATH
    value: /var/lib/weaviate
  • name: BACKUP_FILESYSTEM_PATH

value: /tmp/backups

-name: no_proxy

value: #.#.#.#/16, localhost, .svc,.svc.cluster.local,kubernetes.default
image: semitechnologies/weaviate:1.31.5-68c0927.amd64
imagePullPolicy: IfNotPresent
name: weaviate
ports:

  • containerPort: 8080
    protocol: TCP
  • containerPort: 50051
    protocol: TCP
    resources: {}

Best Regards,
Ashvini Kumar

4

@Ashu your cluster is a single node so leader errors and warnings shall be ignored since your cluster is not multi-node setup.

The cluster should be reachable and fine, right?

Additionally, I would say please upgrade your cluster to 1.32.4 (latest).

Best regards,

Mohamed Shahin
Weaviate Support Engineer
(Ireland, UTC±00:00/+01:00)

5

Hi @Mohamed_Shahin ,

I have tried with suggested changes but still the same issue persist, as after adding no_proxy has no effect.

In this single node setup, if I remove https_proxy the weaviate container is running and leader is held & won and able to use the DB. But if i am removing https_proxy won’t able to use openai models.

And with https_proxy, the leader election is not triggering.

Thanks and Regards,

Ashvini Kumar

6

This looks a bit weird, no_proxy is supposed to contain individual domains, domain suffixes or IPs. Giving an IP range such as #.#.#.#/16 is not supported AFAIK. If possible, you could try adding all possible IPs individually: bash - Set a network range in the no_proxy environment variable - Unix & Linux Stack Exchange

I also see there are spaces within your value (after commas), that might also be a problem, I would advise to remove them.