API Key change in values.yaml when deploying Weaviate in k8s does not update in the cluster

Support
1

Description

Hi,

I’ve deployed Weaviate in a kubernetes cluster using the Helm chart and custom values in the values.yaml file. I have deployed the first time an API key for the text2vec-openai module which is no longer valid. I have updated the value with a new api key, which works (confirmed that is working), but it seems like the older api key is still being used, even after applying the help upgrade... command. Has anyone faced the same issue, and if so, how could I fix this, so that changes such as the API key are effective after applying the helm upgrade ... command?

Server Setup Information

  • Weaviate Server Version: 1.29.0
  • Deployment Method: k8s via Helm chart
  • Multi Node? Number of Running Nodes: 2
  • Client Language and Version: Python v4
  • Multitenancy?: No

Any additional Information

The following is a log that demonstrate the error:

WeaviateQueryError: Query call with protocol GRPC search failed with message <AioRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "explorer: get class: concurrentTargetVectorSearch): explorer: get class: vectorize search vector: vectorize params: vectorize params: vectorize keywords: remote client vectorize: connection to: OpenAI API failed with status: 429 request-id: req_2d22fee09e16eeda10ced1b3692e9339 error: You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors."
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"explorer: get class: concurrentTargetVectorSearch): explorer: get class: vectorize search vector: vectorize params: vectorize params: vectorize keywords: remote client vectorize: connection to: OpenAI API failed with status: 429 request-id: req_2d22fee09e16eeda10ced1b3692e9339 error: You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.", grpc_status:2, created_time:"2025-03-17T22:37:42.023459-03:00"}"
>.

The following is the values.yaml configuration of the api key:

...
  text2vec-openai:
    enabled: true
    apiKey: 'xxxxxxxxx' (it was yyyyy before)
...

Thanks
2

@cfloressuazo That’s strange.

Could you try storing them according to the following and see if that helps?

When you do any changes, running the following should apply the update:

helm upgrade --install \
  "weaviate" \
  weaviate/weaviate \
  --namespace "weaviate" \
  --values ./values.yaml
3

Hi @cfloressuazo !!

When you edit the open ai key as well as any other key or option that will end up as an env var in your helm chart values.yaml, it kubernetes will update the secret.

However, it will not trigger a restart. And we need that!

So after updating your values in helm chart, and upgrading your helm deployment, you need to rollout a restart in your Weaviate cluster:

kubernetes rollout restart statefulset/weaviate -n weaviate

Let me know if this helps!

1 Like
4

Yes this did the trick. Thanks!

1 Like