S3 backup "failed to verify certificate"

Description

With Weaviate 1.23.7, I configured S3 backup in the Helm chart. The S3 endpoint is a local service (on a NetApp SAN) that’s available via https on port 8443 and with a self-signed certificate.

The inevitable happens:

# curl -X POST -H "Content-Type: application/json" -d '{ "id": "test-backup", "include": ["TEST_01"] }' http://10.43.186.32:80/v1/backups/s3
{"error":[{"message":"check if backup \"test-backup\" exists at \"s3://[…internal_bucketname…]/test-backup\": get object 'test-backup/backup_config.json': Get \"https://[…internal_host_fqdn…]:8443/[…internal_bucketname…]/?location=\": tls: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs instead"}]}

Can I somehow disable TLS verification or add the self-signed certificate as trusted? The docs only mention disabling TLS altogether which the endpoint doesn’t allow it seems.

Server Setup Information

  • Weaviate Server Version: 1.23.7
  • Deployment Method: k8s with Helm
  • Multi Node? Number of Running Nodes: 2
  • Client Language and Version: curl 7.61.1
  • Multitenancy?: no

Any additional Information

The backup-specific part of the values.yaml for Helm:

backups:
  filesystem:
    enabled: false

  s3:
    enabled: true

    envconfig:
      BACKUP_S3_BUCKET: […internal_bucketname…]
      BACKUP_S3_ENDPOINT: […internal_host_fqdn…]:8443
      # BACKUP_S3_USE_SSL: true

    envSecrets:
      AWS_ACCESS_KEY_ID: weaviate-s3-backup
      AWS_SECRET_ACCESS_KEY: weaviate-s3-backup

  gcs:
    enabled: false

  azure:
    enabled: false

tl;dr: The self-signed certificate needs subjectAltNames and the environment variable SSL_CERT_FILE allows an override for the truststore.

I continued investigating and found out that the specific self-signed certificate has issues: Golang > 1.17 doesn’t support certificates not defining SANs. With a modified self-signed certificate, the error message looks like this:

{"error":[{"message":"check if backup \"test-backup\" exists at \"s3://[…internal_bucketname…]/test-backup\": get object 'test-backup/backup_config.json': Get \"https://[…internal_host_fqdn…]:8443/[…internal_bucketname…]/?location=\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}]}

Then I found the Golang-specific environment variable SSL_CERT_FILE and added a custom truststore file as follows:

$ vi values.yaml
[…]
env:
  # Add a custom truststore including the self-signed certificate
  SSL_CERT_FILE: /tls/custom-truststore.crt
[…]
extraVolumeMounts:
- name: custom-truststore
  mountPath: /tls

extraVolumes:
- name: custom-truststore
  configMap:
    name: weaviate-custom-truststore
[…]

The referenced ConfigMap contains the complete /etc/ssl/certs/ca-certificates.crt from the image plus my self-signed certificate.

It’s working now:

{"backend":"s3","classes":["TEST_01"],"id":"test-backup","path":"s3://[…internal_bucketname…]/test-backup","status":"STARTED"}

Hi @akerkau,

Welcome to our community! It’s great to have you here.

I’m glad to hear you managed to solve the issue, and I appreciate you sharing the solution for us and other members who may face the same challenge.

Wishing you a lovely week ahead!