Does Weaviate support SSL out of the box?

kokogrr · September 5, 2023, 3:18pm

Hello,
I’m setting up self hosted instance of Weaviate on an Ubuntu VPS with Weaviate running in a Docker container. My question is can Weaviate instance support SSL connections by itself like eg. MySQL server does or do I need a reverse proxy eg. Nginx that will handle that? I looked into docs and searched ‘weaviate ssl’ online but didn’t find anything useful.

DudaNogueira · September 5, 2023, 3:22pm

Hi @kokogrr ! Welcome to our community

No. You will need to reverse-proxy it in order to get SSL.
You can do it with traefik or nginx, for example.

I will add a note about this in our docs, thank you!

kokogrr · September 5, 2023, 3:42pm

Thanks a lot for quick reply!

salik_Lari · September 26, 2023, 6:12am

is document updated for SSL? I cant find anything

lakshminarayana · June 28, 2024, 3:39pm

@DudaNogueira Thanks a lot for your response. Do u have any handy documentation for the same, if any Please share with us.

DudaNogueira · June 28, 2024, 7:20pm

hi @lakshminarayana ! Welcome to our community.

Our documentation usually doesn’t cover the SSL/TLS side of the deployment for two main reasons:

1 - Usually, self deployments will not expose Weaviate directly. Their applications will be exposed. but not Weaviate.
2 - Whenever there is a reason to expose Weaviate under a SSL/TLS connection, one can use a variety of reverse proxies, load balancers and so on.

I have crafted here a gist on how to deploy Weaviate, with a single node, using docker compose and properly exposing it using traefik:

According to this docker compose, this is how you would connect to it:

client = weaviate.connect_to_custom(
    http_host="weaviate.yourcompany.com",
    http_port=443,
    http_secure=True,
    grpc_host="grpc.weaviate.yourcompany.com",
    grpc_port=50051,
    grpc_secure=True
)

If running on a VPS, you will need to have both weaviate.yourcompany.com and grpc.weaviate.yourcompany.com pointing to the public IP of this VPS.

Here the content of the docker compose as of now:

---
services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: cr.weaviate.io/semitechnologies/weaviate:1.25.5
    # uncomment only if you want to connect unnsecured
    #ports:
    #- 8081:8080
    #- 50052:50051
    volumes:
    - weaviate_data:/var/lib/weaviate
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      ENABLE_MODULES: 'text2vec-cohere,text2vec-huggingface,text2vec-palm,text2vec-openai,generative-openai,generative-cohere,generative-palm,ref2vec-centroid,reranker-cohere,qna-openai'
      CLUSTER_HOSTNAME: 'node1'
    labels:
      - "traefik.enable=true"
      # http
      - "traefik.http.services.weaviate_http_service.loadbalancer.server.port=8080"
      - "traefik.http.routers.weaviate_http_router.rule=Host(`weaviate.yourdomain.com`)"
      - "traefik.http.routers.weaviate_http_router.entrypoints=websecure"
      - "traefik.http.routers.weaviate_http_router.service=weaviate_http_service"
      - "traefik.http.routers.weaviate_http_router.tls.certresolver=myresolver"
      # # grpc
      - "traefik.http.services.weaviate_grpc_service.loadbalancer.server.scheme=h2c"
      - "traefik.http.services.weaviate_grpc_service.loadbalancer.server.port=50051"
      - "traefik.http.routers.weaviate_grpc_router.rule=Host(`grpc.weaviate.yourdomain.com`)"
      - "traefik.http.routers.weaviate_grpc_router.entrypoints=grpc"
      - "traefik.http.routers.weaviate_grpc_router.service=weaviate_grpc_service"
      - "traefik.http.routers.weaviate_grpc_router.tls.certresolver=myresolver"

  traefik:
    #image: "traefik:v2.11"
    image: "traefik:v3.0.3"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      - "--entrypoints.grpc.address=:50051"
      - "--providers.docker"
      - "--api"
      # - "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.myresolver.acme.tlschallenge=true"
      - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.myresolver.acme.email=you@yourcompany.com"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"

    ports:
      - "80:80"
      - "443:443"
      - "50051:50051"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

volumes:
  weaviate_data:
...

let me know if this helps

lakshminarayana · July 1, 2024, 2:48am

Thanks a lot for support and help We will update once we were able to access weaviate with https:

lakshminarayana · July 11, 2024, 9:33am

While accessing the gRPC client am facing below error

2024/07/11 09:22:21 [error] 26#26: *9358 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: 10.0.131.66, server: grpc.weaviatedev.test.com, request: “GET / HTTP/1.1”, upstream: “http://10.0.141.246:50051/”, host: “grpc.weaviatedev.test.com”
2024/07/11 09:22:21 [error] 26#26: *9358 recv() failed (104: Connection reset by peer) while reading upstream, client: 10.0.131.66, server: grpc.weaviatedev.test.com, request: “GET / HTTP/1.1”, upstream: “http://10.0.141.246:50051/”, host: “grpc.weaviatedev.test.com”
10.0.131.66 - - [11/Jul/2024:09:22:21 +0000] “GET / HTTP/1.1” 009 15 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 (scanner.ducks.party)” 228 0.003 [weaviate-weaviate-grpc-50051] 10.0.141.246:50051 15 0.003 200 1d8e05cbe04f47be9b601d3a3fac3fa0
10.0.131.66 - - [11/Jul/2024:09:22:24 +0000] “GET / HTTP/1.1” 308 164 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 (scanner.ducks.party)” 228 0.000 [weaviate-weaviate-grpc-50051] - - - - 51cb606133e373c434cd851fdbfc52a9
2024/07/11 09:22:28 [error] 27#27: *9418 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: 10.0.131.66, server: grpc.weaviatedev.test.com, request: “GET / HTTP/1.1”, upstream: “http://10.0.157.18:50051/”, host: “grpc.weaviatedev.test.com”, referrer: “http://grpc.weaviatedev.test.com”
2024/07/11 09:22:28 [error] 27#27: *9418 recv() failed (104: Connection reset by peer) while reading upstream, client: 10.0.131.66, server: grpc.weaviatedev.test.com, request: “GET / HTTP/1.1”, upstream: “http://10.0.157.18:50051/”, host: “grpc.weaviatedev.test.com”, referrer: “http://grpc.weaviatedev.test.com”
10.0.131.66 - - [11/Jul/2024:09:22:28 +0000] “GET / HTTP/1.1” 009 15 “http://grpc.weaviatedev.test.com” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36(scanner.ducks.party)” 278 0.001 [weaviate-weaviate-grpc-50051] 10.0.157.18:50051 15 0.001 200 48ee60f00f83722e720314166b7b2c5a
ubuntu@ip-10-0-9-110:~$

any one can be able to help me on this

@DudaNogueira

lakshminarayana · July 11, 2024, 9:35am

Nginx-Ingress Rule:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: weaviate-ingress
namespace: weaviate
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/ssl-redirect: “true”
nginx.ingress.kubernetes.io/backend-protocol: “GRPC”
nginx.ingress.kubernetes.io/backend-protocol: “HTTP”
spec:
ingressClassName: nginx
tls:

hosts:
- weaviatedev.test.com
- grpc.weaviatedev.test.com
  secretName: weaviate-tls
  rules:
host: weaviatedev.test.com
http:
paths:
- path: /
  pathType: Prefix
  backend:
  service:
  name: weaviate
  port:
  number: 8080
host: grpc.weaviatedev.test.com
http:
paths:
- path: /
  pathType: Prefix
  backend:
  service:
  name: weaviate-grpc
  port:
  number: 50051

@DudaNogueira Any help on the above issue…

DudaNogueira · July 15, 2024, 1:20pm

Hi!

are those logs from nginx?

lakshminarayana · July 16, 2024, 12:00pm

Yes Logs from Nginx only @DudaNogueira

DudaNogueira · July 16, 2024, 2:14pm

Exposing the deployment has a lot of options and varies a lot.

I have crafted a docker compose for traefik here.

because of the new grpc endpoint, it got a little bit more tricky than only a common http port.

I have not played with nginx on k8s yet. I will try to get a hold of those an try to come up with some samples, as exposing weaviate is a common questions I see around.

But bottom line, this is will be about exposing both the http port and grpc port.

for the grpc part, you can use tools like grpcurl to make sure you deployment is serving. Check this thread on this

for instance:

 grpcurl -d '{"service": "Weaviate"}' -proto health.proto grpc.weaviate.mydomain.com:50051 grpc.health.v1.Health/Check

Let me know if this helps and let’s keep it going!

Thanks for using Weaviate and engaging here on forums, we really appreaciate it

lakshminarayana · July 28, 2024, 12:34pm

@DudaNogueira

Need Your help here .

root@i-xxxxxxxxx-server:~# grpcurl -d ‘{“service”: “Weaviate”}’ -proto health.proto grpc.pocweviate.mydomain.com:443 grpc.health.v1.Health/Check
{
“status”: “SERVING”
}

root@i-xxxxxxxxx-server:~# grpcurl -v -insecure -H “apikey: weaviatetest-readOnly-Key-lakshmi” grpc.pocweviate.mydomain.com:443 list
Failed to list services: server does not support the reflection API

root@i-xxxxxxxxx-server:~# grpcurl -v -insecure -H “apikey: weaviatetest-readOnly-Key-lakshmi” grpc.pocweviate.mydomain.com:50051 list
Failed to dial target host “grpc.pocweviate.mydomain.com:50051”: context deadline exceeded
root@i-xxxxxxxxx-server:~#

I have applied SSL using Nginx Ingress

am using NetworkLoadbalencer

Failed to list services: server does not support the reflection API

Failed to dial target host “grpc.pocweviate.mydomain.com:50051”: context deadline exceeded

Why this error, Any help from your side. It’s littile urjent. Your inputs will help me a lot.

Hope GRPC endpoint health looks Good.

status": "SERVING

DudaNogueira · July 29, 2024, 6:14pm

Hi!

This looks like something on k8s

Exposing Weaviate is a topic we can never cover all the basis, as it will depend a lot on how and where you are deploying it.

the reflection error is probably grpcurl hitting somewhere different to what we expect and not being able to list.

Topic		Replies	Views
Unable to connect to weaviate running behind reverse proxy Support bug	14	236	August 13, 2024
Connection_params for docker configuration Support	12	863	June 26, 2024
SSL Certificate Verification Failure When Connecting to Weaviate in Kubernetes Support developer-experience	2	990	April 17, 2024
Weaviate with Traefik and gRPC Support	4	956	August 12, 2024
Unable to use Weaviate due to SSL certificate issue Support	1	188	July 11, 2024

Does Weaviate support SSL out of the box?

Related topics