Hello everyone!. I was trying out weaviate and wanted to implement a reverse proxy. Currently weaviate is hosted on docker that runs with traefik. I was able to connect to weaviate’s REST API without any issues. However, what i was unable to do is proxy the gRPC connection.
Here is the error is see on my debugger:
Exception has occurred: WeaviateQueryException
Query call failed with message Stream removed.
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "Stream removed"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-01-14T21:04:14.775200504-05:00", grpc_status:2, grpc_message:"Stream removed"}"
>
During handling of the above exception, another exception occurred:
File "/mnt/h/src/server/database/connect.py", line 36, in <module>
response = questions.generate.near_text(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weaviate.exceptions.WeaviateQueryException: Query call failed with message Stream removed.
The console had another error (most likely thrown right before the debugger caught the above error:
E0114 21:04:14.765458535 185327 hpack_parser.cc:999] Error parsing 'content-type' metadata: invalid value
Till now i have tried the following configurations with traefik.
- Creating a grpc endpoint and directly proxying it to port 50051. Did not work.
- Creating a H2C endpoint by following this example. Did not work
- Creating an https termination on port 443 and proxying it to port 50051. Did not work
My current docker compose(swarm stack) file is like so:
version: '3.8'
services:
weaviate:
image: semitechnologies/weaviate:1.23.2
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
ports:
- "8080:8080"
- "50051:50051"
volumes:
- nfs:/var/lib/weaviate
networks:
- public
deploy:
replicas: 1
placement:
constraints: [node.role != manager]
labels:
- "traefik.enable=true"
- "traefik.http.routers.weaviate.rule=Host(`weaviate.mydomain.com`)"
- "traefik.http.routers.weaviate.entrypoints=websecure"
- "traefik.http.routers.weaviate.tls=true"
- "traefik.http.routers.weaviate.tls.certresolver=ssl_resolver"
- "traefik.http.routers.weaviate.tls.domains[0].main=weaviate.mydomain.com"
- "traefik.http.routers.weaviate.service=weaviate"
- "traefik.http.services.weaviate.loadbalancer.server.port=8080"
- "traefik.http.routers.weaviategrpc.rule=Host(`grpc.weaviate.mydomain.com`)"
- "traefik.http.routers.weaviategrpc.entrypoints=web"
# - "traefik.http.routers.weaviategrpc.tls=true"
# - "traefik.http.routers.weaviategrpc.tls.certresolver=ssl_resolver"
- "traefik.http.routers.weaviategrpc.tls.domains[0].main=grpc.weaviate.mydomain.com"
- "traefik.http.routers.weaviategrpc.service=weaviate"
- "traefik.http.services.weaviategrpc.loadbalancer.server.port=50051"
- "traefik.http.services.weaviategrpc.loadbalancer.server.scheme=h2c"
environment:
OPENAI_APIKEY: "${OPENAI_APIKEY}"
QUERY_DEFAULTS_LIMIT: "${QUERY_DEFAULTS_LIMIT}"
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "${AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED}"
PERSISTENCE_DATA_PATH: "${PERSISTENCE_DATA_PATH}"
DEFAULT_VECTORIZER_MODULE: "${DEFAULT_VECTORIZER_MODULE}"
ENABLE_MODULES: "${ENABLE_MODULES}"
CLUSTER_HOSTNAME: "${CLUSTER_HOSTNAME}"
LOG_LEVEL: "${LOG_LEVEL}"
volumes:
nfs:
driver_opts:
type: "nfs"
o: "addr=192.168.4.2,nfsvers=4,nolock,soft,rw"
device: ":/mnt/pool/docker_swarm/weaviate"
networks:
public:
external: true
NOTE: Local connections WORK as expected. This is a configuration issue.
