Connect to Weaviate Through v4 Not Working on AWS ECS

Support
1

Description

I’m running into an issue very similar to this thread, but using AWS ECS w/ Application Load balancer to route traffic to my cluster:

When I connect to the EC2 instance hosting the service I don’t have issues, but switching to the load balancer I get this:

 File "/home/ec2-user/miniconda3/envs/flaskenv/lib/python3.11/site-packages/weaviate/connect/helpers.py", line 392, in connect_to_custom
    return __connect(client)
           ^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/miniconda3/envs/flaskenv/lib/python3.11/site-packages/weaviate/connect/helpers.py", line 401, in __connect
    raise e
  File "/home/ec2-user/miniconda3/envs/flaskenv/lib/python3.11/site-packages/weaviate/connect/helpers.py", line 397, in __connect
    client.connect()
  File "/home/ec2-user/miniconda3/envs/flaskenv/lib/python3.11/site-packages/weaviate/client.py", line 287, in connect
    self._connection.connect(self.__skip_init_checks)
  File "/home/ec2-user/miniconda3/envs/flaskenv/lib/python3.11/site-packages/weaviate/connect/v4.py", line 711, in connect
    super().connect(skip_init_checks)
  File "/home/ec2-user/miniconda3/envs/flaskenv/lib/python3.11/site-packages/weaviate/connect/v4.py", line 150, in connect
    self._weaviate_version = _ServerVersion.from_string(self.get_meta()["version"])
                                                        ^^^^^^^^^^^^^^^
  File "/home/ec2-user/miniconda3/envs/flaskenv/lib/python3.11/site-packages/weaviate/connect/v4.py", line 599, in get_meta
    res = _decode_json_response_dict(response, "Meta endpoint")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/miniconda3/envs/flaskenv/lib/python3.11/site-packages/weaviate/util.py", line 943, in _decode_json_response_dict
    raise UnexpectedStatusCodeError(location, response)
weaviate.exceptions.UnexpectedStatusCodeError: Meta endpoint! Unexpected status code: 400, with response body: None.

Here’s my connection setup:

weaviate.connect_to_custom(
        http_host="internal-my-alb.elb.amazonaws.com",
        http_port=80,
        http_secure=False,
        grpc_host="internal-my-alb.elb.amazonaws.com",
        grpc_port=50051,
        grpc_secure=False,
        skip_init_checks=False
 )

I’m using a self-signed cert through AWS Managed Certificates with a https listener that just forwards the traffic to the http/grpc service. Has anyone else ran into this issue? I tried using the environment variables with the cert but didn’t get any luck.

Server Setup Information

  • Weaviate Server Version: 1.24.13
  • Deployment Method: Docker on ECS
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: 4.6.4
  • Multitenancy?: Yes

Any additional Information

I don’t have any issues using curl and grpcurl to hit my service, only when I try to create the client is when I start running into issues.

2

Hi!

I have just updated a docker compose that will spin not only Weaviate, but also traefik along side of it, that will help you define secure endpoints for Weaviate.

This is not exactly your scenario, but can help:

Note that on Traefik case, you need a specific scheme (h2c) for routing GRPC:

      # # grpc
      - "traefik.http.services.weaviate_grpc_service.loadbalancer.server.scheme=h2c"
      - "traefik.http.services.weaviate_grpc_service.loadbalancer.server.port=50051"
      - "traefik.http.routers.weaviate_grpc_router.rule=Host(`grpc.weaviate.yourdomain.com`)"
      - "traefik.http.routers.weaviate_grpc_router.entrypoints=grpc"
      - "traefik.http.routers.weaviate_grpc_router.service=weaviate_grpc_service"
      - "traefik.http.routers.weaviate_grpc_router.tls.certresolver=myresolver"

Let me know if this helps.

Otherwise, I would need some approximate steps on how you deployed this in AWS so I could try reproducing those.

Thanks!

3

I came across that looking around. That solution looks great but this is internally hosted so it doesn’t need a proxy.

At a high level, I’m trying to establish an HTTPS connection to get around HTTP/2 being required for connecting to the gRPC port with a load balancer instead of connecting to the cluster IP itself.

To recreate,

  1. Create an Application Load balancer
  2. Create a self-signed cert using AWS ACM (I followed this: Adding a Self-Signed SSL Certificate to AWS ACM | by chamila de alwis | Medium)
  3. Create 2 listeners for port 80 and port 50051 on HTTPS with the cert.
  4. Create 2 target groups for that same port using HTTP as the protocol.

I can add more details about adding Weaviate to ECS, but it’s a pretty simple setup.

I’m mainly confused because curl and grpcurl work without issue, but the client gives me a 400 error. Maybe it’s a red herring, but it seems like this would work without issue.

4

Quick follow up, is it possible to setup the weaviate client to use https but keep secure off? I think root cause is I want to use a secure connection to hit the ALB, but treat it as internal.

I’m running into a similar issue using gRPC, but I have to use a secure connection.

5

Found a solution for anyone struggling. It’s a hack, but I ended up using Service Discovery which generates one DNS A record if a Load Balancer is routing traffic to it. From that you can grab that the record in Route53 under a private namespace and use that for internal traffic to mimic what Kubernetes does.

1 Like
6

Thanks for sharing, @dhanshew72 !!