WeaviateGRPCUnavailableError: gRPC is not available

I would like to connect to a remote weaviate instance deployed using docker. The problem I have is that when I run connect_to_custom I get WeaviateGRPCUnavailableError.

I am using Python’s weaviate-client v4.4.0, the version of Weaviate is the latest (1.23.7). The equivalent with v3 (using http) works great. The client code:

client2 = weaviate.connect_to_custom(
    http_host="myhost",
    http_port=8080,
    http_secure=False,
    grpc_host="myhost",
    grpc_port=50051,
    grpc_secure=False
)

The docker services:

  weaviate:
    image: semitechnologies/weaviate:latest
    restart: on-failure:0
    ports:
    - 8080:8080
    - 50051:50051
    volumes:
      - /home/...:/var/lib/weaviate
    environment:
      QUERY_DEFAULTS_LIMIT: 20
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: "./data"
      DEFAULT_VECTORIZER_MODULE: text2vec-transformers
      ENABLE_MODULES: text2vec-transformers
      TRANSFORMERS_INFERENCE_API: http://t2v-transformers:8080
      CLUSTER_HOSTNAME: 'node1'
      
  t2v-transformers:
    image: semitechnologies/transformers-inference:sentence-transformers-multi-qa-MiniLM-L6-cos-v1
    environment:
      ENABLE_CUDA: 0 # set to 1 to enable

Hi @EntrustGabri

This has worked.

can you check if the running version is indeed 1.23.7?

You can get this at:
http://localhost:8080/v1/meta

My guess here is that you have an old version with the latest tag.

any version other than 1.23.7 in that endpoint, you can update the image with:

docker compose pull weaviate

Let me know if this helps :slight_smile:

I have checked http://myremotehost:8080/v1/meta and it displays 1.23.7 . In fact, before using “latest” I had “1.23.7”, I changed it to try to fix the error. I have deployed the same locally and it works with connect_to_local(). The problem seems to be when connecting to a remote host. The port is reachable so I don’t know what might be going wrong.

Do you see any outstanding log in server?

if you run a python http server on that port, can you reach it?

Please, feel free to reach me at our Slack and I can try looking into that closely on a call session.

Thanks!

What was the solution here?

I’m having the exact same issue on freshly installed latest Python client (v4.4.4) and the latest image (1.23.9). I can confirm, through pinging from local host (ping -p 50051 hostIp) that the host is reachable. Also, I can see in logs on the remote host, that the gRPC is setup

{"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-02-16T04:14:25Z"}

Could someone from Weaviate please update the documentation on how to setup the gRPC? It says that only exposing port 50051 is enough, yet, when an error on the client shows, it says to make sure it’s configured correctly.

Actually, seems that passing skip_init_checks=True to connect_to_custom fixes the issue, i.e.

client = weaviate.connect_to_custom(
    http_host="myhost",
    http_port=8080,
    http_secure=False,
    grpc_host="myhost",
    grpc_port=50051,
    grpc_secure=False,
   skip_init_checks=True,
)

client.connect()

Hi! I believe this may fix it partially, as it will not leverage the GRPC part of the client.

I have seen this kind of issue, and usually it was related to firewall, both on client and on server.

Please, feel free to ping me in our slack channel so I can take a closer look.

Thanks!

Hello,

As hopefully I have proven, I’m able to access the port (50051) which, according to the documentation, is the only requirement. So, I have the highest doubts that it’s the firewall issue. If that isn’t enough, I’d appreciate providing information on the actual requirements.

I don’t know what slack you refer to, and I think more people would benefit from keeping the conversation open; if they stumble on the same problem, they could also benefit. For example, this ticket was open 10 days ago and I have the exact issue. I don’t know if the previous asker fixed their problem, or just got gave up all together.

Side note: are you suggesting that disabling the initial checks (that’s my guess is behind skip_init_checks=False) disables a feature? Based on the source code, that doesn’t seem to be true. The check makes a requests and then investigates why it has failed. There also seems to be some requirement for openid when doing the init check which is missing from documentation.


I can’t add new response, so editing this one;

Is TLS (https/ssl) required? If so, I’d urge Weaviate to update the documentation. I’m getting exception

❯ grpcurl -d '{"service": "Weaviate"}' -proto health.proto $RHOST:50051 grpc.health.v1.Health/Check
Failed to dial target host "54.173.187.190:50051": remote error: tls: unexpected message

In all honesty, without the check things work, and as I’m setting this up for someone else, I’m going to leave it as is. I’m not keen on slack conversation as I haven’t had smooth experience so far, and this is at the bottom of my priority list.

In case anyone from Weaviate is interested debugging the issue, here’s a terraform script to setup the use case:

resource "aws_iam_role" "ecs_execution_role" {
  name = "mlops-terraform-vector-db"

  managed_policy_arns = [
    "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy",
    "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
    "arn:aws:iam::aws:policy/AmazonBedrockFullAccess",
  ]

  inline_policy {
    name = "bedrock-ecr"
    policy = jsonencode({
      "Version" : "2012-10-17",
      "Statement" : [
        {
          "Sid" : "PermissiveECR",
          "Effect" : "Allow",
          "Action" : [
            "ecr:*"
          ],
          "Resource" : "*"
        },
        {
          "Sid" : "MarketplaceSubscriptionToLLMs",
          "Effect" : "Allow",
          "Action" : [
            "aws-marketplace:Subscribe"
          ],
          "Resource" : "*",
          "Condition" : {
            "ForAnyValue:StringEquals" : {
              "aws-marketplace:ProductId" : [  # check: https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html#model-access-permissions
                "c468b48a-84df-43a4-8c46-8870630108a7", # Anthropic Claude 12k
                "prod-ariujvyzvd2qy"  # Meta Llama 2 13B
              ]
            }
          }
        }
      ]
    })
  }

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        "Sid" : "",
        "Effect" : "Allow",
        "Principal" : {
          "Service" : "ecs-tasks.amazonaws.com"
        },
        "Action" : "sts:AssumeRole"
      }
    ]
  })

}

resource "aws_ecs_cluster" "weaviate_cluster" {
  name = "weaviate_cluster"
}

resource "aws_ecs_task_definition" "weaviate_task" {
  family                   = "weaviate_task"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  execution_role_arn       = aws_iam_role.ecs_execution_role.arn
  cpu                      = "1024"
  memory                   = "2048"


  container_definitions = jsonencode([{
    name    = "weaviate_container"
    image   = "semitechnologies/weaviate:1.23.9" # TODO: Might need to migrate to own ECR
    command = ["--host", "0.0.0.0", "--port", "8080", "--scheme", "http"]

    healthCheck = {
      command     = ["CMD-SHELL", "wget --no-verbose --tries=3 --spider http://localhost:8080/v1/.well-known/ready || exit 1"]
      interval    = 30
      timeout     = 5
      retries     = 3
      startPeriod = 0
    }

    portMappings = [{
      containerPort = 8080
      hostPort      = 8080
      protocol      = "tcp"
      }, {
      containerPort = 443
      hostPort      = 443
      protocol      = "tcp"
      }, {
      containerPort = 80
      hostPort      = 80
      protocol      = "tcp"
      }, {
      containerPort = 50051
      hostPort      = 50051
      protocol      = "tcp"
      }
    ]
    environment = [
      { "name" : "ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE", "value" : "true" },
      { "name" : "AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED", "value" : "true" },
      { "name" : "DEFAULT_VECTORIZER_MODULE", "value" : "text2vec-aws" },
      { "name" : "ENABLE_MODULES", "value" : "text2vec-aws,generative-aws" },
      { "name" : "PERSISTENCE_DATA_PATH", "value" : "/var/lib/weaviate" },
      { "name" : "QUERY_DEFAULTS_LIMIT", "value" : "25" },
      { "name" : "CLUSTER_HOSTNAME", "value" : "node1" },
    ]

    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = aws_cloudwatch_log_group.weaviate_logs.name
        "awslogs-region"        = var.aws_region  # TODO: Might need updating
        "awslogs-stream-prefix" = "ecs"
      }
    }
  }])
}

resource "aws_ecs_service" "weaviate_service" {
  name            = "weaviate_service"
  cluster         = aws_ecs_cluster.weaviate_cluster.id
  task_definition = aws_ecs_task_definition.weaviate_task.arn
  launch_type     = "FARGATE"
  desired_count   = 1


  network_configuration {
    assign_public_ip = true
    subnets = [
      # TODO: Add appropriate subnets
    ]
    security_groups = [
      # TODO: Add appropriate security groups
    ]
  }

}
resource "aws_cloudwatch_log_group" "weaviate_logs" {
  name = "/ecs/weaviate_service"
}

resource "aws_cloudwatch_log_stream" "weaviate_logs_stream" {
  name           = "weaviate_service_logs"
  log_group_name = aws_cloudwatch_log_group.weaviate_logs.name
  depends_on     = [aws_cloudwatch_log_group.weaviate_logs]
}

Hi!

We have a public slack. You can access it from our website, or directly thru:
https://weaviate.io/slack

My suggestion is that you could share your screen and I could take a closer look.

In my experience, this kind of issue is better solved like that :slight_smile:

We can for sure bring back our findings here so others can benefit.

One way to make sure the GRPC is healthy and serving, is to do a grpcurl, like so:

# lets test our grpc connection
❯ wget https://raw.githubusercontent.com/grpc/grpc/master/src/proto/grpc/health/v1/health.proto
❯ grpcurl -d '{"service": "Weaviate"}' -proto health.proto grpc.weaviate.mydomain.com:50051 grpc.health.v1.Health/Check
{
  "status": "SERVING"
}

Here I have created a docker compose that uses Weaviate + Traefik + Let’s Encrypt, so you can server Weaviate behind a reverse proxy with https/ssl: