Hello,
As hopefully I have proven, I’m able to access the port (50051) which, according to the documentation, is the only requirement. So, I have the highest doubts that it’s the firewall issue. If that isn’t enough, I’d appreciate providing information on the actual requirements.
I don’t know what slack you refer to, and I think more people would benefit from keeping the conversation open; if they stumble on the same problem, they could also benefit. For example, this ticket was open 10 days ago and I have the exact issue. I don’t know if the previous asker fixed their problem, or just got gave up all together.
Side note: are you suggesting that disabling the initial checks (that’s my guess is behind skip_init_checks=False) disables a feature? Based on the source code, that doesn’t seem to be true. The check makes a requests and then investigates why it has failed. There also seems to be some requirement for openid when doing the init check which is missing from documentation.
I can’t add new response, so editing this one;
Is TLS (https/ssl) required? If so, I’d urge Weaviate to update the documentation. I’m getting exception
❯ grpcurl -d '{"service": "Weaviate"}' -proto health.proto $RHOST:50051 grpc.health.v1.Health/Check
Failed to dial target host "54.173.187.190:50051": remote error: tls: unexpected message
In all honesty, without the check things work, and as I’m setting this up for someone else, I’m going to leave it as is. I’m not keen on slack conversation as I haven’t had smooth experience so far, and this is at the bottom of my priority list.
In case anyone from Weaviate is interested debugging the issue, here’s a terraform script to setup the use case:
resource "aws_iam_role" "ecs_execution_role" {
name = "mlops-terraform-vector-db"
managed_policy_arns = [
"arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy",
"arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
"arn:aws:iam::aws:policy/AmazonBedrockFullAccess",
]
inline_policy {
name = "bedrock-ecr"
policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Sid" : "PermissiveECR",
"Effect" : "Allow",
"Action" : [
"ecr:*"
],
"Resource" : "*"
},
{
"Sid" : "MarketplaceSubscriptionToLLMs",
"Effect" : "Allow",
"Action" : [
"aws-marketplace:Subscribe"
],
"Resource" : "*",
"Condition" : {
"ForAnyValue:StringEquals" : {
"aws-marketplace:ProductId" : [ # check: https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html#model-access-permissions
"c468b48a-84df-43a4-8c46-8870630108a7", # Anthropic Claude 12k
"prod-ariujvyzvd2qy" # Meta Llama 2 13B
]
}
}
}
]
})
}
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
"Sid" : "",
"Effect" : "Allow",
"Principal" : {
"Service" : "ecs-tasks.amazonaws.com"
},
"Action" : "sts:AssumeRole"
}
]
})
}
resource "aws_ecs_cluster" "weaviate_cluster" {
name = "weaviate_cluster"
}
resource "aws_ecs_task_definition" "weaviate_task" {
family = "weaviate_task"
network_mode = "awsvpc"
requires_compatibilities = ["FARGATE"]
execution_role_arn = aws_iam_role.ecs_execution_role.arn
cpu = "1024"
memory = "2048"
container_definitions = jsonencode([{
name = "weaviate_container"
image = "semitechnologies/weaviate:1.23.9" # TODO: Might need to migrate to own ECR
command = ["--host", "0.0.0.0", "--port", "8080", "--scheme", "http"]
healthCheck = {
command = ["CMD-SHELL", "wget --no-verbose --tries=3 --spider http://localhost:8080/v1/.well-known/ready || exit 1"]
interval = 30
timeout = 5
retries = 3
startPeriod = 0
}
portMappings = [{
containerPort = 8080
hostPort = 8080
protocol = "tcp"
}, {
containerPort = 443
hostPort = 443
protocol = "tcp"
}, {
containerPort = 80
hostPort = 80
protocol = "tcp"
}, {
containerPort = 50051
hostPort = 50051
protocol = "tcp"
}
]
environment = [
{ "name" : "ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE", "value" : "true" },
{ "name" : "AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED", "value" : "true" },
{ "name" : "DEFAULT_VECTORIZER_MODULE", "value" : "text2vec-aws" },
{ "name" : "ENABLE_MODULES", "value" : "text2vec-aws,generative-aws" },
{ "name" : "PERSISTENCE_DATA_PATH", "value" : "/var/lib/weaviate" },
{ "name" : "QUERY_DEFAULTS_LIMIT", "value" : "25" },
{ "name" : "CLUSTER_HOSTNAME", "value" : "node1" },
]
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = aws_cloudwatch_log_group.weaviate_logs.name
"awslogs-region" = var.aws_region # TODO: Might need updating
"awslogs-stream-prefix" = "ecs"
}
}
}])
}
resource "aws_ecs_service" "weaviate_service" {
name = "weaviate_service"
cluster = aws_ecs_cluster.weaviate_cluster.id
task_definition = aws_ecs_task_definition.weaviate_task.arn
launch_type = "FARGATE"
desired_count = 1
network_configuration {
assign_public_ip = true
subnets = [
# TODO: Add appropriate subnets
]
security_groups = [
# TODO: Add appropriate security groups
]
}
}
resource "aws_cloudwatch_log_group" "weaviate_logs" {
name = "/ecs/weaviate_service"
}
resource "aws_cloudwatch_log_stream" "weaviate_logs_stream" {
name = "weaviate_service_logs"
log_group_name = aws_cloudwatch_log_group.weaviate_logs.name
depends_on = [aws_cloudwatch_log_group.weaviate_logs]
}
