Description
Yep, you read the title correctly. The health check is working but the connection doesn’t work. I tried all the solutions so far suggested on this forum but nothing seems to work.
I am using single docker file
version: '3.9'
networks:
redis-network:
driver: bridge
db-network:
driver: bridge
core-api-network:
driver: bridge
rabbitmq-network:
driver: bridge
internal-api-network:
driver: bridge
celery-network:
driver: bridge
weaviate-network:
driver: bridge
volumes:
postgresql-data: {}
weaviate_data: {}
shared-upload-data: {}
internal-mongo-data: {}
redis-data: {}
services:
db:
...
redis:
...
core_api_server:
...
rabbitmq:
...
celery-worker-1:
image: internal_celery
scale: 1
build:
context: internal
dockerfile: Dockerfile/local/Dockerfile.celery
command: python3 -m celery -A app.celery_app:celery_app worker -Q source_op_queue --concurrency=3 --without-heartbeat --without-gossip --without-mingle
env_file:
- 'internal/.env/.api'
- 'internal/.env/.rabbitmq'
- 'internal/.env/.weaviate'
depends_on:
redis:
condition: service_started
rabbitmq:
condition: service_started
weaviate:
condition: service_healthy
volumes:
- shared-upload-data:/.upload-cache # Mount the shared volume in the fastapi_server container
networks:
- celery-network
- redis-network
- rabbitmq-network
- weaviate-network
deploy:
resources:
limits:
cpus: '3'
memory: 2G
reservations:
cpus: '2'
memory: 1.5G
fastapi_server:
image: internal_api_server
build:
context: internal
dockerfile: Dockerfile/local/Dockerfile.api
env_file:
- 'internal/.env/.api'
- 'internal/.env/.rabbitmq'
- 'internal/.env/.weaviate'
ports:
- '8000:8000'
volumes:
- ./internal:/app
- shared-upload-data:/.upload-cache
restart: always
depends_on:
weaviate:
condition: service_healthy
deploy:
replicas: 1
resources:
limits:
cpus: '3'
memory: 2G
reservations:
cpus: '2'
memory: 1.5G
networks:
- redis-network
- rabbitmq-network
- core-api-network
- internal-api-network
- weaviate-network
weaviate:
command:
- "--host=0.0.0.0"
- "--port=8080"
- "--scheme=http"
image: semitechnologies/weaviate:1.26.6
ports:
- 8080:8080
- 50051:50051
env_file: 'internal/.env/.weaviate'
volumes:
- weaviate_data:/var/lib/weaviate
restart: on-failure:0
healthcheck:
test: wget --no-verbose --tries=3 --spider http://localhost:8080/v1/.well-known/ready || exit 1
interval: 5s
timeout: 10s
retries: 5
start_period: 10s
networks:
- weaviate-network
deploy:
resources:
limits:
cpus: '2'
memory: 2.5G
reservations:
cpus: '1'
memory: 1.5G
Execution steps:
The FastAPI app, main.py has an import statement that goes 3-4 levels inside of different files and instantiates the weaviate client.
def warmup_weaviate_client():
weaviate_host = os.environ.get('WEAVIATE_HOST', "localhost")
weaviate_url = f"http://{weaviate_host}:8080"
retries = 5
while retries > 0:
with httpx.Client() as client:
try:
response = client.get(f"{weaviate_url}/v1/.well-known/ready")
if response.status_code == 200:
logger.info("Weaviate is ready")
client = weaviate.connect_to_local(
host=weaviate_host,
headers=headers,
additional_config=AdditionalConfig(
timeout=Timeout(init=5, query=60, insert=120) # Values in seconds
)
)
client.connect()
return client
else:
logger.warning(f"Weaviate not ready, status code: {response.status_code}")
except httpx.RequestError as exc:
logger.warning(f"An error occurred while checking Weaviate readiness: {exc}")
retries -= 1
if retries > 0:
logger.info(f"Retrying in 5 seconds... ({retries} retries left)")
time.sleep(5)
return None
def get_weaviate_client():
weaviate_host = os.environ.get('WEAVIATE_HOST', "localhost")
# weaviate_port = os.environ.get('WEAVIATE_PORT', "8080")
logger.info("XXXXXXX - Creating Weaviate Client")
# is_anonymous = os.environ.get("AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED", "false")
is_anonymous = "true" # For now, we are using anonymous access - API method does not work
if is_anonymous == "true":
auth_client_secret=None
else:
logger.info("Secured API connection to Weaviate is being made")
weaviate_api_key = os.environ.get('WEAVIATE_API_KEY')
if not weaviate_api_key:
logger.error(f"WEAVIATE_API_KEY missing")
# return {"error": "ENV_VARIABLE_MISSING", "reason": "WEAVIATE_API_KEY not found"}
auth_client_secret=Auth.api_key(weaviate_api_key)
client = warmup_weaviate_client()
if not client:
logger.error("WEAVIATE_CONNECTION_ERROR: Weviate connection failed")
logger.info("Weaviate connection established")
# return {"error": "WEAVIATE_CONNECTION_ERROR", "reason": str(e)}
# Wait for Weaviate to be ready
wait_time_limit = 240
while not client.is_ready():
if wait_time_limit <= 0:
logger.error("WEAVIATE_TIMEOUT_ERROR: Weviate waited for too long")
# return {"error": "WEAVIATE_TIMEOUT_ERROR", "reason": ERROR_CODES["WEAVIATE_TIMEOUT_ERROR"]}
sys.stdout.write(f"\rWait for Weaviate to get ready. {wait_time_limit:02d} seconds left.")
sys.stdout.flush()
wait_time_limit -= 5
time.sleep(5.0)
try:
create_collections(client)
except Exception as e:
logger.error(f"WEAVIATE_SCHEMA_ERROR: Issues with Weviate Schema: {str(e)}")
return {"error": "WEAVIATE_SCHEMA_ERROR", "reason": str(e)}
return client
## CREATING THE CLIENT
client = get_weaviate_client()
client.close()
I was so fed up with the whole issue that I had to add the health readiness step inside the code. Anyway, in the logs I see “Weaviate is ready” but not “Weaviate connection established”. It kinda proves that the container has access to http://weaviate:8080 but not able to make the connection.
Server Setup Information
I have a RAG application that has 1 celery containers, 1 fast API server, 1 weaviate, and 1 database.
- Single docker compose file for all.
- Trying to access
http://weaviate:8080but the client is not working. When I go inside the celery or fastapi container, the health check works. - More importantly, when I manually execute the script in python cli inside the container, it works too. But when I try to use it as a whole application (making request from frontend) it just does not work.
Any additional Information
I tried the simpler setup where I created a dummy fastapi service which uses the same weaviate client method. That is working, but for some reason my above setup is not working.
Please help.