Hi!
I am trying to test my weaviate flows using github actions and I’m having trouble starting the weaviate server. I am using the following action.yaml file based on the docker-compose.yaml file in the docs:
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11"]
services:
weaviate:
image: cr.weaviate.io/semitechnologies/weaviate:1.26.1
env:
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'text2vec-openai'
ENABLE_MODULES: 'text2vec-openai'
CLUSTER_HOSTNAME: 'node1'
ports:
- 8080:8080
options: >-
--health-cmd "curl --request GET --url http://localhost:8080/v1/ || exit 1 || exit 1"
--health-interval 10s
--health-timeout 5s
--health-retries 5
But even though I’m able to see weaviate logs in the action logs, the health command times out.
Here are the full logs:
Summary
2024-08-29T05:50:36.1165582Z Current runner version: '2.319.1'
2024-08-29T05:50:36.1188282Z ##[group]Operating System
2024-08-29T05:50:36.1188917Z Ubuntu
2024-08-29T05:50:36.1189390Z 22.04.4
2024-08-29T05:50:36.1189681Z LTS
2024-08-29T05:50:36.1190007Z ##[endgroup]
2024-08-29T05:50:36.1190463Z ##[group]Runner Image
2024-08-29T05:50:36.1190859Z Image: ubuntu-22.04
2024-08-29T05:50:36.1191282Z Version: 20240825.1.0
2024-08-29T05:50:36.1192352Z Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240825.1/images/ubuntu/Ubuntu2204-Readme.md
2024-08-29T05:50:36.1193773Z Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240825.1
2024-08-29T05:50:36.1194659Z ##[endgroup]
2024-08-29T05:50:36.1195131Z ##[group]Runner Image Provisioner
2024-08-29T05:50:36.1195586Z 2.0.384.1
2024-08-29T05:50:36.1195937Z ##[endgroup]
2024-08-29T05:50:36.1210542Z ##[group]GITHUB_TOKEN Permissions
2024-08-29T05:50:36.1212234Z Contents: read
2024-08-29T05:50:36.1212682Z Metadata: read
2024-08-29T05:50:36.1213198Z Packages: read
2024-08-29T05:50:36.1213756Z ##[endgroup]
2024-08-29T05:50:36.1216675Z Secret source: Actions
2024-08-29T05:50:36.1217360Z Prepare workflow directory
2024-08-29T05:50:36.2113503Z Prepare all required actions
2024-08-29T05:50:36.2270076Z Getting action download info
2024-08-29T05:50:36.4186244Z Download action repository 'actions/checkout@v3' (SHA:f43a0e5ff2bd294095638e18286ca9a3d1956744)
2024-08-29T05:50:36.5144524Z Download action repository 'actions/setup-python@v3' (SHA:3542bca2639a428e1796aaa6a2ffef0c0f575566)
2024-08-29T05:50:36.5948105Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32)
2024-08-29T05:50:36.7702724Z Complete job name: build (3.11)
2024-08-29T05:50:36.8282281Z ##[group]Checking docker version
2024-08-29T05:50:36.8296617Z ##[command]/usr/bin/docker version --format '{{.Server.APIVersion}}'
2024-08-29T05:50:36.8847308Z '1.45'
2024-08-29T05:50:36.8885650Z Docker daemon API version: '1.45'
2024-08-29T05:50:36.8886937Z ##[command]/usr/bin/docker version --format '{{.Client.APIVersion}}'
2024-08-29T05:50:36.9049229Z '1.45'
2024-08-29T05:50:36.9062579Z Docker client API version: '1.45'
2024-08-29T05:50:36.9067743Z ##[endgroup]
2024-08-29T05:50:36.9070978Z ##[group]Clean up resources from previous jobs
2024-08-29T05:50:36.9076938Z ##[command]/usr/bin/docker ps --all --quiet --no-trunc --filter "label=c9bac7"
2024-08-29T05:50:36.9226170Z ##[command]/usr/bin/docker network prune --force --filter "label=c9bac7"
2024-08-29T05:50:36.9354940Z ##[endgroup]
2024-08-29T05:50:36.9355453Z ##[group]Create local container network
2024-08-29T05:50:36.9366171Z ##[command]/usr/bin/docker network create --label c9bac7 github_network_53f3d15128bb4222b79e014618e62d45
2024-08-29T05:50:37.0065716Z c20769e4fc16559c0d2b18ad0c4ac0936f7c4ee5a0dcb25df1f2f620992040f5
2024-08-29T05:50:37.0084879Z ##[endgroup]
2024-08-29T05:50:37.0161484Z ##[group]Starting weaviate service container
2024-08-29T05:50:37.0210636Z ##[command]/usr/bin/docker pull cr.weaviate.io/semitechnologies/weaviate:1.26.1
2024-08-29T05:50:38.2632498Z 1.26.1: Pulling from semitechnologies/weaviate
2024-08-29T05:50:38.6226670Z c6a83fedfae6: Already exists
2024-08-29T05:50:38.6236297Z 1b38ab12ecb2: Pulling fs layer
2024-08-29T05:50:38.6238557Z 132c4bf96a52: Pulling fs layer
2024-08-29T05:50:38.6240257Z 0eb39e2e2a5e: Pulling fs layer
2024-08-29T05:50:38.6246150Z 9efb6211700a: Pulling fs layer
2024-08-29T05:50:38.6247094Z 3f7fbcd922fe: Pulling fs layer
2024-08-29T05:50:38.6247930Z 345cef12759b: Pulling fs layer
2024-08-29T05:50:38.6248845Z 9efb6211700a: Waiting
2024-08-29T05:50:38.6249525Z 3f7fbcd922fe: Waiting
2024-08-29T05:50:38.6250167Z 345cef12759b: Waiting
2024-08-29T05:50:38.9660801Z 0eb39e2e2a5e: Verifying Checksum
2024-08-29T05:50:38.9665949Z 0eb39e2e2a5e: Download complete
2024-08-29T05:50:39.1484013Z 1b38ab12ecb2: Verifying Checksum
2024-08-29T05:50:39.1485585Z 1b38ab12ecb2: Download complete
2024-08-29T05:50:39.3134528Z 132c4bf96a52: Verifying Checksum
2024-08-29T05:50:39.3140629Z 132c4bf96a52: Download complete
2024-08-29T05:50:39.4746733Z 1b38ab12ecb2: Pull complete
2024-08-29T05:50:39.5256989Z 9efb6211700a: Verifying Checksum
2024-08-29T05:50:39.5260315Z 9efb6211700a: Download complete
2024-08-29T05:50:39.6483197Z 3f7fbcd922fe: Verifying Checksum
2024-08-29T05:50:39.6484889Z 3f7fbcd922fe: Download complete
2024-08-29T05:50:39.6824520Z 345cef12759b: Verifying Checksum
2024-08-29T05:50:39.6825714Z 345cef12759b: Download complete
2024-08-29T05:50:40.2567027Z 132c4bf96a52: Pull complete
2024-08-29T05:50:40.2765124Z 0eb39e2e2a5e: Pull complete
2024-08-29T05:50:40.6291500Z 9efb6211700a: Pull complete
2024-08-29T05:50:40.7352983Z 3f7fbcd922fe: Pull complete
2024-08-29T05:50:40.7466497Z 345cef12759b: Pull complete
2024-08-29T05:50:40.7523309Z Digest: sha256:987edeaf57b6fefe9b8bea0e62f848028f99a527100259e2760a1c9b57be6f4c
2024-08-29T05:50:40.7592835Z Status: Downloaded newer image for cr.weaviate.io/semitechnologies/weaviate:1.26.1
2024-08-29T05:50:40.7599412Z cr.weaviate.io/semitechnologies/weaviate:1.26.1
2024-08-29T05:50:40.7707579Z ##[command]/usr/bin/docker create --name 70bf7ba7fa50490d83ba3b9b025bc359_crweaviateiosemitechnologiesweaviate1261_bb5cd5 --label c9bac7 --network github_network_53f3d15128bb4222b79e014618e62d45 --network-alias weaviate -p 8080:8080 --health-cmd "curl -f http://localhost:8080/v1/.well-known/ready || exit 1" --health-interval 10s --health-timeout 5s --health-retries 5 -e "QUERY_DEFAULTS_LIMIT=25" -e "AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true" -e "PERSISTENCE_DATA_PATH=/var/lib/weaviate" -e "DEFAULT_VECTORIZER_MODULE=text2vec-openai" -e "ENABLE_MODULES=text2vec-openai" -e "CLUSTER_HOSTNAME=node1" -e GITHUB_ACTIONS=true -e CI=true cr.weaviate.io/semitechnologies/weaviate:1.26.1
2024-08-29T05:50:40.8133048Z c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:50:40.8158907Z ##[command]/usr/bin/docker start c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:50:41.1274133Z c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:50:41.1306414Z ##[command]/usr/bin/docker ps --all --filter id=c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9 --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
2024-08-29T05:50:41.1460732Z c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9 Up Less than a second (health: starting)
2024-08-29T05:50:41.1495560Z ##[command]/usr/bin/docker port c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:50:41.1701091Z 8080/tcp -> 0.0.0.0:8080
2024-08-29T05:50:41.1702166Z 8080/tcp -> [::]:8080
2024-08-29T05:50:41.1813029Z ##[endgroup]
2024-08-29T05:50:41.1848664Z ##[group]Waiting for all services to be ready
2024-08-29T05:50:41.1902692Z ##[command]/usr/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:50:41.2093202Z starting
2024-08-29T05:50:41.2127250Z weaviate service is starting, waiting 2 seconds before checking again.
2024-08-29T05:50:43.2108069Z ##[command]/usr/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:50:43.2267597Z starting
2024-08-29T05:50:43.2281643Z weaviate service is starting, waiting 4 seconds before checking again.
2024-08-29T05:50:47.3536289Z ##[command]/usr/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:50:47.3666962Z starting
2024-08-29T05:50:47.3682013Z weaviate service is starting, waiting 8 seconds before checking again.
2024-08-29T05:50:56.1057165Z ##[command]/usr/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:50:56.1194519Z starting
2024-08-29T05:50:56.1210790Z weaviate service is starting, waiting 18 seconds before checking again.
2024-08-29T05:51:14.3098885Z ##[command]/usr/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:51:14.3232438Z starting
2024-08-29T05:51:14.3247699Z weaviate service is starting, waiting 31 seconds before checking again.
2024-08-29T05:51:45.9027651Z ##[command]/usr/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:51:45.9179298Z unhealthy
2024-08-29T05:51:45.9198290Z ##[endgroup]
2024-08-29T05:51:45.9198660Z ##[group]Service container weaviate failed.
2024-08-29T05:51:45.9204408Z ##[command]/usr/bin/docker logs --details c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:51:45.9344819Z {"action":"startup","default_vectorizer_module":"text2vec-openai","level":"info","msg":"the default vectorizer modules is set to \"text2vec-openai\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9361516Z {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9365017Z {"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9370607Z {"level":"info","msg":"module offload-s3 is enabled","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9373273Z {"level":"warning","msg":"Multiple vector spaces are present, GraphQL Explore and REST API list objects endpoint module include params has been disabled as a result.","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9375595Z {"level":"info","msg":"open cluster service","servers":{"node1":8300},"time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9376946Z {"address":"172.18.0.2:8301","level":"info","msg":"starting cloud rpc server ...","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9377731Z {"level":"info","msg":"starting raft sub-system ...","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9378714Z {"address":"172.18.0.2:8300","level":"info","msg":"tcp transport","tcpMaxPool":3,"tcpTimeout":10000000000,"time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9379591Z {"level":"info","msg":"loading local db","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9380260Z {"level":"info","msg":"local DB successfully loaded","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9380996Z {"level":"info","msg":"schema manager loaded","n":0,"time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9381872Z {"level":"info","metadata_only_voters":false,"msg":"construct a new raft node","name":"node1","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9382856Z {"action":"raft","index":0,"level":"info","msg":"raft initial configuration","servers":"[[]]","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9384196Z {"last_snapshot_index":0,"last_store_applied_index":0,"last_store_log_applied_index":0,"level":"info","msg":"raft node constructed","raft_applied_index":0,"raft_last_index":0,"time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9385499Z {"action":"raft","follower":{},"leader-address":"","leader-id":"","level":"info","msg":"raft entering follower state","time":"2024-08-29T05:50:41Z"}
2024-08-29T05:51:45.9386924Z {"action":"bootstrap","error":"could not join a cluster from [172.18.0.2:8300]","level":"warning","msg":"failed to join cluster, will notify next if voter","servers":["172.18.0.2:8300"],"time":"2024-08-29T05:50:42Z","voter":true}
2024-08-29T05:51:45.9388600Z {"action":"bootstrap","candidates":[{"Suffrage":0,"ID":"node1","Address":"172.18.0.2:8300"}],"level":"info","msg":"starting cluster bootstrapping","time":"2024-08-29T05:50:42Z"}
2024-08-29T05:51:45.9389860Z {"action":"bootstrap","level":"info","msg":"notified peers this node is ready to join as voter","servers":["172.18.0.2:8300"],"time":"2024-08-29T05:50:42Z"}
2024-08-29T05:51:45.9391342Z {"action":"raft","last-leader-addr":"","last-leader-id":"","level":"warning","msg":"raft heartbeat timeout reached, starting election","time":"2024-08-29T05:50:42Z"}
2024-08-29T05:51:45.9392505Z {"action":"raft","level":"info","msg":"raft entering candidate state","node":{},"term":2,"time":"2024-08-29T05:50:42Z"}
2024-08-29T05:51:45.9393413Z {"action":"raft","level":"info","msg":"raft election won","tally":1,"term":2,"time":"2024-08-29T05:50:42Z"}
2024-08-29T05:51:45.9394282Z {"action":"raft","leader":{},"level":"info","msg":"raft entering leader state","time":"2024-08-29T05:50:42Z"}
2024-08-29T05:51:45.9395303Z {"docker_image_tag":"1.26.1","level":"info","msg":"configured versions","server_version":"1.26.1","time":"2024-08-29T05:50:43Z"}
2024-08-29T05:51:45.9396237Z {"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-08-29T05:50:43Z"}
2024-08-29T05:51:45.9397105Z {"address":"172.18.0.2:8300","level":"info","msg":"current Leader","time":"2024-08-29T05:50:43Z"}
2024-08-29T05:51:45.9397948Z {"level":"info","msg":"starting migration from old schema","time":"2024-08-29T05:50:43Z"}
2024-08-29T05:51:45.9398710Z {"level":"info","msg":"legacy schema is empty, nothing to migrate","time":"2024-08-29T05:50:43Z"}
2024-08-29T05:51:45.9399730Z {"level":"info","msg":"migration from the old schema has been successfully completed","time":"2024-08-29T05:50:43Z"}
2024-08-29T05:51:45.9400907Z {"action":"restapi_management","docker_image_tag":"1.26.1","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-08-29T05:50:43Z"}
2024-08-29T05:51:45.9402605Z {"action":"telemetry_push","level":"info","msg":"telemetry started","payload":"\u0026{MachineID:4b0f3da8-92f2-44c1-8575-515b11f04507 Type:INIT Version:1.26.1 NumObjects:0 OS:linux Arch:amd64 UsedModules:[]}","time":"2024-08-29T05:50:43Z"}
2024-08-29T05:51:45.9404836Z {"action":"bootstrap","level":"info","msg":"node reporting ready, node has probably recovered cluster from raft config. Exiting bootstrap process","time":"2024-08-29T05:50:43Z"}
2024-08-29T05:51:45.9417023Z ##[error]Failed to initialize container cr.weaviate.io/semitechnologies/weaviate:1.26.1
2024-08-29T05:51:45.9424345Z ##[endgroup]
2024-08-29T05:51:45.9511677Z ##[error]One or more containers failed to start.
2024-08-29T05:51:45.9856946Z ##[group]Run actions/upload-artifact@v3
2024-08-29T05:51:45.9857459Z with:
2024-08-29T05:51:45.9857898Z name: pytest-results-3.11
2024-08-29T05:51:45.9858287Z path: server/junit/test-results-3.11.xml
2024-08-29T05:51:45.9858756Z if-no-files-found: warn
2024-08-29T05:51:45.9859175Z env:
2024-08-29T05:51:45.9859448Z WEAVIATE_HOST: localhost
2024-08-29T05:51:45.9859831Z WEAVIATE_API_KEY: dummy
2024-08-29T05:51:45.9860299Z WEAVIATE_COLLECTION_NAME: therapists
2024-08-29T05:51:45.9860682Z OPENAI_API_KEY: dummy
2024-08-29T05:51:45.9861044Z ##[endgroup]
2024-08-29T05:51:46.1199811Z ##[warning]No files were found with the provided path: server/junit/test-results-3.11.xml. No artifacts will be uploaded.
2024-08-29T05:51:46.1370448Z Stop and remove container: 70bf7ba7fa50490d83ba3b9b025bc359_crweaviateiosemitechnologiesweaviate1261_bb5cd5
2024-08-29T05:51:46.1375760Z ##[command]/usr/bin/docker rm --force c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:51:46.3320586Z c8457c49e47654330f35237142bb4f39a5b65e3ac64b45c76b41db380092f4d9
2024-08-29T05:51:46.3354006Z Remove container network: github_network_53f3d15128bb4222b79e014618e62d45
2024-08-29T05:51:46.3359122Z ##[command]/usr/bin/docker network rm github_network_53f3d15128bb4222b79e014618e62d45
2024-08-29T05:51:46.5418396Z github_network_53f3d15128bb4222b79e014618e62d45
2024-08-29T05:51:46.5584804Z Cleaning up orphan processes
Note: I have also tried testing with embedded weaviate, but I didn’t find it reliable enough (probably because it is experimental).