Description
I am trying to run a local instance of Weaviate, but when I start the server it quits many minutes later. I am running on macOS 15.4.1.
I built the server with:
git clone https://github.com/weaviate/weaviate.git
cd weaviate
go build -o weaviate ./cmd/weaviate-server/main.go
and start the server with:
DISABLE_TELEMETRY=true \
CLUSTER_ENABLED=false \
DISK_USE_READONLY_PERCENTAGE=95 \
DISK_USE_WARNING_PERCENTAGE=95
PERSISTENCE_DATA_PATH=./weaviate_data \
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true \
DEFAULT_VECTORIZER_MODULE=none \
ENABLE_MODULES=none \
QUERY_DEFAULTS_LIMIT=50 \
RAFT_ENABLE_ONE_NODE_RECOVERY=true \
LOG_LEVEL=debug \
nohup weaviate --host=localhost --port 38080 --scheme=http > weaviate.log 2>&1 &
I found a GitHub issue #5362 that describes the same problem; it was suggested to use RAFT_ENABLE_ONE_NODE_RECOVERY=true
to address this.
The server starts and runs for a few minutes, then quits (full log below).
I also note the DISK_USE...
variables don’t appear to change things as noted in the log.
Server Setup Information
- Weaviate Server Version: 1.31.0-rc.0
- Deployment Method: binary
- Multi Node? Number of Running Nodes: one node
- Client Language and Version: can’t get the server to stay up yet…
- Multitenancy?:
Any additional Information
This is the full output log:
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"created startup context, nothing done so far","startup_time_left":"59m59.998680875s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"Feature flag LD integration disabled: could not locate WEAVIATE_LD_API_KEY env variable","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","auto_schema_enabled":{},"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"auto schema enabled setting is set to \"\u0026{\u003cnil\u003e {{{} {0 0}} 0 0 {{} 0} {{} 0}} true}\"","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"config loaded","startup_time_left":"59m59.997037833s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"configured OIDC and anonymous access client","startup_time_left":"59m59.996798458s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"initialized schema","startup_time_left":"59m59.996789083s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"startup routine complete","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"start registering modules","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"completed registering modules","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"module offload-s3 is not enabled","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"module offload-s3 is enabled","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","flag_key":"collection-retrieval-strategy","level":"info","msg":"feature flag instantiated","time":"2025-05-13T03:02:53-04:00","tool":"feature_flag","value":"LeaderOnly"}
{"action":"cluster_api_startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"serving cluster api on port 7947","port":7947,"time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"start initializing modules","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"finished initializing modules","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"open cluster service","servers":{"Mac-Studio.local":8300},"time":"2025-05-13T03:02:53-04:00"}
{"address":"10.128.42.220:8301","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"starting cloud rpc server ...","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"starting raft sub-system ...","time":"2025-05-13T03:02:53-04:00"}
{"address":"10.128.42.220:8300","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"tcp transport","tcpMaxPool":3,"tcpTimeout":10000000000,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"loading local db","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"local DB successfully loaded","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"schema manager loaded","n":0,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","metadata_only_voters":false,"msg":"construct a new raft node","name":"Mac-Studio.local","time":"2025-05-13T03:02:53-04:00"}
{"action":"raft","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","index":0,"level":"info","msg":"initial configuration","servers":"[[]]","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","candidates":{},"level":"warning","msg":"didn't perform cluster recovery","servers_from_previous_configuration":null,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","last_snapshot_index":0,"last_store_applied_index_on_start":0,"level":"info","msg":"raft node constructed","raft_applied_index":0,"raft_last_index":0,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","hasState":false,"level":"info","msg":"raft init","time":"2025-05-13T03:02:53-04:00"}
{"action":"raft","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","follower":{},"leader-address":"","leader-id":"","level":"info","msg":"entering follower state","time":"2025-05-13T03:02:53-04:00"}
{"action":"read_disk_use","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"disk usage currently at 87.70%, threshold set to 80.00%","path":"./weaviate_data","time":"2025-05-13T03:02:53-04:00"}
{"action":"disk_use_stats","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"total: 926.35GB, free: 113.93GB, used: 812.42GB (avail: 113.93GB)","path":"./weaviate_data","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempting to join","remoteNodes":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:02:54-04:00"}
{"action":"raft","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"no known peers, aborting election","time":"2025-05-13T03:02:55-04:00"}
{"action":"inverted filter2search migration","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"migration skip flag set, skipping migration","time":"2025-05-13T03:02:55-04:00"}
{"action":"inverted filter2search migration","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"starting switching fallback mode","time":"2025-05-13T03:02:55-04:00"}
{"action":"inverted filter2search migration","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"no missing filterable indexes, fallback mode skipped","time":"2025-05-13T03:02:55-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"configured versions","server_version":"1.31.0-rc.0","time":"2025-05-13T03:02:55-04:00","version":"1.31.0-rc.0"}
{"action":"grpc_startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"grpc server listening at [::]:50051","time":"2025-05-13T03:02:55-04:00"}
{"action":"restapi_management","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"Serving weaviate at http://127.0.0.1:38080","time":"2025-05-13T03:02:55-04:00","version":"1.31.0-rc.0"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"error","msg":"telemetry failed to start: push: failed to send request: Post \"https://telemetry.weaviate.io/weaviate-telemetry\": dial tcp 34.149.252.24:443: connect: bad file descriptor","time":"2025-05-13T03:02:55-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempted to join and failed","remoteNode":"10.128.42.220:8300","status":14,"time":"2025-05-13T03:03:14-04:00"}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:03:14-04:00","voter":true}
{"action":"read_disk_use","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"disk usage currently at 87.70%, threshold set to 80.00%","path":"./weaviate_data","time":"2025-05-13T03:03:24-04:00"}
{"action":"disk_use_stats","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"total: 926.35GB, free: 113.92GB, used: 812.43GB (avail: 113.92GB)","path":"./weaviate_data","time":"2025-05-13T03:03:24-04:00"}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.128.42.220:8301: i/o timeout\"","level":"error","msg":"failed to notify peers","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:03:34-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempting to join","remoteNodes":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:03:34-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempted to join and failed","remoteNode":"10.128.42.220:8300","status":14,"time":"2025-05-13T03:03:34-04:00"}
[repeated "failed to notify peers" messages]
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:05:14-04:00","voter":true}
{"action":"read_disk_use","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"disk usage currently at 87.70%, threshold set to 80.00%","path":"./weaviate_data","time":"2025-05-13T03:05:24-04:00"}
{"action":"disk_use_stats","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"total: 926.35GB, free: 113.93GB, used: 812.42GB (avail: 113.93GB)","path":"./weaviate_data","time":"2025-05-13T03:05:24-04:00"}
[repeated "failed to notify peers" messages]
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:12:34-04:00","voter":true}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"rpc error: code = DeadlineExceeded desc = received context error while waiting for new LB policy update: context deadline exceeded","level":"error","msg":"failed to notify peers","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempting to join","remoteNodes":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempted to join and failed","remoteNode":"10.128.42.220:8300","status":4,"time":"2025-05-13T03:13:21-04:00"}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00","voter":true}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"error","msg":"failed to notify peers","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"bootstrap: context deadline exceeded","level":"fatal","msg":"could not open cloud meta store","time":"2025-05-13T03:13:21-04:00"}