Help running local instance of weaviate - server keeps exiting with "fatal" "could not open cloud meta store"

Description

I am trying to run a local instance of Weaviate, but when I start the server it quits many minutes later. I am running on macOS 15.4.1.

I built the server with:

git clone https://github.com/weaviate/weaviate.git
cd weaviate
go build -o weaviate ./cmd/weaviate-server/main.go

and start the server with:

DISABLE_TELEMETRY=true \
CLUSTER_ENABLED=false \
DISK_USE_READONLY_PERCENTAGE=95 \
DISK_USE_WARNING_PERCENTAGE=95
PERSISTENCE_DATA_PATH=./weaviate_data \
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true \
DEFAULT_VECTORIZER_MODULE=none \
ENABLE_MODULES=none \
QUERY_DEFAULTS_LIMIT=50 \
RAFT_ENABLE_ONE_NODE_RECOVERY=true \
LOG_LEVEL=debug \
nohup weaviate --host=localhost --port 38080 --scheme=http > weaviate.log 2>&1 &

I found a GitHub issue #5362 that describes the same problem; it was suggested to use RAFT_ENABLE_ONE_NODE_RECOVERY=true to address this.

The server starts and runs for a few minutes, then quits (full log below).

I also note the DISK_USE... variables don’t appear to change things as noted in the log.

Server Setup Information

  • Weaviate Server Version: 1.31.0-rc.0
  • Deployment Method: binary
  • Multi Node? Number of Running Nodes: one node
  • Client Language and Version: can’t get the server to stay up yet…
  • Multitenancy?:

Any additional Information

This is the full output log:

{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"created startup context, nothing done so far","startup_time_left":"59m59.998680875s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"Feature flag LD integration disabled: could not locate WEAVIATE_LD_API_KEY env variable","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","auto_schema_enabled":{},"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"auto schema enabled setting is set to \"\u0026{\u003cnil\u003e {{{} {0 0}} 0 0 {{} 0} {{} 0}} true}\"","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"config loaded","startup_time_left":"59m59.997037833s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"configured OIDC and anonymous access client","startup_time_left":"59m59.996798458s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"initialized schema","startup_time_left":"59m59.996789083s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"startup routine complete","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"start registering modules","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"completed registering modules","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"module offload-s3 is not enabled","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"module offload-s3 is enabled","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","flag_key":"collection-retrieval-strategy","level":"info","msg":"feature flag instantiated","time":"2025-05-13T03:02:53-04:00","tool":"feature_flag","value":"LeaderOnly"}
{"action":"cluster_api_startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"serving cluster api on port 7947","port":7947,"time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"start initializing modules","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"finished initializing modules","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"open cluster service","servers":{"Mac-Studio.local":8300},"time":"2025-05-13T03:02:53-04:00"}
{"address":"10.128.42.220:8301","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"starting cloud rpc server ...","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"starting raft sub-system ...","time":"2025-05-13T03:02:53-04:00"}
{"address":"10.128.42.220:8300","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"tcp transport","tcpMaxPool":3,"tcpTimeout":10000000000,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"loading local db","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"local DB successfully loaded","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"schema manager loaded","n":0,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","metadata_only_voters":false,"msg":"construct a new raft node","name":"Mac-Studio.local","time":"2025-05-13T03:02:53-04:00"}
{"action":"raft","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","index":0,"level":"info","msg":"initial configuration","servers":"[[]]","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","candidates":{},"level":"warning","msg":"didn't perform cluster recovery","servers_from_previous_configuration":null,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","last_snapshot_index":0,"last_store_applied_index_on_start":0,"level":"info","msg":"raft node constructed","raft_applied_index":0,"raft_last_index":0,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","hasState":false,"level":"info","msg":"raft init","time":"2025-05-13T03:02:53-04:00"}
{"action":"raft","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","follower":{},"leader-address":"","leader-id":"","level":"info","msg":"entering follower state","time":"2025-05-13T03:02:53-04:00"}
{"action":"read_disk_use","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"disk usage currently at 87.70%, threshold set to 80.00%","path":"./weaviate_data","time":"2025-05-13T03:02:53-04:00"}
{"action":"disk_use_stats","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"total: 926.35GB, free: 113.93GB, used: 812.42GB (avail: 113.93GB)","path":"./weaviate_data","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempting to join","remoteNodes":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:02:54-04:00"}
{"action":"raft","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"no known peers, aborting election","time":"2025-05-13T03:02:55-04:00"}
{"action":"inverted filter2search migration","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"migration skip flag set, skipping migration","time":"2025-05-13T03:02:55-04:00"}
{"action":"inverted filter2search migration","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"starting switching fallback mode","time":"2025-05-13T03:02:55-04:00"}
{"action":"inverted filter2search migration","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"no missing filterable indexes, fallback mode skipped","time":"2025-05-13T03:02:55-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"configured versions","server_version":"1.31.0-rc.0","time":"2025-05-13T03:02:55-04:00","version":"1.31.0-rc.0"}
{"action":"grpc_startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"grpc server listening at [::]:50051","time":"2025-05-13T03:02:55-04:00"}
{"action":"restapi_management","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"Serving weaviate at http://127.0.0.1:38080","time":"2025-05-13T03:02:55-04:00","version":"1.31.0-rc.0"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"error","msg":"telemetry failed to start: push: failed to send request: Post \"https://telemetry.weaviate.io/weaviate-telemetry\": dial tcp 34.149.252.24:443: connect: bad file descriptor","time":"2025-05-13T03:02:55-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempted to join and failed","remoteNode":"10.128.42.220:8300","status":14,"time":"2025-05-13T03:03:14-04:00"}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:03:14-04:00","voter":true}
{"action":"read_disk_use","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"disk usage currently at 87.70%, threshold set to 80.00%","path":"./weaviate_data","time":"2025-05-13T03:03:24-04:00"}
{"action":"disk_use_stats","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"total: 926.35GB, free: 113.92GB, used: 812.43GB (avail: 113.92GB)","path":"./weaviate_data","time":"2025-05-13T03:03:24-04:00"}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.128.42.220:8301: i/o timeout\"","level":"error","msg":"failed to notify peers","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:03:34-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempting to join","remoteNodes":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:03:34-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempted to join and failed","remoteNode":"10.128.42.220:8300","status":14,"time":"2025-05-13T03:03:34-04:00"}

[repeated "failed to notify peers" messages]

{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:05:14-04:00","voter":true}
{"action":"read_disk_use","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"disk usage currently at 87.70%, threshold set to 80.00%","path":"./weaviate_data","time":"2025-05-13T03:05:24-04:00"}
{"action":"disk_use_stats","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"total: 926.35GB, free: 113.93GB, used: 812.42GB (avail: 113.93GB)","path":"./weaviate_data","time":"2025-05-13T03:05:24-04:00"}

[repeated "failed to notify peers" messages]

{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:12:34-04:00","voter":true}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"rpc error: code = DeadlineExceeded desc = received context error while waiting for new LB policy update: context deadline exceeded","level":"error","msg":"failed to notify peers","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempting to join","remoteNodes":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempted to join and failed","remoteNode":"10.128.42.220:8300","status":4,"time":"2025-05-13T03:13:21-04:00"}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00","voter":true}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"error","msg":"failed to notify peers","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"bootstrap: context deadline exceeded","level":"fatal","msg":"could not open cloud meta store","time":"2025-05-13T03:13:21-04:00"}

hi @demitri !!

Welcome to our community :hugs:

Do you want to develop Weaviate itself, or develop something on top of Weaviate?

The best way use Weaviate locally, is using docker or k8s.

If you want to run from source, I suggest this resource:

Let me know if this helps!

Thanks!