Help running local instance of weaviate - server keeps exiting with "fatal" "could not open cloud meta store"

demitri · May 13, 2025, 2:56pm

Description

I am trying to run a local instance of Weaviate, but when I start the server it quits many minutes later. I am running on macOS 15.4.1.

I built the server with:

git clone https://github.com/weaviate/weaviate.git
cd weaviate
go build -o weaviate ./cmd/weaviate-server/main.go

and start the server with:

DISABLE_TELEMETRY=true \
CLUSTER_ENABLED=false \
DISK_USE_READONLY_PERCENTAGE=95 \
DISK_USE_WARNING_PERCENTAGE=95
PERSISTENCE_DATA_PATH=./weaviate_data \
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true \
DEFAULT_VECTORIZER_MODULE=none \
ENABLE_MODULES=none \
QUERY_DEFAULTS_LIMIT=50 \
RAFT_ENABLE_ONE_NODE_RECOVERY=true \
LOG_LEVEL=debug \
nohup weaviate --host=localhost --port 38080 --scheme=http > weaviate.log 2>&1 &

I found a GitHub issue #5362 that describes the same problem; it was suggested to use RAFT_ENABLE_ONE_NODE_RECOVERY=true to address this.

The server starts and runs for a few minutes, then quits (full log below).

I also note the DISK_USE... variables don’t appear to change things as noted in the log.

Server Setup Information

Weaviate Server Version: 1.31.0-rc.0
Deployment Method: binary
Multi Node? Number of Running Nodes: one node
Client Language and Version: can’t get the server to stay up yet…
Multitenancy?:

Any additional Information

This is the full output log:

{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"created startup context, nothing done so far","startup_time_left":"59m59.998680875s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"Feature flag LD integration disabled: could not locate WEAVIATE_LD_API_KEY env variable","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","auto_schema_enabled":{},"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"auto schema enabled setting is set to \"\u0026{\u003cnil\u003e {{{} {0 0}} 0 0 {{} 0} {{} 0}} true}\"","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"config loaded","startup_time_left":"59m59.997037833s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"configured OIDC and anonymous access client","startup_time_left":"59m59.996798458s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"initialized schema","startup_time_left":"59m59.996789083s","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"startup routine complete","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"start registering modules","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"completed registering modules","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"module offload-s3 is not enabled","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"module offload-s3 is enabled","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","flag_key":"collection-retrieval-strategy","level":"info","msg":"feature flag instantiated","time":"2025-05-13T03:02:53-04:00","tool":"feature_flag","value":"LeaderOnly"}
{"action":"cluster_api_startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"serving cluster api on port 7947","port":7947,"time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"start initializing modules","time":"2025-05-13T03:02:53-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"finished initializing modules","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"open cluster service","servers":{"Mac-Studio.local":8300},"time":"2025-05-13T03:02:53-04:00"}
{"address":"10.128.42.220:8301","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"starting cloud rpc server ...","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"starting raft sub-system ...","time":"2025-05-13T03:02:53-04:00"}
{"address":"10.128.42.220:8300","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"tcp transport","tcpMaxPool":3,"tcpTimeout":10000000000,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"loading local db","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"local DB successfully loaded","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"schema manager loaded","n":0,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","metadata_only_voters":false,"msg":"construct a new raft node","name":"Mac-Studio.local","time":"2025-05-13T03:02:53-04:00"}
{"action":"raft","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","index":0,"level":"info","msg":"initial configuration","servers":"[[]]","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","candidates":{},"level":"warning","msg":"didn't perform cluster recovery","servers_from_previous_configuration":null,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","last_snapshot_index":0,"last_store_applied_index_on_start":0,"level":"info","msg":"raft node constructed","raft_applied_index":0,"raft_last_index":0,"time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","hasState":false,"level":"info","msg":"raft init","time":"2025-05-13T03:02:53-04:00"}
{"action":"raft","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","follower":{},"leader-address":"","leader-id":"","level":"info","msg":"entering follower state","time":"2025-05-13T03:02:53-04:00"}
{"action":"read_disk_use","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"disk usage currently at 87.70%, threshold set to 80.00%","path":"./weaviate_data","time":"2025-05-13T03:02:53-04:00"}
{"action":"disk_use_stats","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"total: 926.35GB, free: 113.93GB, used: 812.42GB (avail: 113.93GB)","path":"./weaviate_data","time":"2025-05-13T03:02:53-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempting to join","remoteNodes":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:02:54-04:00"}
{"action":"raft","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"no known peers, aborting election","time":"2025-05-13T03:02:55-04:00"}
{"action":"inverted filter2search migration","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"migration skip flag set, skipping migration","time":"2025-05-13T03:02:55-04:00"}
{"action":"inverted filter2search migration","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"starting switching fallback mode","time":"2025-05-13T03:02:55-04:00"}
{"action":"inverted filter2search migration","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"no missing filterable indexes, fallback mode skipped","time":"2025-05-13T03:02:55-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"configured versions","server_version":"1.31.0-rc.0","time":"2025-05-13T03:02:55-04:00","version":"1.31.0-rc.0"}
{"action":"grpc_startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"grpc server listening at [::]:50051","time":"2025-05-13T03:02:55-04:00"}
{"action":"restapi_management","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"Serving weaviate at http://127.0.0.1:38080","time":"2025-05-13T03:02:55-04:00","version":"1.31.0-rc.0"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"error","msg":"telemetry failed to start: push: failed to send request: Post \"https://telemetry.weaviate.io/weaviate-telemetry\": dial tcp 34.149.252.24:443: connect: bad file descriptor","time":"2025-05-13T03:02:55-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempted to join and failed","remoteNode":"10.128.42.220:8300","status":14,"time":"2025-05-13T03:03:14-04:00"}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:03:14-04:00","voter":true}
{"action":"read_disk_use","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"disk usage currently at 87.70%, threshold set to 80.00%","path":"./weaviate_data","time":"2025-05-13T03:03:24-04:00"}
{"action":"disk_use_stats","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"total: 926.35GB, free: 113.92GB, used: 812.43GB (avail: 113.92GB)","path":"./weaviate_data","time":"2025-05-13T03:03:24-04:00"}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.128.42.220:8301: i/o timeout\"","level":"error","msg":"failed to notify peers","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:03:34-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempting to join","remoteNodes":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:03:34-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempted to join and failed","remoteNode":"10.128.42.220:8300","status":14,"time":"2025-05-13T03:03:34-04:00"}

[repeated "failed to notify peers" messages]

{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:05:14-04:00","voter":true}
{"action":"read_disk_use","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"warning","msg":"disk usage currently at 87.70%, threshold set to 80.00%","path":"./weaviate_data","time":"2025-05-13T03:05:24-04:00"}
{"action":"disk_use_stats","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"debug","msg":"total: 926.35GB, free: 113.93GB, used: 812.42GB (avail: 113.93GB)","path":"./weaviate_data","time":"2025-05-13T03:05:24-04:00"}

[repeated "failed to notify peers" messages]

{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:12:34-04:00","voter":true}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"rpc error: code = DeadlineExceeded desc = received context error while waiting for new LB policy update: context deadline exceeded","level":"error","msg":"failed to notify peers","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempting to join","remoteNodes":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00"}
{"build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","level":"info","msg":"attempted to join and failed","remoteNode":"10.128.42.220:8300","status":4,"time":"2025-05-13T03:13:21-04:00"}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"could not join a cluster from map[Mac-Studio.local:10.128.42.220:8300]","level":"warning","msg":"failed to join cluster","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00","voter":true}
{"action":"bootstrap","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"error","msg":"failed to notify peers","servers":{"Mac-Studio.local":"10.128.42.220:8300"},"time":"2025-05-13T03:13:21-04:00"}
{"action":"startup","build_git_commit":"","build_go_version":"go1.24.3","build_image_tag":"","build_wv_version":"1.31.0-rc.0","error":"bootstrap: context deadline exceeded","level":"fatal","msg":"could not open cloud meta store","time":"2025-05-13T03:13:21-04:00"}

DudaNogueira · May 13, 2025, 7:10pm

hi @demitri !!

Welcome to our community

Do you want to develop Weaviate itself, or develop something on top of Weaviate?

The best way use Weaviate locally, is using docker or k8s.

If you want to run from source, I suggest this resource:

github.com/weaviate/weaviate

CONTRIBUTING.md

main

## Thanks for looking into contributing to Weaviate!
We really appreciate that you are willing to spend some time and effort to make Weaviate better for everyone! 

We have a detailed [contributor guide](https://weaviate.io/developers/contributor-guide/current/) as a part of our documentation. If you are new, we recommend reading through the [getting started guide for contributors](https://weaviate.io/developers/contributor-guide/current/getting-started/index.html) after reading this overview.

## Finding a good issue to get started
We use the `good-first-issue` labels on issues that we think are great to get started. Typically, they are issues that are isolated to a specific area of Weaviate, e.g. only the API layer or only a specific package.

## What do you need to contribute
To contribute to Weaviate Core you should have at least basic Golang skills. It is also tremendously helpful if you have successfully used Weaviate before, so that you have a rough understanding of the effects of changes etc. Any PR must include tests. We understand that not everyone is very familiar with testing strategies, and how we have setup our tests. This [guide on testing](https://weaviate.io/developers/contributor-guide/current/weaviate-core/tests.html) should be a great start. If you have completed a change, but are struggling with the test, you can also open a Draft PR and ask someone from our team for help. 

## If in doubt, ask.
The Weaviate team consists of some of the nicest people on this planet. If something is unclear or you'd like a second opinion, please don't hesitate to ask. We are glad that you want to help us, so naturally we will also do our best to help you on this journey

## Development Environment
*Please note that the Weaviate team uses Linux and Mac (darwin/arm64) machines exclusively. Development on Windows may lead to unexpected issues.*

Here are some guides to get started in no time:
* [Development Setup](https://weaviate.io/developers/contributor-guide/current/weaviate-core/setup.html)
* [Code Structure](https://weaviate.io/developers/contributor-guide/current/weaviate-core/structure.html)

This file has been truncated. show original

Let me know if this helps!

Thanks!

Topic		Replies	Views
Fatal error during Weaviate 1.25 startup: could not open cloud meta store Support	10	422	July 16, 2024
Embedded server starts and shuts down right away Support	5	528	May 31, 2024
Weaviate Shutting Down Automatically Support developer-experience	5	248	September 30, 2024
Weaviate Deploying Issue Support bug , integration	4	849	September 26, 2023
Issue with Weaviate Container on Start-up Support integration , developer-experience , technical	4	426	September 27, 2024

Help running local instance of weaviate - server keeps exiting with "fatal" "could not open cloud meta store"

Description

Server Setup Information

Any additional Information

Related topics