Once weaviate.connect_to_embedded , do I need to start it before insert data into collection?

Support
1

I tried local connection with data inserted successfully. When I switch from connect_to_local to connect_to_embeded, data can’t insert .

I saw error on the path now. anyway to suppress those messages to only show error? I tried to set log level which seems not working

INFO:weaviate-client:Started /Users/connie.wang/.cache/weaviate-embedded: process ID 33040
{“action”:“startup”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“default_vectorizer_module”:“none”,“level”:“info”,“msg”:“the default vectorizer modules is set to "none", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer”,“time”:“2024-11-01T17:24:14-05:00”}
{“action”:“startup”,“auto_schema_enabled”:true,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“auto schema enabled setting is set to "true"”,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true”,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“module offload-s3 is enabled”,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“warning”,“msg”:“Multiple vector spaces are present, GraphQL Explore and REST API list objects endpoint module include params has been disabled as a result.”,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“open cluster service”,“servers”:{“Embedded_at_8079”:62794},“time”:“2024-11-01T17:24:14-05:00”}
{“address”:“192.168.1.44:62795”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“starting cloud rpc server …”,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“starting raft sub-system …”,“time”:“2024-11-01T17:24:14-05:00”}
{“address”:“192.168.1.44:62794”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“tcp transport”,“tcpMaxPool”:3,“tcpTimeout”:10000000000,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“loading local db”,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“local DB successfully loaded”,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“schema manager loaded”,“n”:0,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“metadata_only_voters”:false,“msg”:“construct a new raft node”,“name”:“Embedded_at_8079”,“time”:“2024-11-01T17:24:14-05:00”}
{“action”:“raft”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“index”:98,“level”:“info”,“msg”:“raft initial configuration”,“servers”:“[[{Suffrage:Voter ID:Embedded_at_8079 Address:192.168.1.44:62432}]]”,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“last_snapshot_index”:0,“last_store_applied_index_on_start”:72,“level”:“info”,“msg”:“raft node constructed”,“raft_applied_index”:0,“raft_last_index”:98,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“hasState”:true,“level”:“info”,“msg”:“raft init”,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:[“192.168.1.44:62794”],“time”:“2024-11-01T17:24:14-05:00”}
{“action”:“raft”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“follower”:{},“leader-address”:“”,“leader-id”:“”,“level”:“info”,“msg”:“raft entering follower state”,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“192.168.1.44:62794”,“status”:8,“time”:“2024-11-01T17:24:14-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:[“192.168.1.44:62794”],“time”:“2024-11-01T17:24:15-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“attempted to join and failed”,“remoteNode”:“192.168.1.44:62794”,“status”:8,“time”:“2024-11-01T17:24:15-05:00”}
{“action”:“raft”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“last-leader-addr”:“”,“last-leader-id”:“”,“level”:“warning”,“msg”:“raft heartbeat timeout reached, starting election”,“time”:“2024-11-01T17:24:15-05:00”}
{“action”:“raft”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“raft entering candidate state”,“node”:{},“term”:48,“time”:“2024-11-01T17:24:15-05:00”}
{“action”:“raft”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“raft pre-vote successful, starting election”,“refused”:0,“tally”:1,“term”:48,“time”:“2024-11-01T17:24:15-05:00”,“votesNeeded”:1}
{“action”:“raft”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“raft election won”,“tally”:1,“term”:48,“time”:“2024-11-01T17:24:15-05:00”}
{“action”:“raft”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“leader”:{},“level”:“info”,“msg”:“raft entering leader state”,“time”:“2024-11-01T17:24:15-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“reload local db: update schema …”,“time”:“2024-11-01T17:24:15-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“index”:“PDF_COLLECTION”,“level”:“info”,“msg”:“reload local index”,“time”:“2024-11-01T17:24:15-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“docker_image_tag”:“localhost”,“level”:“info”,“msg”:“configured versions”,“server_version”:“1.26.6”,“time”:“2024-11-01T17:24:16-05:00”}
{“action”:“grpc_startup”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“grpc server listening at [::]:50050”,“time”:“2024-11-01T17:24:16-05:00”}
{“address”:“192.168.1.44:62794”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“current Leader”,“time”:“2024-11-01T17:24:16-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“attempting to join”,“remoteNodes”:[“192.168.1.44:62794”],“time”:“2024-11-01T17:24:16-05:00”}
{“action”:“raft”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“command”:0,“level”:“info”,“msg”:“raft updating configuration”,“server-addr”:“192.168.1.44:62794”,“server-id”:“Embedded_at_8079”,“servers”:“[[{Suffrage:Voter ID:Embedded_at_8079 Address:192.168.1.44:62794}]]”,“time”:“2024-11-01T17:24:16-05:00”}
{“action”:“restapi_management”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“docker_image_tag”:“localhost”,“level”:“info”,“msg”:“Serving weaviate at http://127.0.0.1:8079”,“time”:“2024-11-01T17:24:16-05:00”}
Error from /Users/connie.wang/Desktop/connie/inspiration_azure/fastapi_onazure/app/with_weaviate/app/rag/data/constitution.pdf: Invalid PDF file path: /Users/connie.wang/Desktop/connie/inspiration_azure/fastapi_onazure/app/with_weaviate/app/rag/data/constitution.pdf
Error: Invalid PDF file path: /Users/connie.wang/Desktop/connie/inspiration_azure/fastapi_onazure/app/with_weaviate/app/rag/data/constitution.pdf
{“action”:“restapi_management”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“docker_image_tag”:“localhost”,“level”:“info”,“msg”:“Shutting down… “,“time”:“2024-11-01T17:24:16-05:00”}
{“action”:“restapi_management”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“docker_image_tag”:“localhost”,“level”:“info”,“msg”:“Stopped serving weaviate at http://127.0.0.1:8079”,“time”:“2024-11-01T17:24:16-05:00”}
{“action”:“hnsw_prefill_cache_async”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“not waiting for vector cache prefill, running in background”,“time”:“2024-11-01T17:24:17-05:00”,“wait_for_cache_prefill”:false}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“Completed loading shard pdf_collection_uo8EdlUehjpj in 11.185209ms”,“time”:“2024-11-01T17:24:17-05:00”}
{“action”:“hnsw_compressed_vector_cache_prefill”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“count”:0,“level”:“info”,“msg”:“prefilled compressed vector cache”,“time”:“2024-11-01T17:24:17-05:00”,“took”:399084}
{“action”:“telemetry_push”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“telemetry started”,“payload”:”\u0026{MachineID:3eedf617-c28f-47d7-adf0-449c23f70447 Type:INIT Version:1.26.6 NumObjects:0 OS:darwin Arch:arm64 UsedModules:[generative-cohere text2vec-openai]}”,“time”:“2024-11-01T17:24:17-05:00”}
{“action”:“telemetry_push”,“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“telemetry terminated”,“payload”:“\u0026{MachineID:3eedf617-c28f-47d7-adf0-449c23f70447 Type:TERMINATE Version:1.26.6 NumObjects:0 OS:darwin Arch:arm64 UsedModules:[generative-cohere text2vec-openai]}”,“time”:“2024-11-01T17:24:17-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“closing raft FSM store …”,“time”:“2024-11-01T17:24:17-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“shutting down raft sub-system …”,“time”:“2024-11-01T17:24:17-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“transferring leadership to another server”,“time”:“2024-11-01T17:24:17-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“error”:“cannot find peer”,“level”:“error”,“msg”:“transferring leadership”,“time”:“2024-11-01T17:24:17-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“closing raft-net …”,“time”:“2024-11-01T17:24:17-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“closing log store …”,“time”:“2024-11-01T17:24:17-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“closing data store …”,“time”:“2024-11-01T17:24:17-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“closing loaded database …”,“time”:“2024-11-01T17:24:17-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“closing raft-rpc client …”,“time”:“2024-11-01T17:24:17-05:00”}
{“build_git_commit”:“ab0312d5d”,“build_go_version”:“go1.23.1”,“build_image_tag”:“localhost”,“build_wv_version”:“1.26.6”,“level”:“info”,“msg”:“closing raft-rpc server …”,“time”:“2024-11-01T17:24:17-05:00”}

2

Hi!

Once that you get a valid client, with a connected instance behind it, you should be able to insert your objects.

You can check that with client.is_ready()

Please note that Weaviate Embedded is marked as experimental.

Also, that when you run Weaviate Embedded, and close the client, the server will be stopped.

If you do not close the server, it may still be running on background, so if you spin a second embedded, you may have problems with port bindings.

3

yah. i made the connection work yesterday, and observed the issue with connection as you planned already. Will data be persist after client is disconnected?

4

Hi!

Yes, the data will persist.

If you do not specify a persistence_path, as describre here, by default, it will store your data in ~/.local/share/weaviate

If you have issues running the embedded client, because there is another process binding to that port, you will need to kill the process.

So:

ps aux | grep weaviate

then kill the ID of the running process.

Let me know if that helps!