{'error': [{'message': 'leader not found'}]} when using LangChain, and Singularity instead of docker

Support
,
1

Description

When I was trying to populate the database with “db = WeaviateVectorStore.from_documents(docs, hf_emb, client=weaviate_client)”, I received “UnexpectedStatusCodeError: Collection may not exist.! Unexpected status code: 500, with response body: {‘error’: [{‘message’: ‘failed to execute query: leader not found’}]}.”.

Server Setup Information

I set up the weaviate instance with Singularity (since docker is not available in my case). I first pulled a built image, and then run the “singularity exec” command as follows:
singularity exec
–bind ~/weaviate_data:/var/lib/weaviate
–env “QUERY_DEFAULTS_LIMIT=25”
–env “AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true”
–env “PERSISTENCE_DATA_PATH=/var/lib/weaviate”
–env “ENABLE_API_BASED_MODULES=true”
–env “ENABLE_MODULES=text2vec-ollama,generative-ollama”
–env “CLUSTER_HOSTNAME=node1”
~/singularity/weaviate_1.27.6.sif
weaviate --host 0.0.0.0 --port 8095 --scheme http
.

  • Weaviate Server Version: 1.27.6
  • Deployment Method: Singularity
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: Python v4
2

hi @Steven !!

Welcome to our community :hugs:

I see you reproduced the docker env vars, so that shouldn’t happen :thinking:

Can you try setting RAFT_BOOTSTRAP_EXPECT to 1?

I don’t have any experience with singularity :frowning:

Let me know if this will unblock you, otherwise I can play around with it and try reproducing it myself.

Thanks!

3

Hi @DudaNogueira ,

thanks for replying! Unfortunately the issue’s still there:(. Here’s some extra information, maybe that’ll help:

client.is_connected() and client.is_live() are both True, but client.ready() is false.

Here’s the server’s log after starting:
[
{
“build_git_commit”: “9069628”,
“build_go_version”: “go1.22.10”,
“build_image_tag”: “v1.28.0”,
“build_wv_version”: “1.28.0”,
“level”: “info”,
“msg”: “attempting to join”,
“remoteNodes”: [
“10.5.167.151:8300”
],
“time”: “2024-12-17T02:09:49+01:00”
},
{
“build_git_commit”: “9069628”,
“build_go_version”: “go1.22.10”,
“build_image_tag”: “v1.28.0”,
“build_wv_version”: “1.28.0”,
“level”: “info”,
“msg”: “attempted to join and failed”,
“remoteNode”: “10.5.167.151:8300”,
“status”: 14,
“time”: “2024-12-17T02:09:49+01:00”
},
{
“action”: “bootstrap”,
“build_git_commit”: “9069628”,
“build_go_version”: “go1.22.10”,
“build_image_tag”: “v1.28.0”,
“build_wv_version”: “1.28.0”,
“error”: “could not join a cluster from [10.5.167.151:8300]”,
“level”: “warning”,
“msg”: “failed to join cluster”,
“servers”: [
“10.5.167.151:8300”
],
“time”: “2024-12-17T02:09:49+01:00”,
“voter”: true
},
{
“action”: “bootstrap”,
“build_git_commit”: “9069628”,
“build_go_version”: “go1.22.10”,
“build_image_tag”: “v1.28.0”,
“build_wv_version”: “1.28.0”,
“error”: “rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: failed to do connect handshake, response: \"HTTP/1.1 403 Forbidden…”,
“level”: “error”,
“msg”: “failed to notify peers”,
“servers”: [
“10.5.167.151:8300”
],
“time”: “2024-12-17T02:09:49+01:00”
}
]

Thanks again for helping!

4

hi @Steven !!

I will have to save time play around with singularity so I can try figuring it out (never used)

What is the singularity version you are using?