Gpt4all remote service not ready as per weaviate-0 pod after deploying text2vec-gpt4all module in EKS

General
1

I have weaviate installed and running in EKS using helm chart and values yaml. It works for us but as soon as I include text2vec-gpt4all module, the weaviate-0 pod starts showing readiness probe failed error. Though the gpt4all pod shows that it is running fine, still weaviate-0 pod is telling that gpt4all remote service not ready. PFB the respective log snippet for both gpt4all and weaviate pods:

Snippet of gpt4all-inference-7444f6f897-5k6sv pod log:

2024-01-19T08:15:04.794126679Z stderr F INFO:     Started server process [7]
2024-01-19T08:15:04.794205526Z stderr F INFO:     Waiting for application startup.
2024-01-19T08:15:04.794405023Z stderr F INFO:     Loading model...
2024-01-19T08:15:06.803604216Z stderr F INFO:     Running model on CPU
2024-01-19T08:15:06.892779786Z stderr F INFO:     Application startup complete.
2024-01-19T08:15:06.893167769Z stderr F INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
2024-01-19T08:17:03.113105431Z stdout F Found model file at  /root/.cache/gpt4all/ggml-all-MiniLM-L6-v2-f16.bin
2024-01-19T08:17:03.113126423Z stdout F Found model file at  /root/.cache/gpt4all/ggml-all-MiniLM-L6-v2-f16.bin
2024-01-19T08:17:03.113130134Z stdout F INFO:     100.120.18.147:34026 - "GET /.well-known/ready HTTP/1.1" 204 No Content
2024-01-19T08:17:03.11367654Z stdout F INFO:     100.120.18.147:34028 - "GET /.well-known/live HTTP/1.1" 204 No Content
2024-01-19T08:17:06.111440417Z stdout F INFO:     100.120.18.147:34048 - "GET /.well-known/ready HTTP/1.1" 204 No Content
2024-01-19T08:17:06.111894425Z stdout F INFO:     100.120.18.147:34050 - "GET /.well-known/live HTTP/1.1" 204 No Content
2024-01-19T08:17:09.110858487Z stdout F INFO:     100.120.18.147:34070 - "GET /.well-known/ready HTTP/1.1" 204 No Content
2024-01-19T08:17:09.111456862Z stdout F INFO:     100.120.18.147:34072 - "GET /.well-known/live HTTP/1.1" 204 No Content
2024-01-19T08:17:12.111241198Z stdout F INFO:     100.120.18.147:40498 - "GET /.well-known/live HTTP/1.1" 204 No Content
2024-01-19T08:17:12.111687135Z stdout F INFO:     100.120.18.147:40496 - "GET /.well-known/ready HTTP/1.1" 204 No Content
2024-01-19T08:17:15.112430121Z stdout F INFO:     100.120.18.147:40512 - "GET /.well-known/live HTTP/1.1" 204 No Content
2024-01-19T08:17:15.11247627Z stdout F INFO:     100.120.18.147:40528 - "GET /.well-known/ready HTTP/1.1" 204 No Content
2024-01-19T08:17:18.11145407Z stdout F INFO:     100.120.18.147:40538 - "GET /.well-known/ready HTTP/1.1" 204 No Content
2024-01-19T08:17:18.11200604Z stdout F INFO:     100.120.18.147:40536 - "GET /.well-known/live HTTP/1.1" 204 No Content
2024-01-19T08:17:21.111794363Z stdout F INFO:     100.120.18.147:45798 - "GET /.well-known/live HTTP/1.1" 204 No Content
2024-01-19T08:17:21.111840823Z stdout F INFO:     100.120.18.147:45806 - "GET /.well-known/ready HTTP/1.1" 204 No Content
2024-01-19T08:17:24.111347517Z stdout F INFO:     100.120.18.147:45810 - "GET /.well-known/ready HTTP/1.1" 204 No Content
2024-01-19T08:17:24.111903378Z stdout F INFO:     100.120.18.147:45812 - "GET /.well-known/live HTTP/1.1" 204 No Content
2024-01-19T08:17:27.111139278Z stdout F INFO:     100.120.18.147:45834 - "GET /.well-known/live HTTP/1.1" 204 No Content
2024-01-19T08:17:27.111587178Z stdout F INFO:     100.120.18.147:45832 - "GET /.well-known/ready HTTP/1.1" 204 No Content
2024-01-19T08:17:30.110984399Z stdout F INFO:     100.120.18.147:52060 - "GET /.well-known/live HTTP/1.1" 204 No Content
2024-01-19T08:17:30.11151872Z stdout F INFO:     100.120.18.147:52062 - "GET /.well-known/ready HTTP/1.1" 204 No Content
2024-01-19T08:17:33.1113498Z stdout F INFO:     100.120.18.147:52090 - "GET /.well-known/ready HTTP/1.1" 204 No Content
2024-01-19T08:17:33.111951544Z stdout F INFO:     100.120.18.147:52092 - "GET /.well-known/live HTTP/1.1" 204 No Content

Snippet of weaviate-0 pod log:

2024-01-19T12:20:06.109025384Z stderr F {"action":"config_load","config_file_path":"/weaviate-config/conf.yaml","level":"info","msg":"Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.","time":"2024-01-19T12:20:06Z"}
2024-01-19T12:20:06.109238558Z stderr F {"deprecation":{"apiType":"Configuration","id":"config-files","locations":["--config-file=\"\""],"mitigation":"Configure Weaviate using environment variables.","msg":"use of deprecated command line argument --config-file","sinceTime":"2020-09-08T09:46:00.000Z","sinceVersion":"0.22.16","status":"deprecated"},"level":"warning","msg":"use of deprecated command line argument --config-file","time":"2024-01-19T12:20:06Z"}
2024-01-19T12:20:06.109267257Z stderr F {"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-01-19T12:20:06Z"}
2024-01-19T12:20:06.109270093Z stderr F {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-01-19T12:20:06Z"}
2024-01-19T12:20:06.11979485Z stderr F {"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-01-19T12:20:06Z"}
2024-01-19T12:20:07.626770804Z stderr F {"action":"gpt4all_remote_wait_for_startup","error":"send check ready request: Get \http://gpt4all-inference.weaviate2.svc.cluster.local:8080/.well-known/ready\: context deadline exceeded","level":"warning","msg":"gpt4all remote service not ready","time":"2024-01-19T12:20:07Z"}
2024-01-19T12:20:08.626707145Z stderr F {"action":"gpt4all_remote_wait_for_startup","error":"send check ready request: Get \http://gpt4all-inference.weaviate2.svc.cluster.local:8080/.well-known/ready\: context deadline exceeded","level":"warning","msg":"gpt4all remote service not ready","time":"2024-01-19T12:20:08Z"}
2024-01-19T12:20:09.626922843Z stderr F {"action":"gpt4all_remote_wait_for_startup","error":"send check ready request: Get \http://gpt4all-inference.weaviate2.svc.cluster.local:8080/.well-known/ready\: context deadline exceeded","level":"warning","msg":"gpt4all remote service not ready","time":"2024-01-19T12:20:09Z"}
2024-01-19T12:20:10.626825487Z stderr F {"action":"gpt4all_remote_wait_for_startup","error":"send check ready request: Get \http://gpt4all-inference.weaviate2.svc.cluster.local:8080/.well-known/ready\: context deadline exceeded","level":"warning","msg":"gpt4all remote service not ready","time":"2024-01-19T12:20:10Z"}

Please let me know if you need any more information.

2

Hi @Graphic_Noir85 !

I was able to use this model on a K8s cluster. What version of Weaviate are you running?

I changed mine to 1.23.0 tag in values.yaml

To check if the vectorizer is properly working, you can forward the port locally:

kubectl port-forward gpt4all-inference-<replace_yours>-<replace_yours> 8080:8080

then perform a curl to vectorize something:

curl -X POST "http://localhost:8080/vectorize" -H "Content-Type: application/json" -d '{"text": "Your input text here"}'

Let me know if this helps.

3

Hi @DudaNogueira. Thank you for your reply. I am working with @Graphic_Noir85.
I tried your recommendation to test gpt4all. I can verify that gpt4all works when I use the above commands.
Output Snippet

 curl -X POST "http://localhost:8080/vectorize" -H "Content-Type: application/json" -d '{"text": "Your input text here"}' | jq
{
  "text": "Your input text here",
  "vector": [
    0.06790349632501602,
    -0.03317484259605408,
    -0.016778158023953438,

Still when we check the pods in the namespace, weaviate-0 is failling non stop.

NAME                                     READY   STATUS             RESTARTS        AGE
pod/gpt4all-inference-7444f6f897-5k6sv   1/1     Running            0               15h
pod/weaviate-0                           0/1     CrashLoopBackOff   132 (95s ago)   15h

When we look at the logs of pod/weaviate-0, we see the logs contain the following output.

"action":"gpt4all_remote_wait_for_startup",
"error":"send check ready request: Get \"http://gpt4all-inference.weaviate2.svc.cluster.local:8080/.well-known/ready\": context deadline exceeded",
"level":"warning",
"msg":"gpt4all remote service not ready","time":"2024-01-19T23:23:28Z"}

Do you have any other suggestions on how to troubleshoot?

4

Thanks for our answer @DudaNogueira. We are using weaviate 1.23.2

As @oss mentioned, we have tried your suggestion. but Weaviate-0 pod is still failing non-stop.

5

Thats weird.

So gpt4all is working properly, but seems like weaviate cannot communicate with it.

Maybe there is a network policy in place that is preventing this?

One thing to try is to execute curl from Weaviate to make sure you actually can query gpt4all from there.