Local client connection refused issue

jimmy_6 · May 28, 2024, 5:54am

Hi, I have the problem about local client connection refused error.
It worked fine, but the connection is refused during the process of changing vectorizer model.
Since then the connection has been refused.

I think maybe… the many connections are opend but not closed during the process.

I tried all solutions expressed in this community.
But I could not solve this.

And I checked the docker compose containers, their status is ‘restarting’ continuously…
So their ports are not connected.

I need some help.

The error logs are like this:

[root@vector-12 ~]# python
Python 3.9.19 (main, May 13 2024, 09:18:13)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type “help”, “copyright”, “credits” or “license” for more information.

import weaviate
c = weaviate.connect_to_local()
Traceback (most recent call last):
File “/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py”, line 69, in map_httpcore_exceptions
yield
File “/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py”, line 233, in handle_request
resp = self._pool.handle_request(req)
File “/usr/local/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py”, line 216, in handle_request
raise exc from None
File “/usr/local/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py”, line 196, in handle_request
response = connection.handle_request(
File “/usr/local/lib/python3.9/site-packages/httpcore/_sync/connection.py”, line 99, in handle_request
raise exc
File “/usr/local/lib/python3.9/site-packages/httpcore/_sync/connection.py”, line 76, in handle_request
stream = self._connect(request)
File “/usr/local/lib/python3.9/site-packages/httpcore/_sync/connection.py”, line 122, in _connect
stream = self._network_backend.connect_tcp(**kwargs)
File “/usr/local/lib/python3.9/site-packages/httpcore/_backends/sync.py”, line 213, in connect_tcp
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
File “/usr/local/lib/python3.9/contextlib.py”, line 137, in exit
self.gen.throw(typ, value, traceback)
File “/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py”, line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/local/lib/python3.9/site-packages/weaviate/connect/v4.py”, line 250, in _create_clients
response = client.get(oidc_url)
File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 1054, in get
return self.request(
File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 827, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 914, in send
response = self._send_handling_auth(
File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 942, in _send_handling_auth
response = self._send_handling_redirects(
File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 979, in _send_handling_redirects
response = self._send_single_request(request)
File “/usr/local/lib/python3.9/site-packages/httpx/_client.py”, line 1015, in _send_single_request
response = transport.handle_request(request)
File “/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py”, line 233, in handle_request
resp = self._pool.handle_request(req)
File “/usr/local/lib/python3.9/contextlib.py”, line 137, in exit
self.gen.throw(typ, value, traceback)
File “/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py”, line 86, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “”, line 1, in
File “/usr/local/lib/python3.9/site-packages/weaviate/connect/helpers.py”, line 157, in connect_to_local
return __connect(client)
File “/usr/local/lib/python3.9/site-packages/weaviate/connect/helpers.py”, line 345, in __connect
raise e
File “/usr/local/lib/python3.9/site-packages/weaviate/connect/helpers.py”, line 341, in __connect
client.connect()
File “/usr/local/lib/python3.9/site-packages/weaviate/client.py”, line 287, in connect
self._connection.connect(self.__skip_init_checks)
File “/usr/local/lib/python3.9/site-packages/weaviate/connect/v4.py”, line 674, in connect
super().connect(skip_init_checks)
File “/usr/local/lib/python3.9/site-packages/weaviate/connect/v4.py”, line 137, in connect
self._create_clients(self._auth, skip_init_checks)
File “/usr/local/lib/python3.9/site-packages/weaviate/connect/v4.py”, line 252, in _create_clients
raise WeaviateConnectionError(
weaviate.exceptions.WeaviateConnectionError: Connection to Weaviate failed. Error: [Errno 111] Connection refused.
Is Weaviate running and reachable at http://localhost:8080?

I used the weaviate-client v4 / python version 3.9.19
I used the 3 weaviate containers with docker compose

I have to solve this problem in few days.
Anyone help me.
Thanks.

rjalex · May 28, 2024, 1:45pm

Hey, maybe if you would provide the code you’re using to connect, the versions of the server and client and the way you’re deploying the server we could be of more help.
Take care.

jimmy_6 · May 28, 2024, 2:12pm

I used these for connection

weaviate docker image: cr.weaviate.io/semitechnologies/weaviate:1.24.12
weaviate-client version: 4.6.1
python version: 3.9.19
docker-compose version: 2.27.0
OS: CentOS 7

and I did just this code

import weaviate

    client = weaviate.connect_to_local()
    print('done')
    client.close()

but the client can not be connected

my docker-compose file is like this:

version: ‘3.4’
services:
weaviate-node-1:
init: true
command:
- --host
- 0.0.0.0
- --port
- ‘8080’
- --scheme
- http
image: ~ weaviate:1.24.12
ports:
- 8080:8080
- 6060:6060
- 50051:50051
restart: on-failure:0
volumes:
- ./data-node-1:/var/lib/weaviate
environment:
LOG_LEVEL: ‘debug’
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: ‘true’
PERSISTENCE_DATA_PATH: ‘/var/lib/weaviate’
ENABLE_MODULES: ‘text2vec-transformers,text2vec-gpt4all,text2vec-contextionary’
DEFAULT_VECTORIZER_MODULE: ‘none’
CLUSTER_HOSTNAME: ‘node1’
CLUSTER_GOSSIP_BIND_PORT: ‘7100’
CLUSTER_DATA_BIND_PORT: ‘7101’

weaviate-node-2:
init: true
command:
- --host
- 0.0.0.0
- --port
- ‘8080’
- --scheme
- http
image: ~ weaviate:1.24.12
ports:
- 8081:8080
- 6061:6060
- 50052:50051
restart: on-failure:0
volumes:
- ./data-node-2:/var/lib/weaviate
environment:
LOG_LEVEL: ‘debug’
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: ‘true’
PERSISTENCE_DATA_PATH: ‘/var/lib/weaviate’
ENABLE_MODULES: ‘text2vec-transformers,text2vec-gpt4all,text2vec-contextionary’
DEFAULT_VECTORIZER_MODULE: ‘none’
CLUSTER_HOSTNAME: ‘node2’
CLUSTER_GOSSIP_BIND_PORT: ‘7102’
CLUSTER_DATA_BIND_PORT: ‘7103’
CLUSTER_JOIN: ‘weaviate-node-1:7100’

weaviate-node-3:
init: true
command:
- --host
- 0.0.0.0
- --port
- ‘8080’
- --scheme
- http
image: ~ weaviate:1.24.12
ports:
- 8082:8080
- 6062:6060
- 50053:50051
restart: on-failure:0
volumes:
- ./data-node-3:/var/lib/weaviate
environment:
LOG_LEVEL: ‘debug’
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: ‘true’
PERSISTENCE_DATA_PATH: ‘/var/lib/weaviate’
ENABLE_MODULES: ‘text2vec-transformers,text2vec-gpt4all,text2vec-contextionary’
DEFAULT_VECTORIZER_MODULE: ‘none’
CLUSTER_HOSTNAME: ‘node3’
CLUSTER_GOSSIP_BIND_PORT: ‘7104’
CLUSTER_DATA_BIND_PORT: ‘7105’
CLUSTER_JOIN: ‘weaviate-node-1:7100’
I changed the image line like ‘image: ~ weaviate:1.24.12’ for saving the reply (Don’t care about it)

rjalex · May 28, 2024, 2:42pm

Where does that code run? on your host or within a docker container?

first of all please note that connect_to_local() will use default parameters. Here is the doc: Weaviate Packages — Weaviate Python Client 4.6.4.dev0+g3655347.d20240520 documentation

I personally prefer everything explicit such as in:

import weaviate
from weaviate.connect import ConnectionParams
from weaviate.classes.init import AdditionalConfig, Timeout
import os

client = weaviate.WeaviateClient(
    connection_params=ConnectionParams.from_params(
        http_host="localhost",
        http_port="8099",
        http_secure=False,
        grpc_host="localhost",
        grpc_port="50052",
        grpc_secure=False,
    ),
    auth_client_secret=weaviate.auth.AuthApiKey("secr3tk3y"),
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_APIKEY")
    },
    additional_config=AdditionalConfig(
        timeout=Timeout(init=2, query=45, insert=120),  # Values in seconds
    ),
)

client.connect()

Of course please notice that if the code is running on the host localhost is correct but if it is running in a container ran by docker compose the hostname will be the service name you have set in docker compose : weaviate-node-1 or weaviate-node-2

HTH

jimmy_6 · May 28, 2024, 11:50pm

I ran without the docker container

I tried to connect client with your code, but it isn’t worked

the container status is restarting continuously, the ports are not connected

rjalex · May 29, 2024, 6:22am

As you can maybe see from docker ps, the containers are not exposing ports on the host since they’re restarting. Am I right?

Once you get the ports exposed correctly make sure they are aligned with the code I’ve shown you.

FWIW here is my docker compose service declaration:

weaviate:
    image: cr.weaviate.io/semitechnologies/weaviate:1.25.1
    command:
      - "--host=0.0.0.0"
      - "--port=8080"
      - "--scheme=http"
    ports:
      - "8080:8080"
      - "50051:50051"
    volumes:
      - weaviate_data:/var/lib/weaviate
    restart: unless-stopped

Take care.

Dirk · May 29, 2024, 6:27am

That is the problem - you need to check the container logs why they are restarting

jimmy_6 · May 29, 2024, 11:43am

Yes, I will try like that

And it is worked well with docker image - weaviate version: 1.25.1

Thanks for your help

jimmy_6 · May 29, 2024, 11:46am

I will try to check the logs one by one

And it is worked well with docker image - weaviate version: 1.25.1

Thanks for your help

jimmy_6 · June 10, 2024, 12:58am

The problem is occurred again…

I check the logs,
and the logs are saying that

“action”:“startup”,“error”:“init bolt_db: open "/var/lib/weaviate/schema.db": open /var/lib/weaviate/schema.db: permission denied”,“level”:“fatal”,“msg”:“could not initialize schema repo”

I tried to remove the folder /var/lib/weaviate in the container
but it is not done

How can I solve this problem?

I tried many ways for connecting the weaviate container

Dirk · June 10, 2024, 6:13am

Looks like you don’t have permissions to read the data - maybe change the data folder to a location where you have access to, such as /home/your_user/weaviate/data/

jimmy_6 · June 10, 2024, 12:37pm

That folder is in weaviate container, it is not in my server

that folder and the “./weaviate_data” folder in my server are shared the data

and I tried to remove the folder in the container,
but it is failed

Topic		Replies	Views
Weaviate.connect_to_local() error Support	1	605	June 10, 2024
Dial tcp 127.0.0.1:11434: connect: connection refused (Local Ollama) Support bug , python	3	2407	May 29, 2024
Local client connection with localhost:8080 failed Support developer-experience	8	2752	May 29, 2024
Using sentence_transformers together with Weaviate Support bug , python	5	479	July 24, 2024
Vectors are not loading into the local DB using Python Client V4 Support	2	1080	November 17, 2023

Local client connection refused issue

Related topics