[Question] Python - weaviate client 4.0 cannot connect from inside kubernetes

Support
1

I have a behavior that I can’t explain:
I have a docker image - python:3.11 weaviate client 4.7.1

code is super simple:

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=WV_URL,  # Replace with your Weaviate Cloud URL
    auth_credentials=Auth.api_key(WV_APIKEY), 
)```

and the best part - the same docker image when run locally connects fine, but run from other environment - inside kubernetes on amazon fails with:

Traceback (most recent call last):

File "/app/server.py", line 21, in <module>

client = weaviate.connect_to_weaviate_cloud(

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/weaviate/connect/helpers.py", line 79, in connect_to_weaviate_cloud

return __connect(

^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/weaviate/connect/helpers.py", line 410, in __connect

raise e

File "/usr/local/lib/python3.11/site-packages/weaviate/connect/helpers.py", line 406, in __connect

client.connect()

File "/usr/local/lib/python3.11/site-packages/weaviate/syncify.py", line 23, in sync_method

return _EventLoopSingleton.get_instance().run_until_complete(

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/weaviate/event_loop.py", line 40, in run_until_complete

return fut.result()

^^^^^^^^^^^^

File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 456, in result

return self.__get_result()

^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result

raise self._exception

File "/usr/local/lib/python3.11/site-packages/weaviate/client_base.py", line 152, in connect

await self._connection.connect(self._skip_init_checks)

File "/usr/local/lib/python3.11/site-packages/weaviate/connect/v4.py", line 146, in connect

await self._open_connections(self._auth, skip_init_checks)

File "/usr/local/lib/python3.11/site-packages/weaviate/connect/v4.py", line 239, in _open_connections

self.__make_clients()

File "/usr/local/lib/python3.11/site-packages/weaviate/connect/v4.py", line 228, in __make_clients

self._client = self.__make_async_client()

^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/weaviate/connect/v4.py", line 222, in __make_async_client

return AsyncClient(

^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1389, in __init__

super().__init__(

File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 183, in __init__

self.headers = Headers(headers)

^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 72, in __init__

self._list = [

^

File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 76, in <listcomp>

normalize_header_value(v, encoding),

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/httpx/_utils.py", line 53, in normalize_header_value

return value.encode(encoding or "ascii")


Probably environment error, but is there a way to debug it ?
2

hi @marekzebrowski !!

Welcome to our community :hugs:

the connect_to_weaviate_cloud should only be used when connecting to our cloud.

And thats because we have a specific way of exposing our weaviate instances, as you can see here.

If you expose your cluster exactly like we do in our cloud, you could also use that method too.

So client.connect_to* methods will “configure” how the client connects to the server.

With that said, you will want to take a look at connect_to_custom, and provide all the hosts/ports/security you have used to properly expose your Weaviate cluster.

Check here for more on connect_to_custom

for example, this is the equivalent to connect_to_local

import weaviate

client = weaviate.connect_to_custom(
    http_host="localhost",
    http_port="8080",
    http_secure=False,
    grpc_host="localhost",
    grpc_port="50051",
    grpc_secure=False,
    headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_APIKEY")  # Or any other inference API keys
    }
)

Let me know if this helps :slight_smile:

Thanks!

3

but… I am trying to connect to weaviate cloud - in both cases. wqpzmrins6y4jyhqg4pvcw.c0.europe-west3.gcp.weaviate.cloud to be specific.

I don’t expect anything special in the client - I suspect networking inside k8s to be tainted. I just wonder if there is a way to debug such problem - like debug log or something similar.

4

Oh @marekzebrowski !! I am really sorry.

I though you were running Weaviate on Kubernetes :see_no_evil:

Weaviate will provide basically two endpoints, HTTP and GRPC.

for testing http a simple curl to the endpoint will be enough.

For GRPC, you can use a tool called grpcurl.

on this thread there is an explanation on how to do that:

Let me know if this heps.

5

@marekzebrowski Is this issue resolved? I am also facing similar issue.

6

YES. It turned out to be bad decoding of AWS secret, that mangled API key.
in AWS use none instead of auto - that helped me.

7

do you mean weaviate api key was incorrect that was read from AWS secret? I did not follow you completely.

8

Exactly. Root cause of my problem was decoding of AWS secret. In that secret I stored Weaviate api key.

1 Like
9

Thanks for sharing, @marekzebrowski !!!