Auth defaults to 'anonymous' when connecting from host vs container

I have weaviate:latest container (1.21.1) and cannot connect as admin user from the host. If I create another container I can connect (using the same api key).

It seems weaviate defaults to anonymous if connecting from one network and as privileged user when connecting from another network.

To reproduce I am using the starter docker-compose.yml file with a couple of edits:

version: '3.4'
services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: semitechnologies/weaviate:1.21.2
    ports:
    - 8080:8080
    volumes:
    - weaviate_data:/var/lib/weaviate
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      ENABLE_MODULES: 'text2vec-cohere,text2vec-huggingface,text2vec-palm,text2vec-openai,generative-openai,generative-cohere,generative-palm,ref2vec-centroid,reranker-cohere,qna-openai'
      CLUSTER_HOSTNAME: 'node1'
      #######added lines########
      AUTHENTICATION_APIKEY_ENABLED: 'true'
      AUTHENTICATION_APIKEY_ALLOWED_KEYS: 'readonlykey,adminkey'
      AUTHENTICATION_APIKEY_USERS: 'jane@doe.com,john@doe.com'
      AUTHORIZATION_ADMINLIST_ENABLED: 'true'
      AUTHORIZATION_ADMINLIST_USERS: 'john@doe.com'
      AUTHORIZATION_ADMINLIST_READONLY_USERS: 'jane@doe.com'
volumes:
  weaviate_data:
...

Connecting from the host to the weaviate container gives an error with certain commands (ie. node status).

% docker compose up -d
% env -i bash
% rm -rf /tmp/weaviate_test
% cd /tmp
% python -m virtualenv -p 3.10 weaviate_test
% /tmp/weaviate_test/bin/python -m pip install -q requests
% /tmp/weaviate_test/bin/python
Python 3.10.7 (v3.10.7:6cc6b13308, Sep  5 2022, 14:02:52) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> print(requests.get("http://localhost:8080/v1/nodes", headers={"Authorization": "Bearer adminkey"}).content)
b'{"error":[{"message":"forbidden: user \'anonymous\' has insufficient permissions to list nodes"}]}\n'

Connecting from another container succeeds.

dev % docker run -it python:3.10-alpine sh  
/ % pip install -q requests
/ % python
Python 3.10.13 (main, Aug 26 2023, 01:11:00) [GCC 12.2.1 20220924] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> print(requests.get("http://host.docker.internal:8080/v1/nodes", headers={"Authorization": "Bearer adminkey"}).content)
b'{"nodes":[{"batchStats":{"queueLength":0,"ratePerSecond":0},"gitHash":"de059fb","name":"node1","shards":null,"stats":{"objectCount":0,"shardCount":0},"status":"HEALTHY","version":"1.21.1"}]}\n'

The only change in the command is the hostname localhost:8080 >>> host.docker.internal:8080

Not sure it matters but there are different minor versions (3.10.7 for local and 3.10.13 in container).

If I remove the AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: ‘true’ or set it to ‘false’ I get a different error.

>>> print(requests.get("http://localhost:8080/v1/nodes", headers={"Authorization": "Bearer adminkey"}).content)
b'{"code":401,"message":"anonymous access not enabled, please provide an auth scheme such as OIDC"}'

But it still works from container to container.

@DudaNogueira could you please help?

Hi! That is weird.

I could not run it with python 3.10.7 specifically, and I believe that’s the problem here.

dudanogueira@duda-laptop:/tmp$ python -m virtualenv -p 3.10.7 weaviate_test
RuntimeError: failed to find interpreter for Builtin discover of python_spec='3.10.7'

it gave me 3.10.12 when I install 3.10

dudanogueira@duda-laptop:/tmp$ /tmp/weaviate_test/bin/python -m pip install -q requests
dudanogueira@duda-laptop:/tmp$ /tmp/weaviate_test/bin/python
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> print(requests.get("http://localhost:8080/v1/nodes", headers={"Authorization": "Bearer adminkey"}).content)
b'{"nodes":[{"batchStats":{"queueLength":0,"ratePerSecond":0},"gitHash":"8e46731","name":"node1","shards":null,"stats":{"objectCount":0,"shardCount":0},"status":"HEALTHY","version":"1.21.2"}]}\n'

yeah, that is definitely the issue. If you deliberately run Python 3.10.7, it doesn’t seem to deal well with localhost for some reason:

dudanogueira@duda-laptop:/tmp$ docker run -it python:3.10.7-alpine sh
Unable to find image 'python:3.10.7-alpine' locally
3.10.7-alpine: Pulling from library/python
213ec9aee27d: Pull complete 
47858aee13bf: Pull complete 
7cd8ea6e04a3: Pull complete 
b5146d7f5f1f: Pull complete 
91e8f1b6e067: Pull complete 
Digest: sha256:486782edd7f7363ffdc256fc952265a5cbe0a6e047a6a1ff51871d2cdb665351
Status: Downloaded newer image for python:3.10.7-alpine
/ # pip install -q requests
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.2.2 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
/ # python
Python 3.10.7 (main, Oct  7 2022, 02:19:38) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> print(requests.get("http://host.docker.internal:8080/v1/nodes", headers={"Authorization": "Bearer adminkey"}).content)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 203, in _new_conn
    sock = connection.create_connection(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name does not resolve

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 395, in request
    self.endheaders()
  File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 243, in connect
    self.sock = self._new_conn()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 210, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPConnection object at 0x7f12ac027e80>: Failed to resolve 'host.docker.internal' ([Errno -2] Name does not resolve)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='host.docker.internal', port=8080): Max retries exceeded with url: /v1/nodes (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f12ac027e80>: Failed to resolve 'host.docker.internal' ([Errno -2] Name does not resolve)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=8080): Max retries exceeded with url: /v1/nodes (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f12ac027e80>: Failed to resolve 'host.docker.internal' ([Errno -2] Name does not resolve)"))

There seems to be an issue specifically with this python version.

Let me know if that helps :slight_smile:

Strange. It did work for me.

% docker run -it python:3.10.7-alpine sh
Unable to find image 'python:3.10.7-alpine' locally
3.10.7-alpine: Pulling from library/python
9b18e9b68314: Pull complete 
f46cc1940986: Pull complete 
f140e818c22b: Pull complete 
4ef9070366e9: Pull complete 
c0dc22f7803e: Pull complete 
Digest: sha256:486782edd7f7363ffdc256fc952265a5cbe0a6e047a6a1ff51871d2cdb665351
Status: Downloaded newer image for python:3.10.7-alpine
/ # pip install -q requests
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.2.2 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
/ # python
Python 3.10.7 (main, Oct  7 2022, 05:33:10) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> print(requests.get("http://host.docker.internal:8080/v1/nodes", headers={"Authorization": "Bearer adminkey"}).content)
b'{"nodes":[{"batchStats":{"queueLength":0,"ratePerSecond":0},"gitHash":"8e46731","name":"node1","shards":null,"stats":{"objectCount":0,"shardCount":0},"status":"HEALTHY","version":"1.21.2"}]}\n'

Are you using Docker Desktop on a Mac? I think the network names are different with base docker and also on mac vs windows. It could be that you need to use something different from host.docker.internal.

Also to note I’m on a Mac M1 but the container is also running a arm64 image.

% docker inspect python:3.10.7-alpine
...
        "Architecture": "arm64",
        "Variant": "v8",

I tried with an older version of the container (1.20.2) and same issue.

Also I set LOG_LEVEL: 'trace' and the only message is the same for both test cases when it works and when it doesn’t.

test-weaviate-1  | {"action":"restapi_request","level":"debug","method":"GET","msg":"received HTTP request","time":"2023-09-07T06:51:49Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/v1/nodes","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}

Is there a way to turn on more verbose logging for the auth mechanism specifically?

Oh, right. When I was running a second container, it could not connect to weaviate one as they were never at the same network.

But I have used the very same docker compose file you provided and the same comands on python venv with no issues.

I am running it on Linux, I believe that in mac it should also bind to port 8080, so it shouldn’t be different

Quick info about how auth works internally: For API-key based auth Weaviate does not look at the hostname or any other request-related things other than the Authorization header, so Weaviate should not know where the request came from.

My guess would be that - for whatever reason (python versions, client versions, proxies modifying the request, etc.) that in the case where auth is denied that somehow the header does not make it to the Weaviate server.

Hi @mgreg !

Can you please, try the following, considering this docker compose file:

version: '3.4'
services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: semitechnologies/weaviate:1.21.2
    ports:
    - 8080:8080
    volumes:
    - weaviate_data:/var/lib/weaviate
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      ENABLE_MODULES: 'text2vec-cohere,text2vec-huggingface,text2vec-palm,text2vec-openai,generative-openai,generative-cohere,generative-palm,ref2vec-centroid,reranker-cohere,qna-openai'
      CLUSTER_HOSTNAME: 'node1'
      #######added lines########
      AUTHENTICATION_APIKEY_ENABLED: 'true'
      AUTHENTICATION_APIKEY_ALLOWED_KEYS: 'readonlykey,adminkey'
      AUTHENTICATION_APIKEY_USERS: 'jane@doe.com,john@doe.com'
      AUTHORIZATION_ADMINLIST_ENABLED: 'true'
      AUTHORIZATION_ADMINLIST_USERS: 'john@doe.com'
      AUTHORIZATION_ADMINLIST_READONLY_USERS: 'jane@doe.com'

  app:
    image: python:3.10.7-alpine

volumes:
  weaviate_data:
...


I have run:

docker compose up -d weaviate
docker compose run --rm app sh -c 'pip install -q requests;python -c "import requests; print(requests.get(\"http://weaviate:8080/v1/nodes\", headers={\"Authorization\": \"Bearer adminkey\"}).content)";'

with this output:

[+] Running 1/1
 ✔ Container thread-634-weaviate-1  Started    
... pip warnings, etc, etc
b'{"nodes":[{"batchStats":{"queueLength":0,"ratePerSecond":0},"gitHash":"8e46731","name":"node1","shards":null,"stats":{"objectCount":0,"shardCount":0},"status":"HEALTHY","version":"1.21.2"}]}\n'

and this is my arc:

    "Architecture": "amd64",
    "Os": "linux",