Fetch error between ts client and Weaviate when deployed with Docker Compose on Windows

Support
1

Hi,

This may not necessarily be a Weaviate issue and instead a node.js/unidici issue, or more likely, an issue with my lack of understanding of docker-compose networking on Windows, but I thought someone here might have come across it and may be able to help.

I have a node.js app that I run as a service in a docker-compose setup. Weaviate is the other service in the setup.

The node.js tries to communicate through the typescript client on http://weaviate:8080 (weaviate being the correct service name).

This works completely fine on MacOS. However, when deployed using Docker Desktop on Windows, I am experiencing some frustrating behaviour.

The live check succeeds, and so too does setting up the schema in the DB. However, when I make a request to get an item based on uuid (getterById()), the call times out and throws a fetch error.

If I run the node.js source code on the machine with just Weaviate running in docker (using the default docker-compose file), I have no troubles when setting the Weaviate URL in the node.js app to 127.0.0.1:8080 but get the same errors when using localhost:8080. Setting the Weaviate endpoint to 127.0.0.1:8080 when the node.js is running as a container within the docker-compose setup does not work (as http://weaviate:8080 should work).

This suggests some form of networking issue. As I mentioned, it works fine on MacOS machines. I’ve tried with node.js versions from 18.16.0 onwards.

These are the error logs:

ERROR error getting User: TypeError: fetch failed
ERROR TypeError: fetch failed
      at node:internal/deps/undici/undici:12500:13
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async getUser (file:///app/backend/db/user.js:145:25)
      at async file:///app/backend/routes/authRoutes.js:63:22

Server Setup Information

  • Weaviate Server Version: 1.24.10 (also tested on 1.24.0)
  • Deployment Method: Docker compose. Weaviate is service 1, and the node.js app is a separate service within the same docker-compose file.
  • Number of Running Nodes: 1
  • Client Language and Version: typescript latest
2

hi @Kieran-who ! Welcome to our community :hugs: !!

This indeed seems like a misbehavior on Docker for Windows :thinking:

Can you try that same environment but using the python client? Just as a sanity check.

Considering, as you said, both the app and weaviate container are in the same network, they should be able to communicate just fine, as it did in a mac environment.

As I do not have a windows, it is hard for me to try reproducing this. :frowning:

3

Thank you :smile:

I originally suspected it may be tied to something relating to this: TypeError: fetch failed · Issue #1248 · nodejs/undici (github.com)

However, none of the fixes I could implement, such as downgrading node.js to 18.16, seemed to work.

I’ll give the Python client a test run and report back here.

Thanks again for the response!

4

I experienced the same issue, when initializing the client via

this.client = await weaviateClient.client({
	rest: {
		host: process.env.WEAVIATE_HOSTNAME_REST!,
		port: 8080,
		secure: isOnServer() ? true : false,
	},
	grpc: {
		host: process.env.WEAVIATE_HOSTNAME_GRPC!,
		port: 50051,
		secure: isOnServer() ? true : false,
	},
	auth: {
		apiKey: process.env.WEAVIATE_API_KEY!,
	},
});

The problem seems to have been that I set the host for rest to http://localhost. In this case, the client tries a connection on port 80, regardlesss of what I set for port.
When I switch the host for rest to http://localhost:8080 it works correctly.

So it appears to me that the port settings is ignored for rest and must be included in the host setting.

I also checked grpc and interestingly, the port setting is used correctly there.

I am using version 3.0.0-beta.23 of the TypeScript client.

1 Like
5

Created an issue on GitHUb with more info: Port setting is ignored in weaviate.client (rest settings) · Issue #131 · weaviate/typescript-client · GitHub

1 Like
6

nice catch @AccessPointAI !!

Also, from a DX perspective, it would be interesting to have the port as optional, as you could provide the port from the host, like you said, using http://localhost:8080 (which I prefer)

Thanks for opening the issue!

7

Thanks @AccessPointAI,

Before your post, I had a very quick play around trying to recreate it with a single file dummy node.js project that only called the db on start. It wasn’t able to recreate the same issue as I described above when running this in a docker-compose setup with Weaviate.

Good pickup definitely but I was already setting the port in the host string as follows (but I realise now I was using the non-beta ts client):

weaviateHost: process.env.WEAVIATE_HOST || 'weaviate:8080',
weaviateScheme: process.env.WEAVIATE_SCHEME || 'http'
...
weaviateClient = weaviate.client({
    scheme: weaviateScheme,
    host: weaviateHost,
    apiKey: new ApiKey(weaviateApiKey!),
  });

Hopefully, I have a bit more time this week to properly reproduce the issue and see if I can pinpoint it in case it can help anyone else out!

Thanks for the responses!

8

So I don’t think this is a Weaviate issue, but if anyone else comes across this, I’ll put my findings here (although solutions have not been figured out yet…)

I have a docker compose file with the following services:

application 1 - a Docker container with three internal services, including the 1 node.js app mentioned above.
weaviate - using a standard config with an API key for communication
redis - standard redis image

On MacOS (at least on Apple silicon), this set up works with no problems.

However, on a fresh Windows Server 2022 install, using Docker Desktop with WSL backend, there is some form of communication issue between the node.js app and Weaviate, resulting in a timeout error. The error is the same as noted here: fetch times out in under 5 seconds · Issue #1531 · nodejs/undici (github.com)

What I am finding odd (and this may be explained by someone with better knowledge of the ts-client’s setup) is that the node.js app can run a live check, and it can set up the schema in the DB, but once a get object by ID is called, the issue arises.

Using the ‘weaviate’ service name as the host (as per the docker compose specs) resulted in the error; however, when setting the host to the IP address returned by:

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' weav-container

the issue was resolved. However, this is obviously not ideal, as if something restarts, the new IP needs to be retrieved again.

So, that is my update on this issue. I am going to dig further as to why my specific setup is doing this; it seems likely an issue with how I have set up the application 1 docker image, as when trying to reproduce this with a fresh docker image that called only Weaviate from a node.js app, I couldn’t replicate the problem.

1 Like
9

hi @Kieran-who !

What I am finding odd (and this may be explained by someone with better knowledge of the ts-client’s setup) is that the node.js app can run a live check, and it can set up the schema in the DB, but once a get object by ID is called, the issue arises.

This may be because the live check will hit a rest endpoint, while the query (AFAIK) will leverage the GRPC connection.

So it seems that, on a windows host, docker may not be playing well with GRPC for some reason.

10

Thought I’d provide an update on this as I believe I have a fix.

I’m still not sure of the root cause (I still think it’s a node.js thing).

But, if anyone else may stumble across the same issue, the fix was simply to add a DNS section to the Docker compose file for the service of the node.js application. For example:

services:
  nodeJSApp:
    image: whatever/image:latest
    dns:
      - 1.1.1.1
      - 1.0.0.1
      - 8.8.8.8
    ports:
      ...
  weaviate:
      command:
        - --host
        - 0.0.0.0
        - --port
        - '8080'
        - --scheme
        - http
      image: semitechnologies/weaviate:1.24.10
      ports:
        - 8080:8080
        - 50051:50051
        - 2112:2112 
...

Thanks for everyone’s thoughts in this!