Unable to Backup Weaviate Using `backup-filesystem` | Failed Connection

Support
1

Hello!

I’m having some trouble in attempting to create a back-up in Weaviate. I get the error:

UnexpectedStatusCodeException: Backup creation! Unexpected status code: 422, with response body: {'error': [{'message': 'node {"node1" "192.168.112.2:7947"}: unexpected status code 503 (Failed Connection)'}]}.

Whenever I try via the following Python code:

import weaviate

client = weaviate.Client(weaviate_client_addr)

result = client.backup.create(
  backup_id='test-backup-1',
  backend='filesystem',
  include_classes=['Records'],
  wait_for_completion=True,
)
print(result)

For context I’m running it using Docker with the following docker-compose.yml:

version: "3.4"

services:
  weaviate:
    image: semitechnologies/weaviate:1.22.5
    ports:
      - 80:8080
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "true"
      PERSISTENCE_DATA_PATH: "/var/lib/weaviate"
      CLUSTER_HOSTNAME: "node1"
      DEFAULT_VECTORIZER_MODULE: "text2vec-openai"
      ENABLE_MODULES: "backup-filesystem,text2vec-openai,generative-openai"
      TRANSFORMERS_INFERENCE_API: http://t2v-transformers:8080
      NO_PROXY: t2v-transformers,weaviate,localhost,127.0.0.1,node1
      BACKUP_FILESYSTEM_PATH: /opt/weaviate/backup
    volumes:
      - /opt/weaviate/data:/var/lib/weaviate

I can see that the backup-filesystem module is enabled as when querying the client for the meta_info with the following Python script:

import weaviate

client = weaviate.Client(weaviate_client_addr)
meta_info = client.get_meta()
print(meta_info)

I receive the following:

{'hostname': 'http://[::]:8080', 'modules': {'backup-filesystem': {'backupsPath': '/opt/weaviate/backup'}, 'generative-openai': {'documentationHref': 'https://platform.openai.com/docs/api-reference/completions', 'name': 'Generative Search - OpenAI'}, 'text2vec-openai': {'documentationHref': 'https://platform.openai.com/docs/guides/embeddings/what-are-embeddings', 'name': 'OpenAI Module'}}, 'version': '1.22.5'}

Additionally, from checking the status of my nodes with the following Python block:

import weaviate

client = weaviate.Client(weaviate_client_addr)
nodes_status = client.cluster.get_nodes_status()
print(nodes_status)

I can see that the status is live and running:

[{'batchStats': {'queueLength': 0, 'ratePerSecond': 0}, 'gitHash': 'ed066e7', 'name': 'node1', 'shards': [{'class': 'Records', 'name': 'TrqAXeHtXtFA', 'objectCount': 1299618, 'vectorIndexingStatus': 'READY', 'vectorQueueLength': 0}], 'stats': {'objectCount': 1592164, 'shardCount': 3}, 'status': 'HEALTHY', 'version': '1.22.5'}]

Any advice or clarification in how to resolve this issue would be very much so appreciated. Please let me know if you require any additional details.

2

Hi! Welcome to our community :hugs:

Do you see anything in Weaviate log?

This seems like the client never even connected to Weaviate :thinking:

3

Some more info to add, is that our network is behind a corporate proxy, and we have the CLUSTER_NAME set to “node1”

so far we have managed to set the weaviate container a fixed IP, as shown (192.168.16.2) and added extra hosts entry like “node1: 192.168.16.2” to container’s /etc/hosts file.

however, given curl node1:7947 from inside the container works, callling the backup endpoint still returns error

{“error”:[{“message”:“node {"node1" "192.168.16.2:7947"}: unexpected status code 503 (Failed Connection)”}]}

While it happens, we only captured two lines of log at ‘trace’ level

{"action":"restapi_request","level":"debug","method":"POST","msg":"received HTTP request","time":"2023-11-29T22:10:35Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/v1/backups/filesystem","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}
{"action":"try_backup","backend":"filesystem","backup_id":"my-very-first-backup","level":"error","msg":"node {\"node1\" \"192.168.16.2:7947\"}: unexpected status code 503 (Failed Connection)","time":"2023-11-29T22:10:47Z","took":12091909569}

In the running container, we did set a no_proxy env var as
no_proxy: 192.168.16.2

to avoid this request go through the corporate proxy, but we are not sure if the weaviate go app is actually taking that, given the curl command above does show it’s using no_proxy

4

Hi!

Can you try starting the backup using curl?

something like:

curl \
    -X POST \
    -H "Content-Type: application/json" \
    -d '{
         "id": "my-very-first-backup",
         "include": ["Article", "Publication"]
        }' \
    http://localhost:8080/v1/backups/filesystem

Let me know if that works.

5

Sure!
That curl command (using our existing class name though) gave exactly the same error as the previous post shown.

The curl was sent from insde a running weaviate container, where

CLUSTER_NAME is node1
no_proxy env var has set for node1’s container IP and the hostname node1
/etc/hosts has entry to resolve the hostname node1 to container IP

I feel this could be related to how the GO app makes the node status call before backup and it wasn’t using proxy

Plus we want to know if it can disable the cluster mode with a single container deployment, that way it knows there is only one node and skip the steps like node checking and assume itself is in a cluster

6

Were you able to backup? I am having some problem. Server cannot connect.