Cannot resolve hostname after restoring a backup from another cluster

Description

We’re in a process of migrating our dev setup from docker to k8s. On the old instance node1 (deployed with docker-compose) we’ve created a backup like this:

curl -s \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"id": "migration-backup"}' \
  localhost:8492/v1/backups/azure

The backup finished and we then proceeded to restore the backup on a new instance weaviate-0 (deployed with Weaviate Helm Charts | Weaviate Helm chart repo) like this:

curl -s \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"node_mapping": {"node1":"weaviate-0"}}' \
  localhost:8080/v1/backups/azure/migration-backup/restore

The backup was restored successfully, and even the restore_config.json file reflected the node mapping configuration.

{
  "startedAt": "2025-05-30T07:31:56.842745862Z",
  "completedAt": "2025-05-30T07:34:48.884063552Z",
  "id": "migration-backup",
  "nodes": {
    "node1": {
      "classes": [
        "class1",
        "class2",
        "...",
        "classN-1",
        "classN"
      ],
      "status": "SUCCESS",
      "error": ""
    },
    "weaviate-0": {
      "classes": [
        "AIContentData_924a0ff859e1962ab1ac9ef2a87b0cba",
        "AITeamCollectionTest",
        "...",
        "PRODUCT_l2",
        "PRODUCT_manhattan"
      ],
      "status": "SUCCESS",
      "error": ""
    }
  },
  "node_mapping": { "node1": "weaviate-0" },
  "status": "SUCCESS",
  "version": "2.0",
  "serverVersion": "1.24.26",
  "leader": "",
  "error": ""
}

We then proceeded to create a backup from the new instance, like this:

curl -s \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"id": "test-backup"}' \
  localhost:8080/v1/backups/azure

But that operation ended with an error:


{
  "error": [
    {
      "message": "cannot resolve hostname for \"node1\""
    }
  ]
}

We also checked all shards from all classes, and got similar errors:

for class_name in $(curl -s \
  -X GET \
  -H "Content-Type: application/json" \
  localhost:8080/v1/schema | jq -r '.classes[].class'); do
  curl -s \
    -X GET \
    -H "Content-Type: application/json" \
    localhost:8080/v1/schema/${class_name}/shards | jq '.'
done
{
  "error": [
    {
      "message": "shard JPhSUKsFaO4E: resolve node name \"node1\" to host"
    }
  ]
}
{
  "error": [
    {
      "message": "shard 0jWAQYWFvY9g: resolve node name \"node1\" to host"
    }
  ]
}
...
{
  "error": [
    {
      "message": "shard z7BT2Bn3crPf: resolve node name \"node1\" to host"
    }
  ]
}
{
  "error": [
    {
      "message": "shard glGQ6Ao7uyAS: resolve node name \"node1\" to host"
    }
  ]
}

Is there a way to restore a backup from a different instance using different host name?

What if we wanted to restore a backup from a single-node deployment to a multi-node deployment?

Server Setup Information

  • Weaviate Server Version:
    • source: 1.24.26
    • destination: 1.30.0
  • Deployment Method:
    • source: docker
    • destination: k8s
  • Multi Node? Number of Running Nodes:
    • source: 1
    • destination: 1
  • Client Language and Version: raw http requests via cURL
  • Multitenancy?: no

Hi @rakkhin,

Welcome to our community — it’s great to have you here with us! :hugs:

This is far more complicated than just backing up and restoring in your current situation. Even if the backup itself worked, it’s not guaranteed to succeed in this below context due to the deployment differences.

My understanding:

  • From Docker to Kubernetes
  • From 1 node to multiple nodes
  • And across a major version jump from v1.24.26 to v1.30.0

That’s a lot of significant changes, and unfortunately, I don’t see this working as-is.

I would say if anyone has done this specific migration scenario and worked, I would love to learn it as well to document it. :hugs:

To break it down:

  1. Deployment Differences: You’re migrating from Docker to Kubernetes. This is a architectural change that adds complexity to this. Weaviate’s backup and restore functionality assumes you’re restoring to a similar deployment configuration⁠⁠. The change in deployment architecture can affect how the nodes are configured, named, and how they communicate with each other. Even if restore appears successful, it’s failing internally because shard mappings still reference the old node name.
  2. Version Compatibility: Jumping from v1.24 to v1.30 have several major breaking changes. Weaviate supports some backward compatibility, but this version gap will definitely cause issues.
  3. Cluster Differences: I am not sure but I can see that restores may require the same number of nodes and matching mapping. Changing from a single-node to a multi-node setup can breaks this. :thinking:

From my experience, here’s what I’d suggest:

  • Upgrade your Docker instance:
    • v1.24.26 → v1.25.32 (latest minor in 1.25)
    • Then to v1.26.15 (latest minor in 1.26)
  • Use latest Weaviate Python Client (alternatively TS/JS)
  • Follow the migration guide: Migrate data | Weaviate
  • In your K8s cluster, make sure you’re using v1.30.4 (latest stable minor on 1.30).
  • Re-ingest your data from source cluster to target cluster — this ensures clean distribution across nodes in the new one during ingesting.

This way, your data is properly inserted, distributed, and the cluster can then safely handle future backups and restores within same deployment when needed.

Also make sure:

  • Your replication factor matches the number of nodes in your K8s cluster.
  • You use fixed-size batching if using the Python client.

Best,
Mohamed Shahin
Weaviate Support Engineer
(Ireland, GMT/UTC)

1 Like

Hi @rakkhin !!

Welcome to our community :hugs:

Can you check what you have defined (if you have) in CLUSTER_HOSTNAME?