Docker to K8s Backups using s3

Support
,
1

Description

A backup was taken from a multi-node Docker environment using S3 storage, then restored to a Kubernetes environment deployed with Helm and configured with three replicas.

Server Setup Information

  • Weaviate Server Version: semitechnologies/weaviate:1.23.7(Docker), semitechnologies/weaviate:1.27.2(K8s)
  • Deployment Method: docker & k8s
  • Multi Node? Number of Running Nodes: 3
  • Client Language and Version: Python
  • Multitenancy: Nil

Any additional Information

Taking the backup records from weaviate docker running environment and stored the data backups on S3 Bucket.

Logs from K8s Deployment

{“action”:“try_restore”,“backend”:“s3”,“backup_id”:“first-backup”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“level”:“error”,“msg”:“cannot resolve hostname for "node1"”,“time”:“2024-11-11T10:29:59Z”,“took”:6559398683}

K8s Deployment Data Backups Restoring
K8s Deployment Data Backups Restoring1524Ă—115 11.5 KB

Encountered issues while attempting to change the hostname and pod settings in a Kubernetes deployment, using both Helm and the YAML configuration file.

2

hi @Sudharshan !!

Welcome to our community :hugs:

This is an interesting thing to try and write an article. :thinking:

I believe that the issue here is the different cluster hostname.

while in docker, your node hostname were node1, in your k8s it is different.

Depending on the size of your dataset, you can also migrate the data over.

I will play around more with this and poke around internally to see if we can come up with some guide o how to restore on that scenario.

Thanks!

1 Like
3

K8s Deployment using Helm

Provide a reference link for deploying Weaviate on Kubernetes using Helm.

Made changes to the replicas in the values.yaml file and updated the S3 backup configuration with the bucket details and AWS keys. Additionally, attempted to modify the naming convention for the pod but encountered issues preventing the pod service from starting.
[Tested using a single-node cluster for validation purposes only…] :melting_face:

kubectl describe pod/node-0 -n weaviate

Pod Status for Naming Changes
Pod Status for Naming Changes1532Ă—375 33 KB

Log for K8s Pod

{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:32Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:34Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:35Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:36Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:38Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:39Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:40Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:42Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:43Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:44Z”}

Is there a way to migrate data from a Docker environment to a Kubernetes environment in Weaviate?

Thanks :slight_smile:

4

well, if you don’t want to move your data, and is ok with actually mgirating it (copying from source and inserting to target), you can use this migration guide:

If course, it would be interesting to tame this store from different deployments, but that can take some time :grimacing: