Docker to K8s Backups using s3

[Sudharshan]

Description

A backup was taken from a multi-node Docker environment using S3 storage, then restored to a Kubernetes environment deployed with Helm and configured with three replicas.

Server Setup Information

• Weaviate Server Version: semitechnologies/weaviate:1.23.7(Docker), semitechnologies/weaviate:1.27.2(K8s)
• Deployment Method: docker & k8s
• Multi Node? Number of Running Nodes: 3
• Client Language and Version: Python
• Multitenancy: Nil

Any additional Information

Taking the backup records from weaviate docker running environment and stored the data backups on S3 Bucket.

Logs from K8s Deployment

{“action”:“try_restore”,“backend”:“s3”,“backup_id”:“first-backup”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“level”:“error”,“msg”:“cannot resolve hostname for "node1"”,“time”:“2024-11-11T10:29:59Z”,“took”:6559398683}

Encountered issues while attempting to change the hostname and pod settings in a Kubernetes deployment, using both Helm and the YAML configuration file.

[DudaNogueira]

hi @Sudharshan !!

Welcome to our community

This is an interesting thing to try and write an article.

I believe that the issue here is the different cluster hostname.

while in docker, your node hostname were node1, in your k8s it is different.

Depending on the size of your dataset, you can also migrate the data over.

I will play around more with this and poke around internally to see if we can come up with some guide o how to restore on that scenario.

Thanks!

[Sudharshan]

K8s Deployment using Helm

Provide a reference link for deploying Weaviate on Kubernetes using Helm.

Made changes to the replicas in the values.yaml file and updated the S3 backup configuration with the bucket details and AWS keys. Additionally, attempted to modify the naming convention for the pod but encountered issues preventing the pod service from starting.
[Tested using a single-node cluster for validation purposes only…]

kubectl describe pod/node-0 -n weaviate

Log for K8s Pod

{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:32Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:34Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:35Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:36Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:38Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:39Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:40Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:42Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:43Z”}
{“action”:“bootstrap”,“build_git_commit”:“64457c2”,“build_go_version”:“go1.22.9”,“build_image_tag”:“v1.27.2”,“build_wv_version”:“1.27.2”,“join_list”:{“weaviate-0”:8300},“level”:“warning”,“msg”:“unable to resolve any node address to join”,“time”:“2024-11-12T06:23:44Z”}

Is there a way to migrate data from a Docker environment to a Kubernetes environment in Weaviate?

Thanks

[DudaNogueira]

well, if you don’t want to move your data, and is ok with actually mgirating it (copying from source and inserting to target), you can use this migration guide:

Migrate data | Weaviate

If course, it would be interesting to tame this store from different deployments, but that can take some time