Issue with Weaviate Backup Process on Azure

Support
,
1

Description

Hello Team, I’m reaching out regarding an issue we’re experiencing with our Weaviate instance deployed on our Kubernetes development environment.
Recently, we’ve encountered difficulties with our daily database backups to Azure Blob using the Weaviate backup API. Our backup script is failing consistently due to a previous failure of the Weaviate backup instance.
Specifically, we’re receiving the following error message: “Cannot create new backup. Backup ‘gaibot-weaviate-backup_20240417000000’ is not yet released. This means its contents have not yet been fully copied to its destination. Please try again later.”
Unfortunately, we’re unsure about how to proceed to release the old backup and resume our development activities.
Could you please assist us in resolving this issue.

Server Setup Information

  • Weaviate Server Version:
  • Deployment Method: Kubernetes
  • Number of Running Nodes: One node
  • Weaviate Version: 1.19.1
2

hi @sanjeev1678 ! Welcome to our community :hugs:

1.19.1 is a fairly old version :thinking: A lot have changed since.

Is that a big dataset? Can you check if this issue persist on latest version?

Have you tried deleting that backup id?

Thanks!

3

Hello @DudaNogueira, Thank you for your response.

The problem initially started during the execution of our backup API script on April 17th. The error message we encountered was: “Upload file for object ‘gaibot-weaviate-backup_20240417000000/weaviate-0/slbnewsdoc_3D7tXnqXsz49_lsm/objects/segment-1712587053178420744.db’: context deadline exceeded”. Since then, any attempts to initiate a new backup have been unsuccessful, consistently resulting in the following error: “Backup class Insightfirstcalamosoradnews descriptor: cannot create new backup. Backup ‘gaibot-weaviate-backup_20240417000000’ is not yet released. This means its contents have not yet been fully copied to its destination. Please try again later”.

Below are my responses regarding the suggestions provided:

  1. Yes, I understand that the current version is old, but unfortunately, upgrading to the
    latest version is not feasible right now.
  2. The data volume is indeed quite substantial, with multiple class table data present.
  3. I have already attempted to address the issue by deleting specific ID files on the
    Azure Blob. However, even after this action, I am still encountering the same error
    when attempting to create a new backup of our weaviate class.
  4. Until unless we are not able to create backup of our current data, we are not able to
    migrate to new version.

Please let me know if you require any additional information. I want to find a solution to release this ‘gaibot-weaviate-backup_20240417000000’ backup process.
Thank you in advance for your assistance.

4

I have found that stopping the instance/cluster and re starting clears the ‘not yet released’ and allows for a new backup. Had this issue after minio ran out of space during backup.

1 Like
5

Thanks for sharing, @djjeffr !!

@sanjeev1678 ! Have you tried this? :thinking:

Thanks!

6

Yes, I have used this to do backup after minio ran out of space.

1 Like
7

Yes it worked.

Thanks Team!!!

1 Like
8

Hello @DudaNogueira

Now I am getting this error, when I am running weaviate backup(.create) method.

The block list may not contain more than 50,000 blocks.\nERROR CODE: BlockListTooLong\n--------------------------------------------------------------------------------\n\ufeff<?xml version="1.0" encoding="utf-8"?>BlockListTooLongThe block list may not contain more than 50,000 blocks.

This is a big dataset with multiple class.
Please let me know how I can fix this issue

  • Weaviate Server Version:
  • Deployment Method: Kubernetes
  • Number of Running Nodes: One node
  • Weaviate Version: 1.23.7