Weaviate not backing up for a long time

[swetag]

Hi,

1. I batch uploaded around 20k objects and backed them up in s3, worked fine. I can see the backup json file and node1 folder in s3.
2. Moments later I deleted all the data and then batch uploaded them again which worked fine but this time while creating backups I am seeing a lot of this messages in my logs:
   “HTTP Request: GET http://------:8080/v1/backups/s3/{backup_id}"HTTP/1.1 200 OK"”.
   when I see weaviate logs, there are alot of this kind of logging:
   “action”: “backup_status”; “level”: “info”; “msg”: “”,
   but before these logs I got following logs in my record (in sequence):

“action”: “lsm_memtable_flush”; “error”: “flush: unlinkat /var/lib/weaviate/test/{index_name}/w8G6ci6OWwEL/lsm/property__id/segment-1721615078510284041.scratch.d: directory not empty”

"“action”: “lsm_memtable_flush”; “error”: “flush: unlinkat /var/lib/weaviate/test/{index_name}/w8G6ci6OWwEL/lsm/property_title_searchable/segment-1721615075238438328.scratch.d: directory not empty”

“action”: “lsm_compaction”; “error”: “write index: unlinkat /var/lib/weaviate/test/{index_name}/w8G6ci6OWwEL/lsm/property_annotations_id_searchable/segment-1721615076605784103.dbcompaction.scratch.d: directory not empty”;

“action”: “lsm_compaction”; “error”: “write index: clean up previous scratch space: unlinkat /var/lib/weaviate/test/{index_name}/w8G6ci6OWwEL/lsm/property_annotations_id_searchable/segment-1721615076605784103.dbcompaction.scratch.d/.nfs3f46fabddd01f71200000009: device or resource busy”,

“action”: “lsm_compaction”; “error”: “write indices: unlinkat /var/lib/weaviate/test/{index_name}/w8G6ci6OWwEL/lsm/objects/segment-1721615326755105246.dbcompaction.scratch.d: directory not empty”

“action”: “lsm_memtable_flush”; “error”: “flush: unlinkat /var/lib/weaviate/test/{index_name}/w8G6ci6OWwEL/lsm/property_collections/segment-1721615316392568869.scratch.d: directory not empty”,

“action”: “lsm_memtable_flush”; “error”: “flush: unlinkat /var/lib/weaviate/test/{index_name}/w8G6ci6OWwEL/lsm/property_url_searchable/segment-1721615317703263522.scratch.d: directory not empty”
After around 15 mins (having same messages/logs) My server is timed out and restarted. So the backup never completed.

Server Setup Information

• Weaviate Server Version: weaviate:1.25.6
• Deployment Method: docker
• Client Language and Version: python 3.10.9, weaviate-client: 4.6.5

Environment variables:
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: ‘true’
PERSISTENCE_DATA_PATH: ‘/var/lib/weaviate’
DEFAULT_VECTORIZER_MODULE: ‘none’
ENABLE_MODULES: ‘backup-filesystem,backup-s3’
CLUSTER_HOSTNAME: ‘node1’
QUERY_MAXIMUM_RESULTS: 1000000
DISK_USE_READONLY_PERCENTAGE: 100
DISABLE_TELEMETRY: true
DISABLE_LAZY_LOAD_SHARDS: true

Could you please guide me in how to resolve this issue? TIA.

[DudaNogueira]

hi @swetag !!

Welcome to our community

Nice! This looks like one of our chaos pipelines

How have you deleted those objects in step 2?

When you delete the entire collection, it’s simpler, as no object deletion calculation is needed.

If you delete the objects itself, it will generate tombstones, and then a whole new cycle is triggered.

This seems related to this issue:

Backup gets stuck and Weaviate as well

Can you confirm this issue persists in 1.25.8?

Thanks!

[swetag]

Hey @DudaNogueira , thanks for your response.

1. I catch the uuids of the objects by running a conditional search operation and in this scenario the uuid of all the objects(around 27k) are returned but this need not be the case always. So i can not delete the collection and have to delete the objects only. Anyway, after that I run delete operation with this code:

collection.data.delete_many(where=Filter.by_id().contains_any(uuids_batch))

where each batch has 5k uuids. and this code is inside a for loop over all the uuid batches.

After around 5 sec, the function returns from the delete operation and starts the indexing.

2. The issue persists in 1.25.8 as well.

What should I do to resolve this issue?