Compation failed

Support
1

Description

Getting the following error in my docker logs:
{“action”:“lsm_compaction”,“class”:“SearchMedia”,“error”:“write index: unlinkat /var/lib/weaviate/searchmedia/IfEh5Wm9v9no/lsm/property__id/segment-1716445594153867176.dbcompaction.scratch.d: directory not empty”,“index”:“searchmedia”,“level”:“error”,“msg”:“compaction failed”,“path”:“/var/lib/weaviate/searchmedia/IfEh5Wm9v9no/lsm/property__id”,“shard”:“IfEh5Wm9v9no”,“time”:“2024-05-23T06:28:05Z”}

I’m not certain what action is even causing this as I’m currently running a long batch script which does a whole bunch of stuff, like read, write, delete, etc.

Question is, when this error is triggered, does this block data insertion/deletion/update?

How do I avoid/fix this issue?

Thanks.

Server Setup Information

  • Weaviate Server Version: 1.24.12
  • Deployment Method: docker
  • Multi Node? Number of Running Nodes: single node
  • Client Language and Version: graphql api

Any additional Information

the log is full of errors like that across various schemas

{"action":"lsm_compaction","class":"SearchMediaAiData","error":"write index: unlinkat /var/lib/weaviate/searchmediaaidata/MBpQZ57WbgJ4/lsm/property_media_id/segment-1716446470745179201.dbcompaction.scratch.d: directory not empty","index":"searchmediaaidata","level":"error","msg":"compaction failed","path":"/var/lib/weaviate/searchmediaaidata/MBpQZ57WbgJ4/lsm/property_media_id","shard":"MBpQZ57WbgJ4","time":"2024-05-23T06:42:28Z"}
{"action":"lsm_compaction","class":"SearchMedia","error":"write index: unlinkat /var/lib/weaviate/searchmedia/IfEh5Wm9v9no/lsm/property_workspace_id/segment-1716446450236052497.dbcompaction.scratch.d: directory not empty","index":"searchmedia","level":"error","msg":"compaction failed","path":"/var/lib/weaviate/searchmedia/IfEh5Wm9v9no/lsm/property_workspace_id","shard":"IfEh5Wm9v9no","time":"2024-05-23T06:41:54Z"}

2

Just did some poking around and noticed that the folder mentioned in the error, eg

2024-05-25T05:40:42Z ERR action=lsm_memtable_flush class=SearchMedia error=flush: unlinkat /var/lib/weaviate/searchmedia/IfEh5Wm9v9no/lsm/property_species/segment-1716615577153457021.scratch.d: directory not empty index=searchmedia msg=flush and switch failed path=/var/lib/weaviate/searchmedia/IfEh5Wm9v9no/lsm/property_species shard=IfEh5Wm9v9no


searchmedia/IfEh5Wm9v9no/lsm/property_species

has a bunch of temp files within it

-rw-r--r--. 1 root root    44 May 21 13:30 segment-1716288681209710108_1716288762729670859.bloom.tmp
-rw-r--r--. 1 root root     0 May 21 16:51 segment-1716324171938779796.db
drwxr-xr-x. 2 root root  6144 May 21 16:51 segment-1716324171938779796.scratch.d
-rw-r--r--. 1 root root 15957 May 21 16:51 segment-1716324171938779796.wal
-rw-r--r--. 1 root root     0 May 23 02:22 segment-1716445285001700267.db
drwxr-xr-x. 2 root root  6144 May 23 02:22 segment-1716445285001700267.scratch.d
-rw-r--r--. 1 root root 25183 May 23 02:22 segment-1716445285001700267.wal
-rw-r--r--. 1 root root     0 May 23 02:45 segment-1716446691485583063.db
drwxr-xr-x. 2 root root  6144 May 23 02:45 segment-1716446691485583063.scratch.d
-rw-r--r--. 1 root root 12452 May 23 02:45 segment-1716446691485583063.wal
-rw-r--r--. 1 root root   180 May 23 03:15 segment-1716448427893915688.bloom
-rw-r--r--. 1 root root 30594 May 23 03:15 segment-1716448427893915688.db
-rw-r--r--. 1 root root     0 May 23 03:17 segment-1716448617499313354.db
drwxr-xr-x. 2 root root  6144 May 23 03:18 segment-1716448617499313354.scratch.d
-rw-r--r--. 1 root root 32350 May 23 03:17 segment-1716448617499313354.wal
-rw-r--r--. 1 root root    68 May 25 01:24 segment-1716614577068659393.bloom
-rw-r--r--. 1 root root 12986 May 25 01:24 segment-1716614577068659393.db
-rw-r--r--. 1 root root    60 May 25 01:33 segment-1716615087990253309.bloom
-rw-r--r--. 1 root root  6760 May 25 01:33 segment-1716615087990253309.db
-rw-r--r--. 1 root root     0 May 25 01:35 segment-1716615240522512874.db
drwxr-xr-x. 2 root root  6144 May 25 01:35 segment-1716615240522512874.scratch.d
-rw-r--r--. 1 root root  6988 May 25 01:34 segment-1716615240522512874.wal
-rw-r--r--. 1 root root    52 May 25 01:38 segment-1716615443820378940.bloom
-rw-r--r--. 1 root root  4956 May 25 01:38 segment-1716615443820378940.db
-rw-r--r--. 1 root root     0 May 25 01:40 segment-1716615577153457021.db
drwxr-xr-x. 2 root root  6144 May 25 01:40 segment-1716615577153457021.scratch.d
-rw-r--r--. 1 root root 24940 May 25 01:40 segment-1716615577153457021.wal
-rw-r--r--. 1 root root    44 May 25 01:41 segment-1716615642151563559.bloom
-rw-r--r--. 1 root root  2562 May 25 01:41 segment-1716615642151563559.db
-rw-r--r--. 1 root root 19046 May 25 01:42 segment-1716615708499458602.wal

What’s up with these files?
How do I clean these files up to get rid of the errors? Should I make a backup and just delete them?

As far as I can tell all the data is inserting just fine and I’m able to query it.

3

@maddios - I’m also facing similar issue and got below error. Did you find any resolution for this?

{"action":"lsm_memtable_flush","class":"WeaviateDemo3","error":"flush: unlinkat /var/lib/weaviate/weaviatedemo3/PXksOv6syq7H/lsm/objects/segment-1719829104648996364.scratch.d: directory not empty","index":"weaviatedemo3","level":"error","msg":"flush and switch failed","path":"/var/lib/weaviate/weaviatedemo3/PXksOv6syq7H/lsm/objects","shard":"PXksOv6syq7H","time":"2024-07-01T10:18:50Z"}

@jphwang