Compaction Errors

Support
1

Hi all,

I am seeing a continuous stream of compaction errors in the log stream. Is this normal? Does it have any affect on the performance?

ERRO[0679] compaction failed                             action=lsm_compaction class=Document error="replace compacted segments: precompute segment meta: init bloom filter for secondary index at 0: a secondary bloom filter already exists with path /root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects/segment-1689676389149982374.secondary.0.bloom.tmp" index=document path=/root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects shard=kxlWAyuQqVRN
ERRO[0685] compaction failed                             action=lsm_compaction class=Document error="replace compacted segments: precompute segment meta: init bloom filter for secondary index at 0: a secondary bloom filter already exists with path /root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects/segment-1689676389149982374.secondary.0.bloom.tmp" index=document path=/root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects shard=kxlWAyuQqVRN
ERRO[0689] compaction failed                             action=lsm_compaction class=Document error="replace compacted segments: precompute segment meta: init bloom filter for secondary index at 0: a secondary bloom filter already exists with path /root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects/segment-1689676389149982374.secondary.0.bloom.tmp" index=document path=/root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects shard=kxlWAyuQqVRN
ERRO[0694] compaction failed                             action=lsm_compaction class=Document error="replace compacted segments: precompute segment meta: init bloom filter for secondary index at 0: a secondary bloom filter already exists with path /root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects/segment-1689676389149982374.secondary.0.bloom.tmp" index=document path=/root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects shard=kxlWAyuQqVRN
ERRO[0699] compaction failed                             action=lsm_compaction class=Document error="replace compacted segments: precompute segment meta: init bloom filter for secondary index at 0: a secondary bloom filter already exists with path /root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects/segment-1689676389149982374.secondary.0.bloom.tmp" index=document path=/root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects shard=kxlWAyuQqVRN
ERRO[0703] compaction failed                             action=lsm_compaction class=Document error="replace compacted segments: precompute segment meta: init bloom filter for secondary index at 0: a secondary bloom filter already exists with path /root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects/segment-1689676389149982374.secondary.0.bloom.tmp" index=document path=/root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects shard=kxlWAyuQqVRN
ERRO[0708] compaction failed                             action=lsm_compaction class=Document error="replace compacted segments: precompute segment meta: init bloom filter for secondary index at 0: a secondary bloom filter already exists with path /root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects/segment-1689676389149982374.secondary.0.bloom.tmp" index=document path=/root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects shard=kxlWAyuQqVRN
ERRO[0712] compaction failed                             action=lsm_compaction class=Document error="replace compacted segments: precompute segment meta: init bloom filter for secondary index at 0: a secondary bloom filter already exists with path /root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects/segment-1689676389149982374.secondary.0.bloom.tmp" index=document path=/root/vectorstore_weaviate/weaviate_data/data/weaviate/document_kxlWAyuQqVRN_lsm/objects shard=kxlWAyuQqVRN
1 Like
2

Hi! I have asked about this with our team.

Is there any compaction success messages or all of them fails?

3

I have a similar compaction error issue. All of them fail, never a success. For days on end.

Docker image: semitechnologies/weaviate:1.21.1

Schema for Matches:
[
{“name”: “client_id”, “dataType”: [“int”]},
{“name”: “category_id”, “dataType”: [“int”]},
{“name”: “ctn_id”, “dataType”: [“int”]},
{“name”: “is_description_only”, “dataType”: [“boolean”]},
{“name”: “is_material_ctn”, “dataType”: [“boolean”]},
{“name”: “created_at”, “dataType”: [“date”]},
]

Error message:
{“action”:“lsm_compaction”,“class”:“Matches”,“error”:“replace compacted segments: precompute segment meta: a bloom filter already exists with path /var/lib/weaviate/matches_V0cXxiU61Q6I_lsm/property_is_description_only/segment-1697059248081165705.bloom.tmp”,“index”:“matches”,“level”:“error”,“msg”:“compaction failed”,“path”:“/var/lib/weaviate/matches_V0cXxiU61Q6I_lsm/property_is_description_only”,“shard”:“V0cXxiU61Q6I”,“time”:“2023-10-18T08:42:43Z”}

{“action”:“lsm_compaction”,“class”:“Matches”,“error”:“replace compacted segments: precompute segment meta: a bloom filter already exists with path /var/lib/weaviate/matches_V0cXxiU61Q6I_lsm/property_is_description_only/segment-1697059248081165705.bloom.tmp”,“index”:“matches”,“level”:“error”,“msg”:“compaction failed”,“path”:“/var/lib/weaviate/matches_V0cXxiU61Q6I_lsm/property_is_description_only”,“shard”:“V0cXxiU61Q6I”,“time”:“2023-10-18T08:42:46Z”}

{“action”:“lsm_compaction”,“class”:“Matches”,“error”:“replace compacted segments: precompute segment meta: a bloom filter already exists with path /var/lib/weaviate/matches_V0cXxiU61Q6I_lsm/property_is_description_only/segment-1697059248081165705.bloom.tmp”,“index”:“matches”,“level”:“error”,“msg”:“compaction failed”,“path”:“/var/lib/weaviate/matches_V0cXxiU61Q6I_lsm/property_is_description_only”,“shard”:“V0cXxiU61Q6I”,“time”:“2023-10-18T08:42:49Z”}

{“action”:“lsm_compaction”,“class”:“Matches”,“error”:“replace compacted segments: precompute segment meta: a bloom filter already exists with path /var/lib/weaviate/matches_V0cXxiU61Q6I_lsm/property_is_description_only/segment-1697059248081165705.bloom.tmp”,“index”:“matches”,“level”:“error”,“msg”:“compaction failed”,“path”:“/var/lib/weaviate/matches_V0cXxiU61Q6I_lsm/property_is_description_only”,“shard”:“V0cXxiU61Q6I”,“time”:“2023-10-18T08:42:52Z”}

{“action”:“lsm_compaction”,“class”:“Matches”,“error”:“replace compacted segments: precompute segment meta: a bloom filter already exists with path /var/lib/weaviate/matches_V0cXxiU61Q6I_lsm/property_is_description_only/segment-1697059248081165705.bloom.tmp”,“index”:“matches”,“level”:“error”,“msg”:“compaction failed”,“path”:“/var/lib/weaviate/matches_V0cXxiU61Q6I_lsm/property_is_description_only”,“shard”:“V0cXxiU61Q6I”,“time”:“2023-10-18T08:42:55Z”}

4

I am also facing similar issue. Is there any solution?

5

Hi @Yogesh_Sharma, this issue should be solved since v1.24.4

Some temporal files with .tmp extensions were not automatically deleted next compaction iteration and it’s causing the failure.

My recommendation is either to upgrade or directly remove those temporary files from the data folder if the upgrade is not a possibility (always it’s better to make a backup of the data before upgrading or deleting files from the data folder just for in case)

6

I’m using latest version(v1.26.4) but still getting the same error:
It is on Single mode.
{"action":"lsm_compaction","class":"ItemV3","error":"write index: unlinkat /var/lib/weaviate/itemv3/aGmlzfgC3gOv/lsm/property_createdate_searchable/segment-1726498768692898772.dbcompaction.scratch.d: directory not empty","index":"itemv3","level":"error","msg":"compaction failed","path":"/var/lib/weaviate/itemv3/aGmlzfgC3gOv/lsm/property_createdate_searchable","shard":"aGmlzfgC3gOv","time":"2024-09-16T15:01:17Z"}

Due to this we are not able to trigger any new change on the Weaviate.
Could you please help me with the fix of the issue?
We are stuck due to this issue.
@jeronimo_irazabal @DudaNogueira