Panic: index out of range [-1] runtime errors in log

Support
1

Hello again,

We’re running Weaviate 1.22.3 with a few thousand test records of type Page3.

In the weaviate logs we see recurring error:

{"action":"cyclemanager","callback_id":"segmentgroup/compaction//var/lib/weaviate/page3_2bb2c745-1864-43f5-9e2b-81dd305bc14f_lsm/property_article_type_searchable","callbacks_id":"store/compaction/..","class":"Page3","index":"page3","level":"error","msg":"callback panic: runtime error: index out of range [-1]","shard":"2bb2c745-1864-43f5-9e2b-81dd305bc14f","time":"2023-12-13T00:25:25Z"}

It repeats sometimes in less than a minute, other times it’s more like a few minutes but it fills majority of our logs and it’s been running like that for days. What can cause the error and does it impact operations of the DB?

Best regards!

2

Hi @kokogrr ! Sorry for the delay here.

Can you change the LOG LEVEL (doc here) to trace? and try to get more info?

Is this a single node environment?

It seems some issue in a compaction cycle for this shard, maybe because of a corrupted index.

Can you reproduce this consistently? Have you tried newer versions? Eg. the new version 1.23?

3

Hi Duda,

thanks for suggestions, this is a single instance working on Docker with pretty standard docker-compose taken from weaviate generator and a few modified env vars like disk space threshold . I wouldn’t say I can reproduce it as in have minimal repro setup but It happens consistently across container rebuilds both on local machine and DigitalOcean droplet. I’ll try with higher log level.

Best regards,
Konrad

4

I’m getting the same problem. Have you found out why?

5

Hi!
We have seen this kind of issue being fixed in Weaviate 1.24+

I will mark this topic as solved, but please, don’t hesitate to open a new topic if this issue surface again.

Thanks!