Weaviate does not recover after panic

Support
1

Description

Server Setup Information

  • Weaviate Server Version: weaviate: 1.30.8
  • Client Language and Version: python

Any additional Information

During the indexing process, the search request failed.
The approach used for indexing: to remove an existing index (i.e., make the class have 0 items) and then create a new one for the same class.

github.com/weaviate/weaviate/entities/errors.(*ErrorGroupWrapper).setDeferFunc.func1({0xc0069f4da0, 0x1, 0x1})
" /go/src/github.com/weaviate/weaviate/entities/errors/error_group_wrapper.go:76 +0x175"
panic({0x2acc120?, 0xc0946f7cc8?})
" /usr/local/go/src/runtime/panic.go:792 +0x132"
github.com/weaviate/weaviate/entities/errors.(*ErrorGroupWrapper).Go.func1()
" /go/src/github.com/weaviate/weaviate/entities/errors/error_group_wrapper.go:90 +0x8b"
explorer: get class: vector search: object vector search at index taa_europe_be_nl_be: shard taa_europe_be_nl_be_K5NVsGrCahei: panic occurred: runtime error: index out of range [19679] with length 1000

After this error, Weaviate gets stuck.

Weaviate successfully returns responses for search and delete requests. However, write requests do not fail; instead, they remain in a pending state for the entire duration.

Therefore, Weaviate does not recover after this issue occurs.

2

hi @user1505 !

Welcome to our community :hugs:

1.23.7 is an old server, and a lot has changed.

Can you try backing it up and upgrading to 1.23.latest?

3

My bad, pasted a wrong version.
We are using 1.30.8
Updated the message

4

No worries!

Have you tried then to backup (snapshot, as the cluster is not up) and then upgrading to 1.30.latest?

as I write, the latest one is 1.30.19

So all nice patches/fixes that avoid errors are contained on 1.30.latest or 1.31.latest, 1.32.latest and so on.

Let me know if this helps!

THanks!