Panic after restart from tombstone recovery

Support
1

Description

After a restart to enable async indexing, a node is panicking when repairing the indexes during start.
The cluster is running and searching/indexing are working as expected.

Error logs:

{action: hnsw_tombstone_cleanup, error: panic occurred: runtime error: index out of range [0] with length 0, level: error, msg: tombstone cleanup errord}
2024-04-14 09:46:58.000 BST
{level: error, msg: Recovered from panic: runtime error: index out of range [0] with length 0, local variables [[]], additional localVars [] , panic: runtime error: index out of range [0] with length 0}

Stacktrace:

2024-04-14 09:46:58.305 BST
	/go/src/github.com/weaviate/weaviate/entities/errors/error_group_wrapper.go:74 +0x145
2024-04-14 09:46:58.305 BST
panic({0x1953a60?, 0xc07694e078?})
2024-04-14 09:46:58.305 BST
	/usr/local/go/src/runtime/panic.go:914 +0x21f
2024-04-14 09:46:58.305 BST
github.com/weaviate/weaviate/adapters/repos/db/priorityqueue.(*Queue[...]).Top(...)
2024-04-14 09:46:58.305 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/priorityqueue/queue.go:63
2024-04-14 09:46:58.305 BST
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*neighborFinderConnector).processRecursively(0xc07a11bc50, 0x1fd86a7, 0xc07a690f00, {{0xc529858000, 0x210e3d0, 0x210e3d0}}, 0x0, 0xffffffffffffffe3)
2024-04-14 09:46:58.305 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/neighbor_connections.go:139 +0x558
2024-04-14 09:46:58.305 BST
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*neighborFinderConnector).doAtLevel(0xc0791e5c50, 0x0)
2024-04-14 09:46:58.305 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/neighbor_connections.go:198 +0xb4d
2024-04-14 09:46:58.305 BST
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*neighborFinderConnector).Do(0xc0791e5c50)
2024-04-14 09:46:58.305 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/neighbor_connections.go:81 +0x47
2024-04-14 09:46:58.305 BST
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*hnsw).reconnectNeighboursOf(0x450430?, 0xc7f7e6f000?, 0x600?, {0xc7f7e6f000?, 0x0?, 0x0?}, {0x0?, 0x0?}, 0x0?, 0x3, ...)
2024-04-14 09:46:58.305 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/neighbor_connections.go:43 +0xcb
2024-04-14 09:46:58.305 BST
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*hnsw).reassignNeighbor(0xc0001bcf00, 0x17c4415, {0x1e88100, 0xc00281aef0}, 0x0?)
2024-04-14 09:46:58.305 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/delete.go:455 +0x7c5
2024-04-14 09:46:58.305 BST
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*hnsw).reassignNeighborsOf.func1()
2024-04-14 09:46:58.305 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/delete.go:337 +0x199
2024-04-14 09:46:58.305 BST
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*hnsw).reassignNeighborsOf.(*ErrorGroupWrapper).Go.func2()
2024-04-14 09:46:58.305 BST
	/go/src/github.com/weaviate/weaviate/entities/errors/error_group_wrapper.go:88 +0x97
2024-04-14 09:46:58.305 BST
golang.org/x/sync/errgroup.(*Group).Go.func1()
2024-04-14 09:46:58.305 BST
	/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56
2024-04-14 09:46:58.305 BST
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 356924785
2024-04-14 09:46:58.305 BST
	/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96

Server Setup Information

  • Weaviate Server Version: v1.24.8
  • Deployment Method: k8s
  • Multi Node? Number of Running Nodes: 5
  • Client Language and Version: Go v4.13.1
2

hi @jbendotnet !

Thanks for posting here and following up in our slack

I understand our team is already helping you on that.

Thanks!

1 Like