Node: Recovered from panic: assignment to entry in nil map

Support
1

Description

We have one node in our cluster that’s continually panic/recovering with the following error:

2024-06-13 18:13:59.000 BST
{level: error, msg: Recovered from panic: assignment to entry in nil map}
2024-06-13 18:13:59.725 BST
goroutine 9907763 [running]:
2024-06-13 18:13:59.725 BST
runtime/debug.Stack()
2024-06-13 18:13:59.725 BST
	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
2024-06-13 18:13:59.725 BST
runtime/debug.PrintStack()
2024-06-13 18:13:59.725 BST
	/usr/local/go/src/runtime/debug/stack.go:16 +0x13
2024-06-13 18:13:59.725 BST
github.com/weaviate/weaviate/entities/errors.GoWrapper.func1.1()
2024-06-13 18:13:59.725 BST
	/go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:30 +0x110
2024-06-13 18:13:59.725 BST
panic({0x18e8fa0?, 0x200dab0?})
2024-06-13 18:13:59.725 BST
	/usr/local/go/src/runtime/panic.go:914 +0x21f
2024-06-13 18:13:59.725 BST
github.com/weaviate/weaviate/adapters/repos/db/inverted.(*JsonShardMetaData).TrackProperty(0xc005d46600, {0xc00050611c, 0x4}, 0x40000000)
2024-06-13 18:13:59.725 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/inverted/new_prop_length_tracker.go:218 +0x408
2024-06-13 18:13:59.725 BST
github.com/weaviate/weaviate/adapters/repos/db.(*Shard).SetPropertyLengths(0xc005ba9560, {0xc3f8bd9000?, 0x21, 0x41a848?})
2024-06-13 18:13:59.725 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/shard_write_inverted_lsm.go:205 +0x95
2024-06-13 18:13:59.725 BST
github.com/weaviate/weaviate/adapters/repos/db.(*Shard).updateInvertedIndexLSM(0xc005ba9560, 0xc4255ac3c0, {0x36be4d, 0x0, 0x0, 0x0, 0x0}, 0x0)
2024-06-13 18:13:59.725 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/shard_write_put.go:412 +0x38e
2024-06-13 18:13:59.725 BST
github.com/weaviate/weaviate/adapters/repos/db.(*Shard).putObjectLSM(0xc005ba9560, 0xc4255ac3c0, {0xc3f8bf3540, 0x10, 0x10})
2024-06-13 18:13:59.725 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/shard_write_put.go:288 +0x3cc
2024-06-13 18:13:59.725 BST
github.com/weaviate/weaviate/adapters/repos/db.(*objectsBatcher).storeObjectOfBatchInLSM(0xc3f8b51400, {0x202b3d8, 0x2dd7060}, 0x0?, 0xc4255ac3c0)
2024-06-13 18:13:59.725 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/shard_write_batch_objects.go:207 +0xd4
2024-06-13 18:13:59.725 BST
github.com/weaviate/weaviate/adapters/repos/db.(*objectsBatcher).storeSingleBatchInLSM.func1()
2024-06-13 18:13:59.725 BST
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/shard_write_batch_objects.go:177 +0x11a
2024-06-13 18:13:59.725 BST
github.com/weaviate/weaviate/entities/errors.GoWrapper.func1()

This appears to be a similar issue to Crahsloop with `panic: assignment to entry in nil map` · Issue #4128 · weaviate/weaviate · GitHub, which is still open.

Are there any remedial steps we can take to clear up the node?

Server Setup Information

  • Weaviate Server Version: v1.24.19
  • Deployment Method: k8s
  • Multi Node? Number of Running Nodes: 5
  • Client Language and Version: Go, 1.21
  • Multitenancy?: No
2

hi @jbendotnet !!

Thanks for reporting.

I see from this discussion in slack that this was already identified, investigated and fixed.

We’ll add more details here soon. :slight_smile:

3

@DudaNogueira yep, we’re ok now - with @antas-marcin’s help it’s been resolved by restarting the cluster with the RECOUNT_PROPERTIES_AT_STARTUP option.

Steps to fix:

  1. Update the cluster config enabling RECOUNT_PROPERTIES_AT_STARTUP
  2. Restart the STS / cluster
  3. Wait…
  4. Once all nodes have restarted, revert the config change and restart the STS/cluster
  5. If you stopped writes, start them
2 Likes