,"panic":"runtime error: index out of range [2] with length 2"

Support
1

Description

Hi
I am running a single Weaviate instance on K8s. Below is the server spec. I am encountering below error very frequently. Also the query time goes up from 100m to 500ms during certain period of the day even though number of queries remains same. Appreciate if your support on this

Server Setup Information

  • Weaviate Server Version: 1.26.3
  • Deployment Method: k8s
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: python
  • Multitenancy?: No
  • CpU = 24 vcpus
  • Memory = 256G

Any additional Information

weaviate-0 weaviate {"build_git_commit":"9a4ea6d","build_go_version":"go1.21.13","build_image_tag":"1.26.3","build_wv_version":"1.26.3","level":"error","msg":"Recovered from panic: runtime error: index out of range [2] with length 2, local variables [[]], additional localVars []\n","panic":"runtime error: index out of range [2] with length 2","time":"2024-12-02T23:49:00Z"}
weaviate-0 weaviate goroutine 3247119061 [running]:
weaviate-0 weaviate runtime/debug.Stack()
weaviate-0 weaviate 	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
weaviate-0 weaviate runtime/debug.PrintStack()
weaviate-0 weaviate 	/usr/local/go/src/runtime/debug/stack.go:16 +0x13
weaviate-0 weaviate github.com/weaviate/weaviate/entities/errors.(*ErrorGroupWrapper).setDeferFunc.func1({0xddf553d320, 0x1, 0x1})
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/entities/errors/error_group_wrapper.go:76 +0x175
weaviate-0 weaviate panic({0x20565e0?, 0xddf5544738?})
weaviate-0 weaviate 	/usr/local/go/src/runtime/panic.go:914 +0x21f
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/distancer.CosineDistanceProvider.Step(...)
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/distancer/cosine_dist.go:60
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db/vector/compressionhelpers.(*DistanceLookUpTable).LookUp(0xdc9e491d50, {0xd3df43f8c0, 0x20, 0x2923b00?}, 0xc0034b0b40)
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/compressionhelpers/product_quantization.go:97 +0x12d
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db/vector/compressionhelpers.(*ProductQuantizer).Distance(...)
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/compressionhelpers/product_quantization.go:430
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db/vector/compressionhelpers.(*PQDistancer).Distance(0x0?, {0xd3df43f8c0?, 0x51f13c0?, 0x20?})
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/compressionhelpers/product_quantization.go:350 +0x58
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db/vector/compressionhelpers.(*quantizedCompressorDistancer[...]).DistanceToNode(0x2102102102222a52, 0xd44d878918?)
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/compressionhelpers/compression.go:420 +0xb3
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*hnsw).distToNode(0xc003f08dc0?, {0x28fa630?, 0xddf5143590?}, 0x16?, {0xdc4af0b400?, 0x100000a20c28830a?, 0xdda46af700?})
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/index.go:522 +0x42
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*hnsw).knnSearchByVector(0xc003f08dc0, {0xdc4af0b400, 0x300, 0x300}, 0x32, 0xc4bffe9cc0?, {0x0, 0x0})
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/search.go:474 +0x1e5
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*hnsw).SearchByVector(0xc003f08dc0, {0xdc4af0a800?, 0xd44d878b90?, 0xd44d878d10?}, 0x1223b0a?, {0x0, 0x0})
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/search.go:75 +0x2ba
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db.(*IndexQueue).search(0xc0035cd9e0, {0xdc4af0a800?, 0x300, 0x300}, 0xbf800000, 0x32, {0x0?, 0x0?})
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/index_queue.go:546 +0x1f2
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db.(*IndexQueue).SearchByVector(...)
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/index_queue.go:526
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db.(*Shard).ObjectVectorSearch.func2()
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/shard_read.go:437 +0x245
weaviate-0 weaviate github.com/weaviate/weaviate/adapters/repos/db.(*Shard).ObjectVectorSearch.(*ErrorGroupWrapper).Go.func3()
weaviate-0 weaviate 	/go/src/github.com/weaviate/weaviate/entities/errors/error_group_wrapper.go:90 +0x97
weaviate-0 weaviate golang.org/x/sync/errgroup.(*Group).Go.func1()
weaviate-0 weaviate 	/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x56
weaviate-0 weaviate created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 3247118991
weaviate-0 weaviate 	/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0x96
2

hi @Jeewan !!

I suggest you to upgrade to 1.26.latest.

As I write, the latest at 1.26 is 1.26.11

We always backport critical patches to at least 3 versions.

Let me know if this is possible and if that fixes those panics.

THanks!

3

Hi @Jeewan, could you tell us you PQ configuration? I see there is a problem with the lengths of the centroids in your PQ but it will help me to better understand the problem if I could know your PQ settings.
Thanks for reporting the issue!
Looking forward to the information.

4

Hi @Abdel_Rodriguez ,
thanks for response.
Pls find the PQ configuration

'pq': {'bitCompression': False,
     'centroids': 256,
     'enabled': True,
     'encoder': {'distribution': 'log-normal', 'type': 'kmeans'},
     'segments': 64,
     'trainingLimit': 100000}