Weaviate crashing because of following error, can someone help on this

panic: runtime error: slice bounds out of range [:1066] with capacity 922

goroutine 132 [running]:
github.com/weaviate/weaviate/usecases/byteops.(*ReadWriter).CopyBytesFromBuffer(...)
	/go/src/github.com/weaviate/weaviate/usecases/byteops/byteops.go:77
github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segmentindex.(*DiskTree).readNode(0x4002243c00?, {0xffff6ceaf320, 0x39a, 0x39a})
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segmentindex/disk_tree.go:106 +0x448
github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segmentindex.(*DiskTree).AllKeys(0x4002eba930)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segmentindex/disk_tree.go:180 +0x8c
github.com/weaviate/weaviate/adapters/repos/db/lsmkv.(*segment).computeAndStoreBloomFilter(0x40031203c0, {0x4003200770, 0x70})
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segment_bloom_filters.go:88 +0x38
github.com/weaviate/weaviate/adapters/repos/db/lsmkv.(*segment).initBloomFilter(0x40031203c0)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segment_bloom_filters.go:75 +0x90
github.com/weaviate/weaviate/adapters/repos/db/lsmkv.(*segment).initBloomFilters(0x40031203c0, 0xffff6ceae000?)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segment_bloom_filters.go:38 +0x28
github.com/weaviate/weaviate/adapters/repos/db/lsmkv.newSegment({0x4003200310, 0x6d}, {0x181bc18?, 0x4002b638f0}, 0x0, 0x0?, 0x1, 0x1, 0x1)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segment.go:150 +0x738
github.com/weaviate/weaviate/adapters/repos/db/lsmkv.newSegmentGroup({0x181bc18, 0x4002b638f0}, 0x0, {0x17fca40, 0x40031d4840}, {{0x40030facd0, 0x4e}, {0x13f7698, 0xa}, 0x0, ...})
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segment_group.go:140 +0x654
github.com/weaviate/weaviate/adapters/repos/db/lsmkv.NewBucket({0x1805c08, 0x2410b20}, {0x40030facd0, 0x4e}, {0x40030e9280, 0x36}, {0x181bc18?, 0x4002b638f0?}, 0x0, {0x17fca40, ...}, ...)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/bucket.go:151 +0x2b4
github.com/weaviate/weaviate/adapters/repos/db/lsmkv.(*Store).CreateOrLoadBucket(0x40031d47e0, {0x1805c08, 0x2410b20}, {0x40031bf3b0, 0x13}, {0x4002f54960, 0x4, 0x6})
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/store.go:122 +0xec
github.com/weaviate/weaviate/adapters/repos/db.(*Shard).createPropertyValueIndex(0x40031201e0, {0x1805c08, 0x2410b20}, 0x40026e97a0)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/shard.go:660 +0x4b8
github.com/weaviate/weaviate/adapters/repos/db.(*Shard).createPropertyIndex.func1()
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/shard.go:609 +0x48
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 16
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x98

Hi @Kavali_Kranthi_Kumar !

Welcome to our community.

Does this happend when you perform any action or on start up?

Also, what is the version you are using?

To me it happens when trying to fetch all objects of my class Chunk using offset 10000, or trying to use a limit higher than 10000:

Code:

chunks_query_response_1 = (
client.query
.get(“Chunk”,“post_id”)
.with_limit(10000)
.do()
)

chunks_query_response_2 = (
client.query
.get(“Chunk”,“post_id”)
# .with_limit(10000)
.with_offset(10000)
.do()
)

Schema:

class_Document= {
“class”: “Document”,
“description”: “Represents a document with metadata and associated text chunks.”,
“properties”: [
{
“name”: “tags”,
“dataType”: [“string”],
“description”: “Tags from WordPress”
},
{
“name”: “categories”,
“dataType”: [“string”],
“description”: “Categories from WordPress”
},
{
“name”: “postUpdatedAt”,
“dataType”: [“date”],
“description”: “WordPress post last updated timestamp”
},
{
“name”: “title”,
“dataType”: [“string”],
“description”: “Title of the post”
},
{
“name”: “author”,
“dataType”: [“string”],
“description”: “author of the post”
},
{
“name”: “permalink”,
“dataType”: [“string”],
“description”: “Link of the post”
},
{
“name”: “post_id”,
“dataType”: [“int”],
“description”: “WordPress ID for the post”
},
{
“name”: “post_type”,
“dataType”: [“string”],
“description”: “Type of the post, e.g., report, blog, etc.”
}
]
}

class_Chunk= {
“class”: “Chunk”,
“description”: “Represents a chunk of text from a document.”,
“vectorizer”: “text2vec-openai”,
“moduleConfig”: {
“text2vec-openai”: {
“model”: “text-embedding-3-large”,
“type”: “text”,
“dimensions”:1024
}
},
“properties”: [
{
“name”: “text”,
“dataType”: [“text”],
“description”: “Plain text of the chunk”,
“moduleConfig”: {
“text2vec-openai”: {
“skip”: False
}
}
},
{
“name”: “post_id”,
“dataType”: [“int”],
“description”: “WordPress ID for the post”,
“moduleConfig”: {
“text2vec-openai”: {
“skip”: True
}
}
}
]
}

HI @J_Marcos_del_Ser ! Welcome to our community :hugs:

Does it also happens with the new python client v4?

the v3 client only uses REST, while v4 uses GRPC and that brings a huge improvement in connectivity.

This may be reaching a connection limit

Let me know if this helps!