Panic: unable to load

Support
1

Description

One of the nodes is not working in our cluster. logs from the node are pasted below. Any help is appreciated

Server Setup Information

  • Weaviate Server Version: 1.24.0
  • Deployment Method: docker
  • Multi Node? Number of Running Nodes: 3
  • Client Language and Version: python

Any additional Information

vectorstore_weaviate-weaviate-node-3-1  | {"error":"init shard \"document_82laEsY68dDd\": init shard \"document_82laEsY68dDd\": init per property indices: init properties on shard 'document_82laEsY68dDd': create property 'document_type' null index on shard 'document_82laEsY68dDd': init disk segments: invalid compacted segment file name \"segment-1709258917044357353.db.tmp\"","level":"error","msg":"Unable to load shard 82laEsY68dDd: init shard \"document_82laEsY68dDd\": init shard \"document_82laEsY68dDd\": init per property indices: init properties on shard 'document_82laEsY68dDd': create property 'document_type' null index on shard 'document_82laEsY68dDd': init disk segments: invalid compacted segment file name \"segment-1709258917044357353.db.tmp\"","time":"2024-03-01T14:13:47Z"}
vectorstore_weaviate-weaviate-node-3-1  | panic: Unable to load shard 82laEsY68dDd: init shard "document_82laEsY68dDd": init shard "document_82laEsY68dDd": init per property indices: init properties on shard 'document_82laEsY68dDd': create property 'document_type' null index on shard 'document_82laEsY68dDd': init disk segments: invalid compacted segment file name "segment-1709258917044357353.db.tmp"
vectorstore_weaviate-weaviate-node-3-1  | 
vectorstore_weaviate-weaviate-node-3-1  | 
vectorstore_weaviate-weaviate-node-3-1  | goroutine 226 [running]:
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/adapters/repos/db.(*LazyLoadShard).mustLoadCtx(0x48?, {0x1dfd9e0?, 0x2ae97c0?})
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/shard_lazyloader.go:84 +0x48
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/adapters/repos/db.(*LazyLoadShard).mustLoad(...)
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/shard_lazyloader.go:79
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/adapters/repos/db.(*LazyLoadShard).hasTargetVectors(0xc000169800)
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/shard_lazyloader.go:435 +0x27
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/adapters/repos/db.(*DB).localNodeShardStats.(*Index).getShardsNodeStatus.func1({0xc00392fec0, 0xc}, {0x1e1b6a0, 0xc000169800})
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/nodes.go:180 +0xba
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/adapters/repos/db.(*Index).ForEachShard.(*shardMap).Range.func1({0x1674760?, 0xc00235fdb0?}, {0x19cb120?, 0xc000169800})
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/index.go:79 +0x90
vectorstore_weaviate-weaviate-node-3-1  | sync.(*Map).Range(0xc002b326d0, 0xc0059c9970)
vectorstore_weaviate-weaviate-node-3-1  | 	/usr/local/go/src/sync/map.go:476 +0x228
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/adapters/repos/db.(*shardMap).Range(...)
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/index.go:78
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/adapters/repos/db.(*Index).ForEachShard(0xc0059c9a98?, 0x0?)
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/index.go:386 +0x4e
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/adapters/repos/db.(*Index).getShardsNodeStatus(...)
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/nodes.go:170
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/adapters/repos/db.(*DB).localNodeShardStats(0xc00399a340, {0x1dfd9e0?, 0x2ae97c0}, 0xc0059c9b68, {0x0?, 0xc0028ec030?})
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/nodes.go:134 +0x3bf
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/adapters/repos/db.(*DB).LocalNodeStatus(0xc00399a340, {0x1dfd9e0?, 0x2ae97c0?}, {0x0?, 0x5000105?}, {0x19d901c?, 0xffffffffffffffff?})
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/repos/db/nodes.go:100 +0x108
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/usecases/telemetry.(*Telemeter).getObjectCount(0x2000?, {0x1dfd9e0?, 0x2ae97c0?})
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/usecases/telemetry/telemetry.go:208 +0x3b
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/usecases/telemetry.(*Telemeter).buildPayload(0xc0027b7f80, {0x1dfd9e0, 0x2ae97c0}, {0x19d5641, 0x4})
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/usecases/telemetry/telemetry.go:176 +0x9a
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/usecases/telemetry.(*Telemeter).push(0xc0027b7f80, {0x1dfd9e0?, 0x2ae97c0?}, {0x19d5641?, 0x0?})
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/usecases/telemetry/telemetry.go:144 +0x65
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/usecases/telemetry.(*Telemeter).Start(0xc0027b7f80, {0x1dfd9e0?, 0x2ae97c0})
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/usecases/telemetry/telemetry.go:79 +0x85
vectorstore_weaviate-weaviate-node-3-1  | github.com/weaviate/weaviate/adapters/handlers/rest.configureAPI.func2()
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_api.go:426 +0x32
vectorstore_weaviate-weaviate-node-3-1  | created by github.com/weaviate/weaviate/adapters/handlers/rest.configureAPI in goroutine 1
vectorstore_weaviate-weaviate-node-3-1  | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_api.go:425 +0x19c5
2

Hi @vamsi, this error is caused by a temporary file present in the filesystem before the upgrade. Those files may be removed (suggest to have a backup of them just for in case).

An official release will be done quite soon, I suggest to wait for it.

Best,
Jeronimo

1 Like
3

@jeronimo_irazabal @DudaNogueira

Hi I have upgraded the server to 1.24.1. The problem isn’t resolved in the latest version

I am getting following response for

/v1/objects/Document/{uuid}

repo: object by id: search index document: get remote object: 
shard=82laEsY68dDd: unexpected status code 500 (Unable to load shard 
82laEsY68dDd: init shard \"document_82laEsY68dDd\": init shard 
\"document_82laEsY68dDd\": init per property indices: init properties on shard 
'document_82laEsY68dDd': create property 'document_type' null index on shard 
'document_82laEsY68dDd': init disk segments: invalid compacted segment file
 name \"segment-1709258917044357353.db.tmp\"\n)

why is this happening all the time? Is there anything that can be done?

4

The problem is that there are temporary files e.g. segment-1709258917044357353.db.tmp in the filesystem. Those were generated in a previous release but were not automatically removed.

I suggest to move out of the data folder those temporary files.