The error occurs when I upgrade from version 1.24.12 to 1.25.1

user3 · May 22, 2024, 9:48am

Error reported when starting:

[+] Running 2/0
Container multi2vec-clip-service Created 0.0s
emoji.discourse-cdn.com/twitter/heavy_check_mark.png?v=12" title=":heavy_check_mark:" class="emoji" alt=":heavy_check_mark:" loading="lazy" width="20" height="20"> Container weaviate-multi2vec Recreated 0.0s
to multi2vec-clip-service, weaviate-multi2vec
| {“level”:“error”,“msg”:“Recovered from panic: runtime error: invalid memory address or nil pointer dereference”,“time”:“2024-05-22T09:34:15Z”}
| goroutine 125 [running]:
| runtime/debug.Stack()
| /usr/local/go/src/runtime/debug/stack.go:24 +0x64
| runtime/debug.PrintStack()
| /usr/local/go/src/runtime/debug/stack.go:16 +0x1c
| github.com/weaviate/weaviate/entities/errors.GoWrapper.func1.1()
| /go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:30 +0x138
| panic({0x13b9fe0?, 0x288f3b0?})
| /usr/local/go/src/runtime/panic.go:914 +0x218
| github.com/weaviate/weaviate/modules/text2vec-openai.(*OpenAIModule).Arguments(0x4002392d18?)
| /go/src/github.com/weaviate/weaviate/modules/text2vec-openai/nearText.go:26 +0x20
| github.com/weaviate/weaviate/usecases/modules.(*Provider).GetArguments(0x4003762a00, 0x40028e2498)
| /go/src/github.com/weaviate/weaviate/usecases/modules/modules.go:356 +0x1b0
| github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.buildGetClassField(0x13a7000?, 0x40028e2498, {0x1b25580?, 0x4003762a00}, 0xffff4ae0e3b0?)
| /go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder_fields.go:242 +0x728
| github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.(*classBuilder).classField(0x40036edd40, 0x40028e2498, 0x4002cea780?)
| /go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder.go:102 +0xb4
| github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.(*classBuilder).kinds(0x40001a36b0?, 0x40036edd00)
| /go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder.go:83 +0x1b8
| github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.(*classBuilder).objects(...)
| /go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder.go:65
| github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.Build(0x40001a36b0, {0x1b383a8?, 0x40036e2700?}, {0x1b25580?, 0x4003762a00?})
| /go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/get.go:47 +0x64
| github.com/weaviate/weaviate/adapters/handlers/graphql/local.Build(, {, }, {{0x0, 0x0}, 0x0, {0x19}, 0x2710, 0x186a0, {{0x0, …}}, …}, …)
| /go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/local.go:29 +0x40
| github.com/weaviate/weaviate/adapters/handlers/graphql.buildGraphqlSchema(, {, }, {{0x0, 0x0}, 0x0, {0x19}, 0x2710, 0x186a0, {{0x0, …}}, …}, …)
| /go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/schema.go:87 +0x60
| github.com/weaviate/weaviate/adapters/handlers/graphql.Build(, {, }, {, }, {{0x0, 0x0}, 0x0, {0x19}, 0x2710, …}, …)
| /go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/schema.go:57 +0x170
| github.com/weaviate/weaviate/adapters/handlers/rest.rebuildGraphQL({}, {_, _}, {{0x0, 0x0}, 0x0, {0x19}, 0x2710, 0x186a0, {{0x0, …}}, …}, …)
| /go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_server.go:70 +0xa8
| github.com/weaviate/weaviate/adapters/handlers/rest.MakeAppState.makeUpdateSchemaCall.func6({0x28f7240?})
| /go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_server.go:52 +0x84
| github.com/weaviate/weaviate/usecases/schema.(*executor).rebuildGQL(0x40037d2000, {{0x40037d2e00, 0xe, 0xe}, {0x0, 0x0}, {0x0, 0x0}})
| /go/src/github.com/weaviate/weaviate/usecases/schema/executor.go:239 +0x13c
| github.com/weaviate/weaviate/usecases/schema.(*executor).ReloadLocalDB(0x40037d2000, {0x1b1fb48, 0x28f7240}, {0x4003393c00, 0xe, 0x59658c?})
| /go/src/github.com/weaviate/weaviate/usecases/schema/executor.go:64 +0x204
| github.com/weaviate/weaviate/cluster/store.(*Store).reloadDBFromSchema(0x400375e120)
| /go/src/github.com/weaviate/weaviate/cluster/store/store.go:944 +0x19c
| github.com/weaviate/weaviate/cluster/store.(*Store).reloadDBFromSnapshot.func1()
| /go/src/github.com/weaviate/weaviate/cluster/store/store.go:916 +0x2c
| github.com/weaviate/weaviate/cluster/store.(*Store).reloadDBFromSnapshot(0x400375e120)
| /go/src/github.com/weaviate/weaviate/cluster/store/store.go:928 +0x294
| github.com/weaviate/weaviate/cluster/store.(*Store).Restore(0x400375e120, {0x1b16c88?, 0x4003748ab0})
| /go/src/github.com/weaviate/weaviate/cluster/store/store.go:767 +0x1e0
| github.com/hashicorp/raft.fsmRestoreAndMeasure({0x1b328a0, 0x4003748a98}, {0x1b1b790, 0x400375e120}, {0x1b16cb0?, 0x40037586a0}, 0x10?)
| /go/pkg/mod/github.com/hashicorp/raft@v1.5.0/fsm.go:271 +0x178
| github.com/hashicorp/raft.(*Raft).tryRestoreSingleSnapshot(0x4003416840, 0x40036e2480)
| /go/pkg/mod/github.com/hashicorp/raft@v1.5.0/api.go:679 +0x3c4
| github.com/hashicorp/raft.(*Raft).restoreSnapshot(0x4003416840)
| /go/pkg/mod/github.com/hashicorp/raft@v1.5.0/api.go:622 +0x11c
| github.com/hashicorp/raft.NewRaft(0x40024700a0, {0x1b1b790?, 0x400375e120}, {0x1b24c30?, 0x400370c380}, {0x1b1ff00?, 0x4003748318}, {0x1b1b7c0?, 0x40037fedb0}, {0x1b2b3a8?, …})
| /go/pkg/mod/github.com/hashicorp/raft@v1.5.0/api.go:575 +0x7dc
| github.com/weaviate/weaviate/cluster/store.(*Store).Open(0x400375e120, {0x1b1fb48, 0x28f7240})
| /go/src/github.com/weaviate/weaviate/cluster/store/store.go:250 +0x314
| github.com/weaviate/weaviate/cluster/store.(*Service).Open(0x40037be0a0, {0x1b1fb48, 0x28f7240}, {0x1b2ddd0?, 0x40037d2000})
| /go/src/github.com/weaviate/weaviate/cluster/store/service.go:58 +0x104
| github.com/weaviate/weaviate/cluster.(*Service).Open(0x40000c0050, {0x1b1fb48, 0x28f7240}, {0x1b2ddd0, 0x40037d2000})
| /go/src/github.com/weaviate/weaviate/cluster/cluster.go:67 +0x110
| github.com/weaviate/weaviate/adapters/handlers/rest.MakeAppState.func3()
| /go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_api.go:340 +0x4c
| github.com/weaviate/weaviate/entities/errors.GoWrapper.func1()
| /go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:34 +0x64
| created by github.com/weaviate/weaviate/entities/errors.GoWrapper in goroutine 1
| /go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:25 +0x80
| INFO: Started server process [7]
| INFO: Waiting for application startup.
| INFO: Running on CPU
| INFO: Model initialization complete
| INFO: Application startup complete.
| INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
| INFO: 192.168.247.2:33270 - “GET /.well-known/ready HTTP/1.1” 204 No Content
| INFO: 192.168.247.2:33270 - “GET /meta HTTP/1.1” 200 OK
| INFO: 192.168.247.2:48108 - “GET /meta HTTP/1.1” 200 OK

When creating a collection, the error is reported as follows:
panic: status code: 422, error: {“error”:[{“message”:“leader not found”}]}
docker-compose.yml

version: ‘3.4’
services:
multi2vec-clip:
image: semitechnologies/multi2vec-clip:sentence-transformers-clip-ViT-B-32-multilingual-v1
container_name: multi2vec-clip-service
weaviate:
command:
- --host
- 0.0.0.0
- --port
- ‘8070’
- --scheme
- http
image: semitechnologies/weaviate:1.25.1
container_name: weaviate-multi2vec
ports:
- 8070:8070
volumes:
- $HOME/weaviate/weaviate_data:/var/lib/weaviate
- $HOME/weaviate/backup:/var/lib/weaviate_backup
restart: on-failure:0
environment:
LOG_LEVEL: error
CLIP_INFERENCE_API: ‘http://multi2vec-clip-service:8080’
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: ‘true’
PERSISTENCE_DATA_PATH: ‘/var/lib/weaviate’
# DEFAULT_VECTORIZER_MODULE: ‘text2vec-openai’
ENABLE_MODULES: ‘text2vec-aws,text2vec-palm,text2vec-openai,backup-filesystem,multi2vec-clip,generative-openai’
BACKUP_FILESYSTEM_PATH: ‘/var/lib/weaviate_backup’
CLUSTER_HOSTNAME: node1

DudaNogueira · May 22, 2024, 10:07am

Hi!

I assume this is a multi node setup.

On that case, be aware that there are some new env variables to define.

Check here the guide for K8S:

And here for the environment variables for multi node:

Let me know if this helps!

Thanks!

user3 · May 22, 2024, 1:24pm

It looks like this is a bug in version 1.25.1
It has been fixed one

antas-marcin · May 22, 2024, 1:44pm

@user3 I think that this thread contains a solution to your problem, especially check here my response.

user3 · May 23, 2024, 2:52am

The error is gone .thanks

Topic		Replies	Views
Fatal error: Panic: runtime error: invalid memory address or nil pointer dereference Support	2	243	March 20, 2024
GraphQL rebuild error after rolling one or more replicas Support	0	8	July 17, 2024
Weaviate crashing because of following error, can someone help on this Support	6	246	July 24, 2024
Weaviate Cluster OOMs and Recovery Support	4	472	June 14, 2023
Panic: index out of range [-1] runtime errors in log Support	4	489	March 20, 2024

The error occurs when I upgrade from version 1.24.12 to 1.25.1

Related Topics