Description
i’m seeing some weird behavior with the latest weaviate version running on kubernetes, where cycling pods results in one of the replicas (i have 3) reporting failures (one of them has already been updated and is running, the other has not been terminated/updated yet, and this one erroring is reporting the below). has anyone seen this before?
As far as I can tell, the replica that is failing is not the main replica.
I also observed similar behavior with the simple docker compose setup after docker compose up, then ctrl+c, then docker compose up again.
the last line in the log below about data restoration the only thing that the node is logging, about every 10s.
This did not happen on 1.25.1, and only started happening once I tried to update my cluster to 1.25.7.
Server Setup Information
- Weaviate Server Version: 1.25.7
- Deployment Method: k8s
- Multi Node? Number of Running Nodes: 3
- Client Language and Version: Python, V3
- Multitenancy?: No
Any additional Information
{"action":"graphql_rebuild","error":"Could not build GraphQL schema, because: runtime error: invalid memory address or nil pointer dereference at goroutine 108 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/weaviate/weaviate/adapters/handlers/graphql.buildGraphqlSchema.func1.1()\n\t/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/schema.go:105 +0x3d\npanic({0x19dfea0?, 0x471ccc0?})\n\t/usr/local/go/src/runtime/panic.go:914 +0x21f\ngithub.com/weaviate/weaviate/modules/reranker-transformers.(*ReRankerModule).AdditionalProperties(0xc002a157e0?)\n\t/go/src/github.com/weaviate/weaviate/modules/reranker-transformers/module.go:92 +0x16\ngithub.com/weaviate/weaviate/usecases/modules.(*Provider).GetAdditionalFields(0xc003c1a3c0, 0xc0027dcf20)\n\t/go/src/github.com/weaviate/weaviate/usecases/modules/modules.go:493 +0x204\ngithub.com/weaviate/weaviate/adapters/handlers/graphql/local/get.(*classBuilder).additionalFields(0xc002b77d40, 0xc0030440f0?, 0xc0027dcf20)\n\t/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder.go:169 +0x6df\ngithub.com/weaviate/weaviate/adapters/handlers/graphql/local/get.(*classBuilder).classObject.func1()\n\t/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder.go:143 +0x469\ngithub.com/tailor-inc/graphql.(*Object).Fields(0xc0027e0960)\n\t/go/pkg/mod/github.com/tailor-inc/graphql@v0.2.1/definition.go:452 +0x63\ngithub.com/tailor-inc/graphql.typeMapReducer(0xc002a15df0?, 0xc00302dc80, {0x216c020, 0xc0027e0960?})\n\t/go/pkg/mod/github.com/tailor-inc/graphql@v0.2.1/schema.go:333 +0x405\ngithub.com/tailor-inc/graphql.typeMapReducer(0xc0027e0a50?, 0xc00302dc80, {0x216bfb0, 0xc002b20c60?})\n\t/go/pkg/mod/github.com/tailor-inc/graphql@v0.2.1/schema.go:281 +0xc7\ngithub.com/tailor-inc/graphql.typeMapReducer(0xc0027e0e10?, 0xc00302dc80, {0x216c020, 0xc0027e0a50?})\n\t/go/pkg/mod/github.com/tailor-inc/graphql@v0.2.1/schema.go:344 +0x7f8\ngithub.com/tailor-inc/graphql.typeMapReducer(0xc0027250c0?, 0xc00302dc80, {0x216c020, 0xc0027e0e10?})\n\t/go/pkg/mod/github.com/tailor-inc/graphql@v0.2.1/schema.go:344 +0x7f8\ngithub.com/tailor-inc/graphql.NewSchema({0xc0027e0e10, 0x0, 0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, ...}})\n\t/go/pkg/mod/github.com/tailor-inc/graphql@v0.2.1/schema.go:104 +0x5bf\ngithub.com/weaviate/weaviate/adapters/handlers/graphql.buildGraphqlSchema.func1(0xc002a16918, 0xc002a16938, {{0x1ce2ebe, 0xb}, {0x0, 0x0}, {0x1a01900, 0xc00302dc50}, 0x0, {0x1d047c0, ...}, ...})\n\t/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/schema.go:109 +0xfc\ngithub.com/weaviate/weaviate/adapters/handlers/graphql.buildGraphqlSchema(_, {_, _}, {{0x0, 0x0}, 0x0, {0x64}, 0x186a0, 0x186a0, {{0x0, ...}}, ...}, ...)\n\t/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/schema.go:112 +0x188\ngithub.com/weaviate/weaviate/adapters/handlers/graphql.Build(_, {_, _}, {_, _}, {{0x0, 0x0}, 0x0, {0x64}, 0x186a0, ...}, ...)\n\t/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/schema.go:57 +0x205\ngithub.com/weaviate/weaviate/adapters/handlers/rest.rebuildGraphQL({_}, {_, _}, {{0x0, 0x0}, 0x0, {0x64}, 0x186a0, 0x186a0, {{0x0, ...}}, ...}, ...)\n\t/go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_server.go:70 +0xe5\ngithub.com/weaviate/weaviate/adapters/handlers/rest.MakeAppState.makeUpdateSchemaCall.func6({0xc003c05b00?})\n\t/go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_server.go:52 +0xde\ngithub.com/weaviate/weaviate/usecases/schema.(*executor).TriggerSchemaUpdateCallbacks(0xc003cb56c0)\n\t/go/src/github.com/weaviate/weaviate/usecases/schema/executor.go:242 +0x14f\ngithub.com/weaviate/weaviate/cluster/schema.(*SchemaManager).apply(0xc003c05b30, {{0xc002ce3633, 0xe}, 0xc002b20c40, 0xc002b390e0, 0x1, 0x1})\n\t/go/src/github.com/weaviate/weaviate/cluster/schema/manager.go:360 +0x287\ngithub.com/weaviate/weaviate/cluster/schema.(*SchemaManager).AddClass(0xc003c05b30, 0xc003c037a0, {0xc003b03790, 0xa}, 0x1)\n\t/go/src/github.com/weaviate/weaviate/cluster/schema/manager.go:143 +0x369\ngithub.com/weaviate/weaviate/cluster.(*Store).Apply.func2()\n\t/go/src/github.com/weaviate/weaviate/cluster/store_apply.go:140 +0x37\ngithub.com/weaviate/weaviate/cluster.(*Store).Apply.func12()\n\t/go/src/github.com/weaviate/weaviate/cluster/store_apply.go:205 +0x23\ngithub.com/weaviate/weaviate/entities/errors.GoWrapper.func1()\n\t/go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:34 +0x62\ncreated by github.com/weaviate/weaviate/entities/errors.GoWrapper in goroutine 44\n\t/go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:25 +0x79\n","level":"error","msg":"could not (re)build graphql provider","time":"2024-07-17T16:18:16Z"}
{"level":"error","msg":"Recovered from panic: runtime error: invalid memory address or nil pointer dereference","time":"2024-07-17T16:18:16Z"}
goroutine 109 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
runtime/debug.PrintStack()
/usr/local/go/src/runtime/debug/stack.go:16 +0x13
github.com/weaviate/weaviate/entities/errors.GoWrapper.func1.1()
/go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:30 +0x110
panic({0x19dfea0?, 0x471ccc0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/weaviate/weaviate/modules/text2vec-transformers.(*TransformersModule).Arguments(0xc0029dbc90?)
/go/src/github.com/weaviate/weaviate/modules/text2vec-transformers/nearText.go:26 +0x16
github.com/weaviate/weaviate/usecases/modules.(*Provider).GetArguments(0xc003c1a3c0, 0xc0027dd130)
/go/src/github.com/weaviate/weaviate/usecases/modules/modules.go:356 +0x204
github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.buildGetClassField(0x412ba5?, 0xc0027dd130, {0x2170b20?, 0xc003c1a3c0}, 0xc0029dbf20?)
/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder_fields.go:242 +0x769
github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.(*classBuilder).classField(0xc002d05680, 0xc0027dd130, 0xc003044930?)
/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder.go:102 +0xde
github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.(*classBuilder).kinds(0xc000126a98?, 0xc002d05640)
/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder.go:83 +0x21c
github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.(*classBuilder).objects(...)
/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder.go:65
github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.Build(0xc000126a98, {0x2182d10?, 0xc003af2480?}, {0x2170b20?, 0xc003c1a3c0?})
/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/get.go:47 +0x5d
github.com/weaviate/weaviate/adapters/handlers/graphql/local.Build(_, {_, _}, {{0x0, 0x0}, 0x0, {0x64}, 0x186a0, 0x186a0, {{0x0, ...}}, ...}, ...)
/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/local.go:29 +0x4c
github.com/weaviate/weaviate/adapters/handlers/graphql.buildGraphqlSchema(_, {_, _}, {{0x0, 0x0}, 0x0, {0x64}, 0x186a0, 0x186a0, {{0x0, ...}}, ...}, ...)
/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/schema.go:87 +0x87
github.com/weaviate/weaviate/adapters/handlers/graphql.Build(_, {_, _}, {_, _}, {{0x0, 0x0}, 0x0, {0x64}, 0x186a0, ...}, ...)
/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/schema.go:57 +0x205
github.com/weaviate/weaviate/adapters/handlers/rest.rebuildGraphQL({_}, {_, _}, {{0x0, 0x0}, 0x0, {0x64}, 0x186a0, 0x186a0, {{0x0, ...}}, ...}, ...)
/go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_server.go:70 +0xe5
github.com/weaviate/weaviate/adapters/handlers/rest.MakeAppState.makeUpdateSchemaCall.func6({0xc003c05b00?})
/go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_server.go:52 +0xde
github.com/weaviate/weaviate/usecases/schema.(*executor).TriggerSchemaUpdateCallbacks(0xc003cb56c0)
/go/src/github.com/weaviate/weaviate/usecases/schema/executor.go:242 +0x14f
github.com/weaviate/weaviate/cluster/schema.(*SchemaManager).apply(0xc003c05b30, {{0xc002ce3633, 0xe}, 0xc002b218c0, 0xc002b391a0, 0x1, 0x1})
/go/src/github.com/weaviate/weaviate/cluster/schema/manager.go:360 +0x287
github.com/weaviate/weaviate/cluster/schema.(*SchemaManager).AddClass(0xc003c05b30, 0xc003c039e0, {0xc003b03790, 0xa}, 0x1)
/go/src/github.com/weaviate/weaviate/cluster/schema/manager.go:143 +0x369
github.com/weaviate/weaviate/cluster.(*Store).Apply.func2()
/go/src/github.com/weaviate/weaviate/cluster/store_apply.go:140 +0x37
github.com/weaviate/weaviate/cluster.(*Store).Apply.func12()
/go/src/github.com/weaviate/weaviate/cluster/store_apply.go:205 +0x23
github.com/weaviate/weaviate/entities/errors.GoWrapper.func1()
/go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:34 +0x62
created by github.com/weaviate/weaviate/entities/errors.GoWrapper in goroutine 44
/go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:25 +0x79
{"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-07-17T16:18:16Z"}
{"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-07-17T16:18:16Z"}
{"address":"10.0.4.169:8300","level":"info","msg":"current Leader","time":"2024-07-17T16:18:17Z"}
{"level":"info","msg":"waiting for database to be restored","time":"2024-07-17T16:18:24Z"}
{"level":"info","msg":"waiting for database to be restored","time":"2024-07-17T16:18:34Z"}
