Fatal error: Panic: runtime error: invalid memory address or nil pointer dereference

Support
1

Description

Hello Team,

We are now having a fatal error which makes our Weaviate down and restart. The log level is error but it actually led Weaviate unhealthy.

Mar 14 16:39:48 weaviate-rerank-second-0 weaviate error panic: runtime error: invalid memory address or nil pointer dereference
Mar 14 16:39:48 weaviate-rerank-second-0 weaviate [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xa75c83]
Mar 14 16:39:48 weaviate-rerank-second-0 weaviate goroutine 5396921 [running]:
Mar 14 16:39:48 weaviate-rerank-second-0 weaviate github.com/weaviate/weaviate/entities/schema.GetClassByName(0x7f5ba17d11c8?, {0xc5714a4380, 0x1a})
	/go/src/github.com/weaviate/weaviate/entities/schema/backward_compat.go:30 +0xa3
Mar 14 16:39:48 weaviate-rerank-second-0 weaviate github.com/weaviate/weaviate/entities/schema.(*Schema).FindClassByName(...)
	/go/src/github.com/weaviate/weaviate/entities/schema/accessors.go:29
Mar 14 16:39:48 weaviate-rerank-second-0 weaviate github.com/weaviate/weaviate/usecases/schema.(*Manager).getClassByName(0x4d4c37?, {0xc5714a4380?, 0xbd8b1d?})
	/go/src/github.com/weaviate/weaviate/usecases/schema/get.go:77 +0x25
Mar 14 16:39:48 weaviate-rerank-second-0 weaviate github.com/weaviate/weaviate/usecases/schema.(*Manager).GetClass(0xc000501800, {0xc2fe8fbef0?, 0x184322d?}, 0x12?, {0xc5714a4380, 0x1a})
	/go/src/github.com/weaviate/weaviate/usecases/schema/get.go:69 +0x7f
Mar 14 16:39:48 weaviate-rerank-second-0 weaviate github.com/weaviate/weaviate/usecases/objects.(*BatchManager).validateObject(0xc003cd8280, {0x1c0b288, 0xc2fe8fb950}, 0x6?, 0xc21be84a80, 0x15, 0xc4e37b08b0, 0xb6d106?, 0x0)
	/go/src/github.com/weaviate/weaviate/usecases/objects/batch_add.go:162 +0x61d
Mar 14 16:39:48 weaviate-rerank-second-0 weaviate github.com/weaviate/weaviate/usecases/objects.(*BatchManager).validateObjectsConcurrently.func1()
	/go/src/github.com/weaviate/weaviate/usecases/objects/batch_add.go:105 +0x3d
Mar 14 16:39:48 weaviate-rerank-second-0 weaviate golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64
Mar 14 16:39:48 weaviate-rerank-second-0 weaviate created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa5
Mar 14 16:39:52 weaviate-rerank-second-0 weaviate info {"action":"config_load","config_file_path":"/weaviate-config/conf.yaml","level":"info","msg":"Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.","time":"2024-03-14T08:39:52Z"}
Mar 14 16:39:52 weaviate-rerank-second-0 weaviate warning {"deprecation":{"apiType":"Configuration","id":"config-files","locations":["--config-file=\"\""],"mitigation":"Configure Weaviate using environment variables.","msg":"use of deprecated command line argument --config-file","sinceTime":"2020-09-08T09:46:00.000Z","sinceVersion":"0.22.16","status":"deprecated"},"level":"warning","msg":"use of deprecated command line argument --config-file","time":"2024-03-14T08:39:52Z"}
Mar 14 16:39:52 weaviate-rerank-second-0 weaviate info {"action":"startup","default_vectorizer_module":"text2vec-huggingface","level":"info","msg":"the default vectorizer modules is set to \"text2vec-huggingface\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-03-14T08:39:52Z"}
Mar 14 16:39:52 weaviate-rerank-second-0 weaviate info {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-03-14T08:39:52Z"}

We cannot actually reproduce it, but we observe the same behavior patterns have led to this fatal error:

Our service on pod1 is batch ingesting data objects for class A, while pod2 is batch deleting data objects for class B. Right after the deletion, Weaviate encounters an error and restarts, causing both the ingestion and deletion processes to fail.

Server Setup Information

  • Weaviate Server Version: 1.20
  • Deployment Method: k8s
  • Standalone
  • Client Language and Version: Python 3.11
2

Hi!

This may be related to this issue:

So my first advice would be ingesting your dataset on a Weaviate latest version (1.24.2), and make sure that this still happens.

1.20 is a fairly old version already. A lot was fixed since.

Let me know if this helps!

3

Thank you @DudaNogueira , we will first upgrade to 1.24 and see if it is still there.

1 Like