Error when upgrading to Weaviate 1.25.1

evenfrost · May 20, 2024, 11:34am

Description

When upgrading Weaviate from 1.24.12 to 1.25.1 we started getting this error:

Error: usage error (403): {"error":[{"message":"could not read schema with strong consistency: failed to execute query: leader not found"}]}

I’m also having some errors in the Docker Weaviate logs:

weaviate_1          | {"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-20T11:24:54Z","took":322041}
weaviate_1          | {"level":"info","msg":"Completed loading shard zendeskmacro_b6vs7sSm0XLH in 18.477878ms","time":"2024-05-20T11:24:54Z"}
weaviate_1          | {"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-20T11:24:54Z","took":252341}
weaviate_1          | {"action":"","id":"2-3-1715697088738","last-index":3,"last-term":2,"level":"info","msg":"snapshot restore progress","percent-complete":"[100.00]%","read-bytes":201722,"size-in-bytes":201722,"time":"2024-05-20T11:24:54Z"}
weaviate_1          | {"level":"error","msg":"Recovered from panic: runtime error: invalid memory address or nil pointer dereference","time":"2024-05-20T11:24:54Z"}
weaviate_1          | goroutine 136 [running]:
weaviate_1          | runtime/debug.Stack()
weaviate_1          | 	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
weaviate_1          | runtime/debug.PrintStack()
weaviate_1          | 	/usr/local/go/src/runtime/debug/stack.go:16 +0x13
weaviate_1          | github.com/weaviate/weaviate/entities/errors.GoWrapper.func1.1()
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:30 +0x110
weaviate_1          | panic({0x1981580?, 0x2eba4f0?})
weaviate_1          | 	/usr/local/go/src/runtime/panic.go:914 +0x21f

I’m using the standard Docker image with the following docker-compose.yml:

---
version: '3.4'
services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: semitechnologies/weaviate:1.25.1
    ports:
    - ${WEAVIATE_PORT}:8080
    restart: always
    environment:
      CONTEXTIONARY_URL: contextionary:9999
      QNA_INFERENCE_API: 'http://qna-transformers:8080'
      NER_INFERENCE_API: 'http://ner-transformers:8080'
      SPELLCHECK_INFERENCE_API: 'http://text-spellcheck:8080'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-contextionary'
      ENABLE_MODULES: 'text2vec-contextionary,qna-transformers,ner-transformers,text-spellcheck,generative-openai,text2vec-openai'
      CLUSTER_HOSTNAME: 'node1'
      AUTOSCHEMA_ENABLED: 'false'
      QUERY_MAXIMUM_RESULTS: 50000
    volumes:
      - /var/weaviate:/var/lib/weaviate
  contextionary:
    restart: always
    environment:
      OCCURRENCE_WEIGHT_LINEAR_FACTOR: 0.75
      EXTENSIONS_STORAGE_MODE: weaviate
      EXTENSIONS_STORAGE_ORIGIN: http://weaviate:8080
      NEIGHBOR_OCCURRENCE_IGNORE_PERCENTILE: 5
      ENABLE_COMPOUND_SPLITTING: 'false'
    image: semitechnologies/contextionary:en0.16.0-v1.2.1
    ports:
    - 9999:9999
  qna-transformers:
    restart: always
    image: semitechnologies/qna-transformers:distilbert-base-uncased-distilled-squad
    environment:
      ENABLE_CUDA: '0'
  ner-transformers:
    restart: always
    image: semitechnologies/ner-transformers:dbmdz-bert-large-cased-finetuned-conll03-english
    environment:
      ENABLE_CUDA: '0'
  text-spellcheck:
    restart: always
    image: semitechnologies/text-spellcheck-model:pyspellchecker-en
...

When reverting to 1.24.12, the error goes away.

Server Setup Information

Weaviate Server Version: 1.25.1
Deployment Method: docker
Multi Node? not sure
Client Language and Version: weaviate-ts-client@2.1.1

Any additional Information

antas-marcin · May 20, 2024, 1:03pm

Are you are running Weaviate in a single-node setup, right? You don’t scale your docker compose?

If you are running Weaviate in a single node configuration try adding those environment variables to your configuration when running it with v1.25

RAFT_BOOTSTRAP_EXPECT: "1"
RAFT_JOIN: "node1:8300"

evenfrost · May 20, 2024, 1:18pm

@antas-marcin added the variables as follows, got the same error:

services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: semitechnologies/weaviate:1.25.1
    ports:
    - ${WEAVIATE_PORT}:8080
    restart: always
    environment:
      CONTEXTIONARY_URL: contextionary:9999
      QNA_INFERENCE_API: 'http://qna-transformers:8080'
      NER_INFERENCE_API: 'http://ner-transformers:8080'
      SPELLCHECK_INFERENCE_API: 'http://text-spellcheck:8080'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-contextionary'
      ENABLE_MODULES: 'text2vec-contextionary,qna-transformers,ner-transformers,text-spellcheck,generative-openai,text2vec-openai'
      CLUSTER_HOSTNAME: 'node1'
      AUTOSCHEMA_ENABLED: 'false'
      QUERY_MAXIMUM_RESULTS: 50000
      RAFT_BOOTSTRAP_EXPECT: '1'
      RAFT_JOIN: 'node1:8300'

evenfrost · May 20, 2024, 1:22pm

Also, initially I was upgrading from v1.22.6, but the error happens only on v1.25.x, and v1.24.x works just fine.

antas-marcin · May 21, 2024, 12:43pm

I have copied your docker compose file and first run it with v1.24.14, populated data, then I changed Weaviate version to v1.25.1 (without adding the RAFT_JOIN and RAFT_BOOTSTRAP_EXPECT env variables) and all worked fine.

Please note that with v1.25 Weaviate introduces new V2 schema and it takes some time to migrate to v1.25 version (more here)

One thing I noticed that you can also expose gRPC port in your environment, that way you will notice a significant performance boost.

evenfrost · May 21, 2024, 1:36pm

I think the main error is found in the Docker logs:

weaviate_1          | {"level":"info","msg":"Completed loading shard entityhcarticle_0dd238d5-b9e0-4e50-a859-62b0adc62c33 in 1.410112ms","time":"2024-05-21T13:33:31Z"}
weaviate_1          | {"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-21T13:33:31Z","took":103835}
weaviate_1          | {"level":"info","msg":"Completed loading shard entityhcarticle_5b84382c-966f-4501-a6de-fa605db3c1bb in 1.286014ms","time":"2024-05-21T13:33:31Z"}
weaviate_1          | {"index":"entityMacro","level":"info","msg":"restore local index","time":"2024-05-21T13:33:31Z"}
weaviate_1          | {"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-21T13:33:31Z","took":100400}
weaviate_1          | {"level":"info","msg":"Completed loading shard entitymacro_b6vs7sSm0XLH in 4.975246ms","time":"2024-05-21T13:33:31Z"}
weaviate_1          | {"index":"entityShortcutMulti","level":"info","msg":"restore local index","time":"2024-05-21T13:33:31Z"}
weaviate_1          | {"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-21T13:33:31Z","took":92548}
weaviate_1          | {"level":"info","msg":"Completed loading shard entityshortcutmulti_fc1e6dde-ca3a-4608-912a-3825c2a95f82 in 8.19134ms","time":"2024-05-21T13:33:31Z"}
weaviate_1          | {"index":"entityTicket","level":"info","msg":"restore local index","time":"2024-05-21T13:33:31Z"}
weaviate_1          | {"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-21T13:33:31Z","took":128149}
weaviate_1          | {"level":"info","msg":"Completed loading shard entityticket_0dd238d5-b9e0-4e50-a859-62b0adc62c33 in 4.140086ms","time":"2024-05-21T13:33:32Z"}
weaviate_1          | {"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-21T13:33:32Z","took":296914}
weaviate_1          | {"level":"info","msg":"Completed loading shard entityticket_5b84382c-966f-4501-a6de-fa605db3c1bb in 4.678586ms","time":"2024-05-21T13:33:32Z"}
weaviate_1          | {"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-21T13:33:32Z","took":98953}
weaviate_1          | {"action":"","id":"2-3-1715697088738","last-index":3,"last-term":2,"level":"info","msg":"snapshot restore progress","percent-complete":"[100.00]%","read-bytes":201722,"size-in-bytes":201722,"time":"2024-05-21T13:33:32Z"}
weaviate_1          | {"level":"error","msg":"Recovered from panic: runtime error: invalid memory address or nil pointer dereference","time":"2024-05-21T13:33:32Z"}
weaviate_1          | goroutine 122 [running]:
weaviate_1          | runtime/debug.Stack()
weaviate_1          | 	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
weaviate_1          | runtime/debug.PrintStack()
weaviate_1          | 	/usr/local/go/src/runtime/debug/stack.go:16 +0x13
weaviate_1          | github.com/weaviate/weaviate/entities/errors.GoWrapper.func1.1()
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:30 +0x110
weaviate_1          | panic({0x1981580?, 0x2eba4f0?})
weaviate_1          | 	/usr/local/go/src/runtime/panic.go:914 +0x21f
weaviate_1          | github.com/weaviate/weaviate/modules/text2vec-contextionary.(*ContextionaryModule).Arguments(0xc00258ce10?)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/modules/text2vec-contextionary/module.go:247 +0x16
weaviate_1          | github.com/weaviate/weaviate/usecases/modules.(*Provider).GetArguments(0xc002b54c30, 0xc002903998)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/usecases/modules/modules.go:356 +0x204
weaviate_1          | github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.buildGetClassField(0x412b85?, 0xc002903998, {0x20f2d80?, 0xc002b54c30}, 0xc00258d0a0?)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder_fields.go:242 +0x769
weaviate_1          | github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.(*classBuilder).classField(0xc013d705c0, 0xc002903998, 0xc013d41e30?)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder.go:102 +0xde
weaviate_1          | github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.(*classBuilder).kinds(0xc013d1a3c8?, 0xc013d70580)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder.go:83 +0x21c
weaviate_1          | github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.(*classBuilder).objects(...)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/class_builder.go:65
weaviate_1          | github.com/weaviate/weaviate/adapters/handlers/graphql/local/get.Build(0xc013d1a3c8, {0x2105e10?, 0xc0037ac500?}, {0x20f2d80?, 0xc002b54c30?})
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/get/get.go:47 +0x5d
weaviate_1          | github.com/weaviate/weaviate/adapters/handlers/graphql/local.Build(_, {_, _}, {{0x0, 0x0}, 0x0, {0x19}, 0xc350, 0x186a0, {{0xc00005a012, ...}}, ...}, ...)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/local/local.go:29 +0x4c
weaviate_1          | github.com/weaviate/weaviate/adapters/handlers/graphql.buildGraphqlSchema(_, {_, _}, {{0x0, 0x0}, 0x0, {0x19}, 0xc350, 0x186a0, {{0xc00005a012, ...}}, ...}, ...)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/schema.go:87 +0x7b
weaviate_1          | github.com/weaviate/weaviate/adapters/handlers/graphql.Build(_, {_, _}, {_, _}, {{0x0, 0x0}, 0x0, {0x19}, 0xc350, ...}, ...)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/graphql/schema.go:57 +0x1f8
weaviate_1          | github.com/weaviate/weaviate/adapters/handlers/rest.rebuildGraphQL({_}, {_, _}, {{0x0, 0x0}, 0x0, {0x19}, 0xc350, 0x186a0, {{0xc00005a012, ...}}, ...}, ...)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_server.go:70 +0xdf
weaviate_1          | github.com/weaviate/weaviate/adapters/handlers/rest.MakeAppState.makeUpdateSchemaCall.func6({0x2f1c2c0?})
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_server.go:52 +0xa5
weaviate_1          | github.com/weaviate/weaviate/usecases/schema.(*executor).rebuildGQL(0xc003878000, {{0xc00348b260, 0xc, 0xc}, {0x0, 0x0}, {0x0, 0x0}})
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/usecases/schema/executor.go:239 +0x133
weaviate_1          | github.com/weaviate/weaviate/usecases/schema.(*executor).ReloadLocalDB(0xc003878000, {0x20ed490, 0x2f1c2c0}, {0xc000142f00, 0xc, 0xc003898690?})
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/usecases/schema/executor.go:64 +0x291
weaviate_1          | github.com/weaviate/weaviate/cluster/store.(*Store).reloadDBFromSchema(0xc003800000)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/cluster/store/store.go:944 +0x1b2
weaviate_1          | github.com/weaviate/weaviate/cluster/store.(*Store).reloadDBFromSnapshot.func1()
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/cluster/store/store.go:916 +0x25
weaviate_1          | github.com/weaviate/weaviate/cluster/store.(*Store).reloadDBFromSnapshot(0xc0037ac500?)
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/cluster/store/store.go:928 +0x298
weaviate_1          | github.com/weaviate/weaviate/cluster/store.(*Store).Restore(0xc003800000, {0x20e48a8?, 0xc003746900})
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/cluster/store/store.go:767 +0x299
weaviate_1          | github.com/hashicorp/raft.fsmRestoreAndMeasure({0x2100100, 0xc0037468d0}, {0x20e9298, 0xc003800000}, {0x20e48d0?, 0xc00386eaa0}, 0xc002280400?)
weaviate_1          | 	/go/pkg/mod/github.com/hashicorp/raft@v1.5.0/fsm.go:271 +0x1c5
weaviate_1          | github.com/hashicorp/raft.(*Raft).tryRestoreSingleSnapshot(0xc0034d8000, 0xc0037ac600)
weaviate_1          | 	/go/pkg/mod/github.com/hashicorp/raft@v1.5.0/api.go:679 +0x3f8
weaviate_1          | github.com/hashicorp/raft.(*Raft).restoreSnapshot(0xc0034d8000)
weaviate_1          | 	/go/pkg/mod/github.com/hashicorp/raft@v1.5.0/api.go:622 +0x125
weaviate_1          | github.com/hashicorp/raft.NewRaft(0xc0028101e0, {0x20e9298?, 0xc003800000}, {0x20f2470?, 0xc003720440}, {0x20ed810?, 0xc003746258}, {0x20e92c8?, 0xc003898db0}, {0x20f8b88, ...})
weaviate_1          | 	/go/pkg/mod/github.com/hashicorp/raft@v1.5.0/api.go:575 +0xa05
weaviate_1          | github.com/weaviate/weaviate/cluster/store.(*Store).Open(0xc003800000, {0x20ed490, 0x2f1c2c0})
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/cluster/store/store.go:250 +0x425
weaviate_1          | github.com/weaviate/weaviate/cluster/store.(*Service).Open(0xc00385e0c0, {0x20ed490, 0x2f1c2c0}, {0x20fb630?, 0xc003878000})
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/cluster/store/service.go:58 +0x10a
weaviate_1          | github.com/weaviate/weaviate/cluster.(*Service).Open(0xc002b54050, {0x20ed490, 0x2f1c2c0}, {0x20fb630, 0xc003878000})
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/cluster/cluster.go:67 +0x167
weaviate_1          | github.com/weaviate/weaviate/adapters/handlers/rest.MakeAppState.func3()
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/adapters/handlers/rest/configure_api.go:340 +0x4b
weaviate_1          | github.com/weaviate/weaviate/entities/errors.GoWrapper.func1()
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:34 +0x62
weaviate_1          | created by github.com/weaviate/weaviate/entities/errors.GoWrapper in goroutine 1
weaviate_1          | 	/go/src/github.com/weaviate/weaviate/entities/errors/go_wrapper.go:25 +0x79

After this point, all requests to Weaviate fail with the same error:

msg:search index entityarticle code:500 err:failed to execute query: leader not found

evenfrost · May 21, 2024, 1:38pm

@antas-marcin as for exposing the gRPC port, is there any particular reason why there would be a performance boost? AFAIK, this setting only remaps the service from the default port to another one.

antas-marcin · May 21, 2024, 2:46pm

This setting exposes our gRPC port along side with HTTP port that you already exposed by you.

gRPC is a high-performance, binary protocol it greatly improves inserts and queries times.

evenfrost · May 21, 2024, 3:01pm

Yes, but this setting only exposes the port to the host machine and nothing else.

UPD: nevermind, I see that the connection seems to be handled by the client library under the hood.

antas-marcin · May 22, 2024, 11:37am

@evenfrost I think that your error looks very similar to this one that we fixed recently.

If you want to try it I’m pasting you here a preview image built on top of v1.25.1 that contains this fix:

semitechnologies/weaviate:preview--d629956

You can try it out and check if it helps you.

evenfrost · May 22, 2024, 12:42pm

@antas-marcin yes, the error is gone in this version, thanks. The fix should in v1.25.2 then?

antas-marcin · May 22, 2024, 1:04pm

Yes, it will be included in v1.25.2.

So now you are able to run your setup with Weaviate v1.25, yes? all is working fine, right?

evenfrost · May 22, 2024, 1:15pm

Yeah, the error is gone and I’m able to perform queries towards Weaviate as usual. Thanks again for the help.

Topic		Replies	Views
Unable to restart my weaviate container Support	8	1193	May 20, 2024
Data unavailable after migration of weaviate version Support	1	167	March 25, 2024
Weaviate error "transferring leadership" on single node cluster Support integration , technical	12	156	December 21, 2024
Panic after migration to 1.25.1 Support bug	3	207	June 3, 2024
Weaviate.connect_to_embedded broken in 4.7.1 Support	2	90	September 20, 2024

Error when upgrading to Weaviate 1.25.1

Description

Server Setup Information

Any additional Information

Related topics