# Panic when using GSE tokenizer with replication (nil Segmenter, EOF on /replicas/…:commit) in v1.34.10 **URL:** \ **Category:** General\ **Created:** [January 20, 2026, 9:45am UTC](https://forum.weaviate.io/t/panic-when-using-gse-tokenizer-with-replication-nil-segmenter-eof-on-replicas-commit-in-v1-34-10/22222 "2026-01-20T09:45:12Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![longfei\_yao](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/longfei_yao/32/2329_2.png) [@longfei\_yao](https://forum.weaviate.io/u/longfei_yao)\ **Post date:** [January 20, 2026, 9:45am UTC](https://forum.weaviate.io/t/panic-when-using-gse-tokenizer-with-replication-nil-segmenter-eof-on-replicas-commit-in-v1-34-10/22222/1 "2026-01-20T09:45:12Z")
build\_wv\_version: 1.34.10 build\_image\_tag: v1.34.10 build\_go\_version: go1.24.11 Docker image: semitechnologies/weaviate:latest (resolves to 1.34.10) * * * ### Deployment - 3‑node Docker Compose cluster (multi‑node with replication) - Example node environment (simplified): ```yaml environment: RAFT_JOIN: node1,node2,node3 DEFAULT_VECTORIZER_MODULE: text2vec-ollama LOG_LEVEL: debug ENABLE_TOKENIZER_GSE: 'true' volumes: - ./data-node-3:/home/vector/lib/weaviate ports: - "8082:8080" - "6062:6060" - "50015:50051" ``` - Internal Docker network: `192.168.16.0/20` - Internal data ports (cluster‑internal API) like `7103`, `7105`, `7111` are used for `/replicas/...:commit` calls. * * * ### Schema / collection definition Using Python client v4 to create a collection `Test_media` with: - Replication factor: 3 - Multiple named vectors (text2vec‑ollama) - Several Chinese text fields using **GSE** tokenization ```python from weaviate.classes.config import ( Configure, Property, DataType, Tokenization, VectorDistances, VectorFilterStrategy, ) client.collections.create( "Test_media", replication_config=Configure.replication( factor=3, ), vector_config=[ Configure.Vectors.text2vec_ollama( name="title", api_endpoint=ollama_api_endpoint, model="quentinz/bge-large-zh-v1.5:latest", source_properties=["title"], quantizer=Configure.VectorIndex.Quantizer.rq(), vector_index_config=Configure.VectorIndex.hnsw( ef_construction=300, distance_metric=VectorDistances.COSINE, filter_strategy=VectorFilterStrategy.ACORN, ), ), ], properties=[ Property(name="title", data_type=DataType.TEXT, tokenization=Tokenization.GSE), Property(name="location", data_type=DataType.TEXT_ARRAY, skip_vectorization=True, tokenization=Tokenization.FIELD), Property(name="p_id", data_type=DataType.TEXT, skip_vectorization=True), ], ) ``` GSE is used as the tokenizer for several Chinese text fields. GSE is documented as a language‑specific tokenizer for Japanese/Chinese, enabled via `ENABLE_TOKENIZER_GSE`. [[Language-specific tokenization](https://docs.weaviate.io/weaviate/config-refs/collections#language-specific-tokenization)] * * * ### Steps to reproduce 1. Start a 3‑node Docker Compose Weaviate cluster with replication and `ENABLE_TOKENIZER_GSE=true`. 2. Create the `Test_media` collection as above. 3. Insert objects via `POST /v1/objects?consistency_level=ONE` (or Python client v4), where `title`, `content`, `summary`, `catalog` contain Chinese text. 4. During import, observe: - 500 errors on the client; - EOF errors on internal `/replicas/...:commit` calls; - panics in the logs on the replica nodes. * * * ### Observed behavior **Client‑side error example** ```plaintext Object was not added! Unexpected status code: 500, with response body: { "error": [{ "message": "put object: import into index test_media: replicate insertion: shard=\"d2X2cUCHZ3WK\": 192.168.16.4:7111: connect: Post \"http://192.168.16.4:7111/replicas/indices/Test_media/shards/d2X2cUCHZ3WK:commit?request_id=node1-01-19bdaba28e0-28\": EOF" }] } ``` Similar errors appear for other shards and nodes, always on internal `/replicas/indices/Test_media/shards/:commit` calls to the cluster‑internal data port (e.g. `7111`). [[Multi-node config](https://docs.weaviate.io/deploy/installation-guides/docker-installation#multi-node-configuration)] **Server‑side logs around the error** ```json {"class":"Test_media","level":"error", "msg":"192.168.16.4:7111: connect: Post \"http://192.168.16.4:7111/replicas/indices/Test_media/shards/d2X2cUCHZ3WK:commit?request_id=node1-01-19bdaba28e0-28\": EOF", "op":"put","shard":"d2X2cUCHZ3WK"} {"action":"requests_total","api":"rest","class_name":"Test_media","level":"error", "error":"put object: import into index test_media: replicate insertion: shard=\"d2X2cUCHZ3WK\": 192.168.16.4:7111: connect: Post \"http://192.168.16.4:7111/replicas/indices/Test_media/shards/d2X2cUCHZ3WK:commit?request_id=node1-01-19bdaba28e0-28\": EOF", "msg":"unexpected error","query_type":"objects"} ``` **Panic stack trace on the replica node** ```plaintext 2026/01/20 09:26:32 http: panic serving 192.168.16.4:51360: runtime error: invalid memory address or nil pointer dereference goroutine 471281 [running]: net/http.(*conn).serve.func1() net/http/server.go:1947 +0xbe panic({0x2e39640?, 0x7dd97c0?}) runtime/panic.go:792 +0x132 github.com/go-ego/gse.(*Segmenter).Find(...) github.com/go-ego/gse@v0.80.3/dag.go:40 github.com/go-ego/gse.(*Segmenter).getDag(0x0, {0xc002741d10, 0x17, 0x20}) github.com/go-ego/gse@v0.80.3/dag.go:142 +0x1f1 github.com/go-ego/gse.(*Segmenter).cutAll(0x0, {0xc0017eab70, 0x2f}) github.com/go-ego/gse@v0.80.3/dag.go:318 +0xd5 github.com/go-ego/gse.(*Segmenter).CutAll(...) github.com/go-ego/gse@v0.80.3/gse.go:92 github.com/weaviate/weaviate/entities/tokenizer.tokenizeGSE({0xc0017eab70, 0x2f}) github.com/weaviate/weaviate/entities/tokenizer/tokenizer.go:278 +0xea github.com/weaviate/weaviate/entities/tokenizer.Tokenize({0xc001104b69?, 0x49c90e0?}, {0xc0017eab70, 0x2f}) github.com/weaviate/weaviate/entities/tokenizer/tokenizer.go:163 +0xbe github.com/weaviate/weaviate/entities/tokenizer.TokenizeForClass({0xc001104b69, 0x3}, {0xc0017eab70, 0x2f}, {0xc0013c42e0?, 0x6?}) github.com/weaviate/weaviate/entities/tokenizer/tokenizer.go:144 +0x24e github.com/weaviate/weaviate/adapters/repos/db/inverted.(*Analyzer).TextArray(0xc0027428d0, {0xc001104b69, 0x3}, {0xc002742480?, ...}) github.com/weaviate/weaviate/adapters/repos/db/inverted/analyzer.go:81 +0xfd github.com/weaviate/weaviate/adapters/repos/db/inverted.(*Analyzer).Text(...) github.com/weaviate/weaviate/adapters/repos/db/inverted.(*Analyzer).analyzePrimitiveProp(...) github.com/weaviate/weaviate/adapters/repos/db/inverted.(*Analyzer).Object(...) github.com/weaviate/weaviate/adapters/repos/db.(*Shard).AnalyzeObject(...) github.com/weaviate/weaviate/adapters/repos/db.(*Shard).updateInvertedIndexLSM(...) github.com/weaviate/weaviate/adapters/repos/db.(*Shard).putObjectLSM(...) github.com/weaviate/weaviate/adapters/repos/db.(*Shard).putOne(...) github.com/weaviate/weaviate/adapters/repos/db.(*Shard).preparePutObject.func1(...) github.com/weaviate/weaviate/adapters/repos/db.(*Shard).commitReplication(...) github.com/weaviate/weaviate/adapters/repos/db.(*LazyLoadShard).commitReplication(...) github.com/weaviate/weaviate/adapters/repos/db.(*Index).CommitReplication(...) github.com/weaviate/weaviate/usecases/replica.(*RemoteReplicaIncoming).CommitReplication(...) github.com/weaviate/weaviate/adapters/handlers/rest/clusterapi.(*replicatedIndices).executeCommitPhase.func12(...) ... net/http.(*conn).serve(...) ``` Key points from the stack trace: - `github.com/go-ego/gse.(*Segmenter).getDag(0x0, ...)` → `Segmenter` is `nil`. - Called via `tokenizeGSE` → `TokenizeForClass` → inverted index analyzer → `Shard.commitReplication`. - The panic happens while handling the internal `/replicas/...:commit` request, which explains the EOF seen by the coordinator node. * * * ### Expected behavior - When `tokenization=GSE` is configured on TEXT properties and `ENABLE_TOKENIZER_GSE=true` is set, GSE should initialize correctly and tokenize text for inverted index / BM25 as documented. [[Language-specific tokenization](https://docs.weaviate.io/weaviate/config-refs/collections#language-specific-tokenization)] - Replication commits (`/replicas/...:commit`) should not panic, and clients should not see 500 errors due to EOF on internal replica calls. * * * ### Actual behavior - With `tokenization=GSE` on Chinese text fields in a replicated, multi‑node cluster: - Replica nodes panic in `tokenizeGSE` because `Segmenter` is nil. - Coordinator nodes see EOF on `/replicas/indices//shards/:commit` calls to the cluster‑internal data port. - Clients receive 500 errors and objects are not added. * * * ### Request - Please confirm whether this is the same underlying problem as the known GSE dictionary loading issue in 1.34.x, or a separate bug in GSE initialization. - Ideally: - Ensure GSE dictionaries are loaded correctly in 1.34.x+; - Or, if initialization fails, avoid leaving `Segmenter` as nil so that `tokenizeGSE` does not panic in the replication commit path. - I can provide full startup logs (including any `Could not load dictionaries` messages) and a minimal Docker Compose + Python script to reproduce, if needed. ---
**Author:** ![Shahin](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/shahin/32/7944_2.png) [@Shahin](https://forum.weaviate.io/u/Shahin)\ **Post date:** [January 20, 2026, 3:04pm UTC](https://forum.weaviate.io/t/panic-when-using-gse-tokenizer-with-replication-nil-segmenter-eof-on-replicas-commit-in-v1-34-10/22222/2 "2026-01-20T15:04:17Z")
Hey @longfei_yao, You should definitely open a GitHub issue for this. Could you please copy your detailed report and stack trace to a new issue here: [https://github.com/weaviate/weaviate/issues](https://github.com/weaviate/weaviate/issues)? also including startup logs. The stack trace confirms a nil pointer dereference panic. Best regards, Mohamed Shahin Weaviate Admin (Ireland, UTC±00:00 / +01:00)