Description I am using the English word document for testing, it works perfect if I specified the “tokenization” as “word”, but get nothing if I switched to “gse”. Server Setup Information weaviate: image: semitechnologies/weaviate:1.24.11 command: - --host - 0.0.0.0 - --port…

hi @Cheng_Hao ! Check if this helps: import weaviate from weaviate import classes as wvc client = weaviate.connect_to_local() client.collections.delete("Test") collection = client.collections.create( name="Test", description="gse test", properties=[ wvc.config.Property(name="te…

And I do want to process the Chinese/Japanese document, that’s why I put the “gse” for tokenization, but after search GSE, I only got https://github.com/weaviate/weaviate/blob/257b2d366201faabf8bd0256f7e21a386bfdf08b/adapters/repos/db/helpers/tokenizer.go#L44, so I tried put the USE_GSE in the yaml …

Also, Welcome to our community :hugs:

Thank you so much DudaNogueira for the quick response. The demo code works well for me. But I have another question, will the “insert” API wait until the “index building” finished in the weaviate server? Or the “index building” will auto kick off some time later in the weaviate server after the inse…

Hi! Glad to hear that! insert will not wait. It will insert one object with one request. When you want to insert many objects with a single request, you can leverage insert_many. Now, if you want to insert a truck load of data, you will need to do some batch: [image] Batch import | Weavi…

Thank you very much, that’s really helpful. I guess the previous failure on keyword search probably due to the index building asynchronously. Now it works perfectly.

Glad to hear that @Cheng_Hao !! If you have any other issues in your Weaviate journey, we are here to help :slight_smile:

How to enable GSE tokenization for keyword search?

Support

DudaNogueira May 15, 2024, 12:59am 4

Also, Welcome to our community

Topic		Replies	Views
The GSE tokenizer has been configured, but the tokenization effect is poor when performing BM25 retrieval General technical	4	166	September 26, 2025
[Question] How to support keyword search in Chinese Support technical	1	403	November 18, 2024
GSE Tokenizer on WCD General wcs , technical	2	483	August 11, 2025
How can I config the gse tokenization Support python , technical	4	620	November 17, 2025
Cannot do keywords search for Chinese content in Python Support python	4	477	July 2, 2025

How to enable GSE tokenization for keyword search?

Related topics