# BM25 returns high scores for English-heavy documents when query contains English tokens in Chinese-English mxied chunks (enable GSE)
**URL:** https://forum.weaviate.io/t/bm25-returns-high-scores-for-english-heavy-documents-when-query-contains-english-tokens-in-chinese-english-mxied-chunks-enable-gse/22248
**Category:** Support
**Tags:** python
**Created:** [February 28, 2026, 7:03pm UTC](https://forum.weaviate.io/t/bm25-returns-high-scores-for-english-heavy-documents-when-query-contains-english-tokens-in-chinese-english-mxied-chunks-enable-gse/22248 "2026-02-28T19:03:38Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Carloszone](https://forum.weaviate.io/u/Carloszone)
#### Post date: [February 28, 2026, 7:03pm UTC](https://forum.weaviate.io/t/bm25-returns-high-scores-for-english-heavy-documents-when-query-contains-english-tokens-in-chinese-english-mxied-chunks-enable-gse/22248/1 "2026-02-28T19:03:38Z")
### Description
I’m encountering unexpected behavior when using BM25 (BM25F) for keyword retrieval in a Chinese + English mixed corpus. I’d like to confirm whether this is expected behavior or a tokenizer/index configuration issue.
Relevant text properties use:
```auto
tokenization = gse
indexSearchable = true
```
#### Observed behavior
When the query contains English tokens (e.g. `SELECT 文本段落` ), documents that contain large amounts of English text receive disproportionately high BM25 scores, even when they are semantically irrelevant.
#### Explain score findings
Using `explain_score`, I observed that the query appears to be tokenized into both word-level and character-level tokens.
Example (simplified):
```auto
BM25F_select_frequency:100
BM25F_s_frequency:39
BM25F_e_frequency:29
BM25F_l_frequency:108
BM25F_c_frequency:10
BM25F_段落_frequency:145
BM25F_本_frequency:48
```
This suggests:
- English words are indexed both as whole tokens (`select`) and individual characters (`s`, `e`, `l`, `c`, `t`)
- Chinese text is also partially indexed at character level
Because single-character tokens occur very frequently, English-heavy documents accumulate large BM25 scores.
I guess the `GSE` tokenization setting cause the issue, but that is recomonded tokenization in Chinses or Janpanese text.
#### What I already tested
- Spaces in the query do not affect results significantly
- Lowercasing English tokens does not solve the issue
- Behavior appears strongly correlated with English token presence
#### Expected behavior
I would expect BM25 ranking to rely primarily on meaningful tokens (words or segmented phrases), not character-level matches that dominate scoring.
#### Questions
1. Is it expected that `gse` tokenization produces character-level tokens that participate in BM25 scoring?
2. Is BM25F designed to use both word-level and character-level tokens simultaneously?
3. What is the recommended tokenizer for BM25 in mixed Chinese + English corpora?
- `gse`
- `trigram`
- `word`
- Separate BM25 field without GSE?
1. Is it a recommended pattern to maintain separate fields for:
- semantic embedding (GSE)
- keyword BM25 (non-GSE tokenizer)?
1. Are there ways to reduce the impact of single-character tokens on BM25 scoring?
### Goal
I want keyword retrieval to:
- Work well for Chinese queries
- Not over-score documents simply because they contain large amounts of English text
- Avoid heavy query routing or custom filtering logic
Any guidance or best practices would be greatly appreciated.
Thanks!
### Server Setup Information
- Weaviate Server Version:1.31.2
- Deployment Method: docker
- Multi Node? Number of Running Nodes: 1 node
- Client Language and Version: 4.15.2
- Multitenancy?: No
### Any additional Information
Code blocks:
```auto
res_keywords = self.db_instance.child_collection.query.bm25(
query="SELECT 文本段落", # I tried "SELECT文本段落" to check if " " caused this issue
return_properties=['parent_uuid', 'content', "file_id", "content_chapter_info", "file_name"],
return_metadata=MetadataQuery(distance=True, score=True, explain_score=True),
limit=max_return_limit,
filters=file_id_filter)
for i, obj in enumerate(res_keywords.objects[:10], 1):
print("="*10)
print(f"[{i}] score={getattr(obj.metadata, 'score', None)} distance={getattr(obj.metadata, 'distance', None)}")
explain = getattr(obj.metadata, "explain_score", None) or getattr(obj.metadata, "explainScore", None)
print("EXPLAIN:\n", explain)
```
---
### Author:  [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)
#### Post date: [March 2, 2026, 2:31pm UTC](https://forum.weaviate.io/t/bm25-returns-high-scores-for-english-heavy-documents-when-query-contains-english-tokens-in-chinese-english-mxied-chunks-enable-gse/22248/2 "2026-03-02T14:31:39Z")
Hi @Carloszone !!
This is the expected behavior. GSE produces both word-level and character-level tokens, and BM25F uses all tokens for scoring. Character tokens like ‘s’, ‘e’, ‘l’ appear frequently in English text, causing high scores for English-heavy documents.
So I believe a good approach would be:
### 1. Use Different Tokenizer for BM25
For mixed Chinese-English corpora where you want to avoid character-level token scoring, consider using:
- `word` tokenization for BM25 (better for mixed languages)
- Keep GSE only for semantic embedding fields
### 2. Separate Fields Pattern
Yes, maintaining separate fields is a recommended pattern:
- One field with GSE tokenization for semantic search/embeddings
- Another field with `word` or `trigram` tokenization for BM25 keyword search
Let me know if this helps!
Thanks!
---
### Author:  [@Carloszone](https://forum.weaviate.io/u/Carloszone)
#### Post date: [March 4, 2026, 5:38am UTC](https://forum.weaviate.io/t/bm25-returns-high-scores-for-english-heavy-documents-when-query-contains-english-tokens-in-chinese-english-mxied-chunks-enable-gse/22248/3 "2026-03-04T05:38:50Z")
Hi, @DudaNogueira
Thank you for your reply.
I have redesigned my collection structure to solve the issue.
In short, I added two additional attributes: `content_zh_bm25` (which stores only Chinese content with GSE tokenization) and `content_en_bm25` (which stores only English content with LOWERCASE tokenization).
Now the performance of the new structure is very good. It only costs a little more storage space but provides more accurate keyword search results.
---
### Author:  [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)
#### Post date: [March 4, 2026, 4:59pm UTC](https://forum.weaviate.io/t/bm25-returns-high-scores-for-english-heavy-documents-when-query-contains-english-tokens-in-chinese-english-mxied-chunks-enable-gse/22248/4 "2026-03-04T16:59:35Z")
Awesome! Glad it helped!