Select tokenizer for search/filter

mgreg · November 10, 2023, 5:11pm

I can select a tokenizer to use during ingest but how can I select a tokenizer for the query during search/filtering?

If I have a keyword like my_env_var_dont_split I can use ‘lowercase’ or ‘whitespace’ for tokenization during ingest and the keyword will not be split in the index (as far as I can tell). Please confirm.

But during search/filter it appears that I cannot specify a tokenizer and instead weaviate will try all tokenization methods and combine the results.

If the individual words of the keyword are all used in many objects it appears that this is diminishing the search effectiveness.

Is there another way to tell bm25 to only use a specific tokenizer?

mgreg · November 14, 2023, 8:00am

Thanks to @hsm207 for the answer. You don’t have to specify tokenization during search/filtering as the search term is automatically tokenized based on the property’s setting:

github.com

weaviate/weaviate/blob/6b66e6daa6418c93a9106409788abe1c43c22420/adapters/repos/db/inverted/searcher.go#L521-L532


      
          	switch propType {
          	case schema.DataTypeText:
          		// if the operator is like, we cannot apply the regular text-splitting
          		// logic as it would remove all wildcard symbols
          		if operator == filters.OperatorLike {
          			terms = helpers.TokenizeWithWildcards(prop.Tokenization, valueString)
          		} else {
          			terms = helpers.Tokenize(prop.Tokenization, valueString)
          		}
          	default:
          		return nil, fmt.Errorf("expected value type to be text, got %v", propType)
          	}

DudaNogueira · November 16, 2023, 8:43pm

Hi! Glad you sorted it out.

Thanks for sharing!