Null indexing doesn't work with newly added property

Support
1

I have the following class:

{'class': 'Label',
 'description': 'Labels for documents',
 'invertedIndexConfig': {'bm25': {'b': 0.75, 'k1': 1.2},
  'cleanupIntervalSeconds': 60,
  'indexNullState': True,
  'stopwords': {'additions': None, 'preset': 'en', 'removals': None}},
 'moduleConfig': {'text2vec-openai': {'model': 'ada',
   'modelVersion': '002',
   'type': 'text',
   'vectorizeClassName': False}},
 'properties': [{'dataType': ['text'],
   'description': 'Content of the label',
   'indexFilterable': True,
   'indexSearchable': True,
   'moduleConfig': {'text2vec-openai': {'skip': False,
     'vectorizePropertyName': False}},
   'name': 'content',
   'tokenization': 'field'},
  {'dataType': ['Knowledge'],
   'description': '(label) belongs to knowledge',
   'indexFilterable': True,
   'indexSearchable': False,
   'moduleConfig': {'text2vec-openai': {'skip': True,
     'vectorizePropertyName': False}},
   'name': 'belongsToKnowledge'}],
 'replicationConfig': {'factor': 1},
 'shardingConfig': {'virtualPerPhysical': 128,
  'desiredCount': 1,
  'actualCount': 1,
  'desiredVirtualCount': 128,
  'actualVirtualCount': 128,
  'key': '_id',
  'strategy': 'hash',
  'function': 'murmur3'},
 'vectorIndexConfig': {'skip': False,
  'cleanupIntervalSeconds': 300,
  'maxConnections': 64,
  'efConstruction': 128,
  'ef': -1,
  'dynamicEfMin': 100,
  'dynamicEfMax': 500,
  'dynamicEfFactor': 8,
  'vectorCacheMaxObjects': 1000000000000,
  'flatSearchCutoff': 40000,
  'distance': 'cosine',
  'pq': {'enabled': False,
   'bitCompression': False,
   'segments': 0,
   'centroids': 256,
   'encoder': {'type': 'kmeans', 'distribution': 'log-normal'}}},
 'vectorIndexType': 'hnsw',
 'vectorizer': 'text2vec-openai'}

I added three more properties to the class for better filtering. One of them is type.
The new schema is this:

{'class': 'Label',
 'description': 'Labels for documents',
 'invertedIndexConfig': {'bm25': {'b': 0.75, 'k1': 1.2},
  'cleanupIntervalSeconds': 60,
  'indexNullState': True,
  'stopwords': {'additions': None, 'preset': 'en', 'removals': None}},
 'moduleConfig': {'text2vec-openai': {'model': 'ada',
   'modelVersion': '002',
   'type': 'text',
   'vectorizeClassName': False}},
 'properties': [{'dataType': ['text'],
   'description': 'Content of the label',
   'indexFilterable': True,
   'indexSearchable': True,
   'moduleConfig': {'text2vec-openai': {'skip': False,
     'vectorizePropertyName': False}},
   'name': 'content',
   'tokenization': 'field'},
  {'dataType': ['Knowledge'],
   'description': '(label) belongs to knowledge',
   'indexFilterable': True,
   'indexSearchable': False,
   'moduleConfig': {'text2vec-openai': {'skip': True,
     'vectorizePropertyName': False}},
   'name': 'belongsToKnowledge'},
  {'dataType': ['text'],
   'description': 'type or substype of the label, used for filtering',
   'indexFilterable': True,
   'indexSearchable': True,
   'moduleConfig': {'text2vec-openai': {'skip': True,
     'vectorizePropertyName': False}},
   'name': 'type',
   'tokenization': 'word'},
  {'dataType': ['int'],
   'description': 'postgres ID of the entity, if applicable. None if N/A',
   'indexFilterable': True,
   'indexSearchable': False,
   'moduleConfig': {'text2vec-openai': {'skip': True,
     'vectorizePropertyName': False}},
   'name': 'postgresId'},
  {'dataType': ['text'],
   'description': 'additional data of the entity that the label describes. In the format of a JSON object string',
   'indexFilterable': True,
   'indexSearchable': True,
   'moduleConfig': {'text2vec-openai': {'skip': True,
     'vectorizePropertyName': False}},
   'name': 'additional_data',
   'tokenization': 'word'}],
 'replicationConfig': {'factor': 1},
 'shardingConfig': {'virtualPerPhysical': 128,
  'desiredCount': 1,
  'actualCount': 1,
  'desiredVirtualCount': 128,
  'actualVirtualCount': 128,
  'key': '_id',
  'strategy': 'hash',
  'function': 'murmur3'},
 'vectorIndexConfig': {'skip': False,
  'cleanupIntervalSeconds': 300,
  'maxConnections': 64,
  'efConstruction': 128,
  'ef': -1,
  'dynamicEfMin': 100,
  'dynamicEfMax': 500,
  'dynamicEfFactor': 8,
  'vectorCacheMaxObjects': 1000000000000,
  'flatSearchCutoff': 40000,
  'distance': 'cosine',
  'pq': {'enabled': False,
   'bitCompression': False,
   'segments': 0,
   'centroids': 256,
   'encoder': {'type': 'kmeans', 'distribution': 'log-normal'}}},
 'vectorIndexType': 'hnsw',
 'vectorizer': 'text2vec-openai'}

Theoretically, all existing Labels in the database would have type Null, and the isNull operator should work. However, the following graphql returns nothing.

{
	Get {
		Label (
			limit: 10
			nearText: {concepts:["is a coffee shop"]}
			where: {
				operator: IsNull,
				path: ["type"],
				valueBoolean: true
			}
		){
			content
			type
			additional_data
			postgresId
			_additional {id}
		}
	}
}

Is this supposed to happen? How do I overcome this?

2

Hi! I will create a notebook with this scenario later today so we can debug it further.

thanks!

3

Hi! I have created the notebook here:

and asked this question internally. I’ll get back here as soon as I have some feedback.

Thanks!

4

Hi! Indeed it will not add that field to objects created before the addition of this new property.

A solution here is move those objects to a new class using the cursor