Specifying properties with multi-tenancy causes bug

Tejas_Sharma · September 1, 2024, 4:57am

Description

When I just specify multi-tenancy, all my insert / updates work.

However, when I just add properties to the schema declaration, I start to get this error:

weaviate.exceptions.UnexpectedStatusCodeError: Object was not updated.! Unexpected status code: 422, with response body: {‘error’: [{‘message’: ‘msg:repo.object code:422 err:search index nodes: class Nodes has multi-tenancy disabled, but request was with tenant’}]}.

This is the code:

nodes_collection = client.collections.create(
		name="Nodes",
		vectorizer_config=wvc.config.Configure.Vectorizer.none(),
		vector_index_config=Configure.VectorIndex.hnsw(
			distance_metric=VectorDistances.COSINE
		),
		# Multi tenancy to separate each user's data
		multi_tenancy_config=Configure.multi_tenancy(enabled=True, auto_tenant_creation=True, auto_tenant_activation=True),
		# Specify some properties beforehand to set right data type (i.e. obj[] instead of string[])
		properties=[
			Property(name="tags", data_type=DataType.OBJECT_ARRAY),
		]
	)

If I remove the properties field, then it will work again.

This is an example insert code:

def get_tenant_collection(name):
	try:
		return nodes_collection.with_tenant(name)
	except:
		nodes_collection.tenants.create(name)
		return nodes_collection.with_tenant(name)

tenant_collection = get_tenant_collection(tenant_name)
	
tenant_collection.data.insert(
	vector=record.vector,
	properties=record.properties,
	uuid=record.uniqueid
)

Even update / delete, all operations trigger the multi-tenancy error. I can confirm the rest of the code is working including getting the tenant collection because just commenting out properties, will make everything start working

nodes_collection = client.collections.create(
		name="Nodes",
		vectorizer_config=wvc.config.Configure.Vectorizer.none(),
		vector_index_config=Configure.VectorIndex.hnsw(
			distance_metric=VectorDistances.COSINE
		),
		# Multi tenancy to separate each user's data
		multi_tenancy_config=Configure.multi_tenancy(enabled=True, auto_tenant_creation=True, auto_tenant_activation=True),
		# Commenting this out makes all operations work again
		# properties=[
		# 	Property(name="tags", data_type=DataType.OBJECT_ARRAY),
		# ]
	)

Server Setup Information

Weaviate Server Version: Cloud
Multi Node? Number of Running Nodes: No
Client Language and Version: Python v4
Multitenancy?: Yes

DudaNogueira · September 2, 2024, 6:09pm

hi @Tejas_Sharma !!

I believe you are facing this issue:

github.com/weaviate/weaviate

WeaviateQueryError when searching for objects stored as a refrence property in a multitenant collection or schema

opened 11:55AM - 02 Jul 24 UTC

rthiiyer82

bug

### How to reproduce this bug? As per the weaviate documentation one should be …able to cross reference from a multi tenant to a non multi tenant. Run the below script ``` import random import string from weaviate.classes.config import Property, DataType, ReferenceProperty import weaviate import weaviate.classes.config as wvc from weaviate.classes.tenants import Tenant from weaviate.classes.query import QueryReference client = weaviate.connect_to_local() # Create collection if (client.collections.exists("JeopardyCategory")): # delete collection "Article" - THIS WILL DELETE THE COLLECTION AND ALL ITS DATA client.collections.delete("JeopardyCategory") # Replace with your collection name # Create collection if (client.collections.exists("JeopardyQuestion")): # delete collection "Article" - THIS WILL DELETE THE COLLECTION AND ALL ITS DATA client.collections.delete("JeopardyQuestion") # Replace with your collection name category = client.collections.create( name="JeopardyCategory", description="A Jeopardy! category", properties=[ Property(name="title", data_type=DataType.TEXT) ], replication_config=wvc.Configure.replication(factor=1), ) jeopardy_question= client.collections.create( name="JeopardyQuestion", description="A Jeopardy! question", properties=[ Property(name="question", data_type=DataType.TEXT), Property(name="answer", data_type=DataType.TEXT), ], references=[ ReferenceProperty( name="hasCategory", target_collection="JeopardyCategory" ) ], replication_config=wvc.Configure.replication(factor=1), multi_tenancy_config=wvc.Configure.multi_tenancy(enabled=True, auto_tenant_creation=True, auto_tenant_activation=True) ) # Create tenant on JeopardyQuestion jeopardy_question.tenants.create( tenants=[ Tenant(name="tenantA"), Tenant(name="tenantB"), ] ) # Insert objects to the collection def get_random_string(length): # choose from all lowercase letter letters = string.ascii_lowercase result_str = ''.join(random.choice(letters) for i in range(length)) return result_str # print("Random string of length", length, "is:", result_str) # Add category to JeopardyCategory jeopardy = client.collections.get("JeopardyCategory") category_uuid = jeopardy.data.insert({ "title": get_random_string(10) }) questions= jeopardy_question.with_tenant(tenant="tenantA") questions.data.insert({ "question": "question" + get_random_string(10), "answer" : "answer"+ get_random_string(10) }, references={"hasCategory": category_uuid}, # e.g. {"hasCategory": "583876f3-e293-5b5b-9839-03f455f14575"} ) response = questions.query.fetch_objects(return_references=QueryReference(link_on="hasCategory", return_properties=["question", "answer"])) for o in response.objects: print(o.properties) client.close() ``` ### What is the expected behavior? No error should be displayed and user should be able to query the reference property in a multitenant collection. ### What is the actual behavior? ``` weaviate.exceptions.WeaviateQueryError: Query call with protocol GRPC search failed with message <AioRpcError of RPC that terminated with: status = StatusCode.UNKNOWN details = "explorer: list class: search: resolve cross-refs: build reference cache: build request cache: fetch job list: index "jeopardycategory": class JeopardyCategory has multi-tenancy disabled, but request was with tenant" debug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-07-01T19:38:14.318273+01:00", grpc_status:2, grpc_message:"explorer: list class: search: resolve cross-refs: build reference cache: build request cache: fetch job list: index \"jeopardycategory\": class JeopardyCategory has multi-tenancy disabled, but request was with tenant"}" >. ``` ### Supporting information https://weaviate.io/developers/weaviate/manage-data/multi-tenancy#cross-references ### Server Version 1.25.6 ### Code of Conduct - [X] I have read and agree to the Weaviate's [Contributor Guide](https://weaviate.io/developers/contributor-guide) and [Code of Conduct](https://weaviate.io/service/code-of-conduct)

Please, make sure to leave a on that issue so we can prioritize it.

THanks!

Tejas_Sharma · September 3, 2024, 5:05am

Hi @DudaNogueira ,

I think mine is completely different because their bug is about multi-tenant to non-tenant but mine is that there is a bug in the SDK that if both multi tenancy and properties are specified, then everything breaks

DudaNogueira · September 3, 2024, 12:15pm

Oh, sorry.

Do you think you can create a reproducible python code?

With that we better understand the issue by reproducing or fixing any code or assumptions along the way.

That issue has a python code that can server as starting point.

Thanks!

Tejas_Sharma · September 3, 2024, 4:54pm

Okay thanks Duda, will try to get it out soon

Tejas_Sharma · September 12, 2024, 4:22pm

Hi @DudaNogueira , aplogies for the delay, here’s the minimal reproducible code:

import uuid
import weaviate
import weaviate.classes as wvc
from weaviate.classes.query import Filter, MetadataQuery
from weaviate.classes.config import Configure, VectorDistances, Property, DataType
from weaviate.classes.tenants import Tenant
from datetime import datetime, timedelta
import pytz

try:
	# TODO: use different credentials for production
	# Best practice: store your credentials in environment variables
	wcd_url = ""
	wcd_api_key = ""

	client = weaviate.connect_to_weaviate_cloud(
		cluster_url=wcd_url,                                    # Replace with your Weaviate Cloud URL
		auth_credentials=wvc.init.Auth.api_key(wcd_api_key),    # Replace with your Weaviate Cloud key
	)
except Exception as e:
	print(e)
	# exit
	pass


try:
	# For all objects
	nodes_collection = client.collections.create(
		name="Nodes",
		vectorizer_config=wvc.config.Configure.Vectorizer.none(),
		vector_index_config=Configure.VectorIndex.hnsw(
			distance_metric=VectorDistances.COSINE
		),
		# Multi tenancy to separate each user's data
		multi_tenancy_config=Configure.multi_tenancy(enabled=True, auto_tenant_creation=True, auto_tenant_activation=True),
		inverted_index_config=Configure.inverted_index( 
			index_null_state=True,
        	index_property_length=True,
			index_timestamps=True
		)
		# Specify some properties beforehand to set right data type (i.e. obj[] instead of string[])
		properties=[
			Property(name="tags", data_type=DataType.OBJECT_ARRAY),
		]
	)
except:
	nodes_collection = client.collections.get("Nodes")

try:
	# Create tenant on Node
	nodes_collection.tenants.create(
		tenants=[
			Tenant(name="tenantA"),
			Tenant(name="tenantB"),
		]
	)
except:
	pass


# Will cause an error
nodes_collection.with_tenant('tenantA').data.insert(
	vector=[0.0] * 384,
	properties={'lastUpdateDeviceId': 'device-78C24351-F40A-4E37-8953-F003FA474877'},
	uuid=str(uuid.uuid4())
)

DudaNogueira · September 16, 2024, 1:59pm

Hi @Tejas_Sharma !!

You need to specify at least one nested property if using the object. Also, there was a missing , in your code, right after inverted_index_config

Here is a working example:

import uuid
import weaviate
import weaviate.classes as wvc
from weaviate.classes.query import Filter, MetadataQuery
from weaviate.classes.config import Configure, VectorDistances, Property, DataType
from weaviate.classes.tenants import Tenant
from datetime import datetime, timedelta
import pytz

# For all objects
client.collections.delete("Test")
nodes_collection = client.collections.create(
    name="Test",
    vectorizer_config=wvc.config.Configure.Vectorizer.none(),
    vector_index_config=Configure.VectorIndex.hnsw(
        distance_metric=VectorDistances.COSINE
    ),
    # Multi tenancy to separate each user's data
    multi_tenancy_config=Configure.multi_tenancy(enabled=True, auto_tenant_creation=True, auto_tenant_activation=True),
    inverted_index_config=Configure.inverted_index( 
        index_null_state=True,
        index_property_length=True,
        index_timestamps=True
    ),
    # Specify some properties beforehand to set right data type (i.e. obj[] instead of string[])
    properties=[
        wvc.config.Property(
            name="tags",
            data_type=wvc.config.DataType.OBJECT,
            nested_properties=[
                wvc.config.Property(
                    name="sub_property1",
                    data_type=wvc.config.DataType.TEXT
                ),
                wvc.config.Property(
                    name="sub_property2",
                    data_type=wvc.config.DataType.INT
                )
            ]
        )
    ]
)


# Create tenant on Node
nodes_collection.tenants.create(
    tenants=[
        Tenant(name="tenantA"),
        Tenant(name="tenantB"),
    ]
)

# Will cause an error
nodes_collection.with_tenant('tenantA').data.insert(
	vector=[0.0] * 384,
	properties={
        'lastUpdateDeviceId': 'device-78C24351-F40A-4E37-8953-F003FA474877',
        "tags": {"sub_property1": "somedata", "sub_property2": 123}        
    },
	uuid=str(uuid.uuid4())
)

Let me know if this helps.

Thanks!

Tejas_Sharma · September 16, 2024, 6:14pm

Ah okay, thanks Duda. We actually found a workaround solution of just creating the first record with tags specified to prevent this error but will try this out in the future.

I would like to add though that it would help if it specified that error during the node creation rather than the multi tenancy error since the error it was giving was not related to the fix and would thus be hard to debug

DudaNogueira · September 16, 2024, 9:08pm

Oh, I see. This is a pitfall of the AUTOSCHEMA_ENABLED feature.

Note that we do not recommend leaving AUTOSCHEMA_ENABLED in production as it can create new properties if you specify a wrong property name.

If you have AUTOSCHEMA_ENABLED and do not specify a property at collection creation, the first property inserted will change your schema for you.

This code will fail on collection creation, and not on tenant creation:

nodes_collection = client.collections.create(
    name="Nodes",
    vectorizer_config=wvc.config.Configure.Vectorizer.none(),
    vector_index_config=Configure.VectorIndex.hnsw(
        distance_metric=VectorDistances.COSINE
    ),
    # Multi tenancy to separate each user's data
    multi_tenancy_config=Configure.multi_tenancy(enabled=True, auto_tenant_creation=True, auto_tenant_activation=True),
    inverted_index_config=Configure.inverted_index( 
        index_null_state=True,
        index_property_length=True,
        index_timestamps=True
    ),
    # Specify some properties beforehand to set right data type (i.e. obj[] instead of string[])
    properties=[
        Property(name="tags", data_type=DataType.OBJECT_ARRAY),
    ]
)

error message:

UnexpectedStatusCodeError: Collection may not have been created properly.! Unexpected status code: 422, with response body: {‘error’: [{‘message’: “Property ‘tags’: At least one nested property is required for data type object/object”}]}.

Topic		Replies	Views
Cannot Add Tenancy to Weaviate Class Support	6	758	August 15, 2023
"Tenant not found" with automatic tenant creation enabled Support	5	157	June 2, 2024
BUG: Unable to delete with Multi-tenancy v1.21.7 & python client 3.25.2 Support bug	1	455	November 16, 2023
Nested objects in schema cause Error: Property "nestedProperties" is not known Support	2	728	November 16, 2023
Error: Open cluster-wide transaction: concurrent transaction Support	1	86	June 3, 2024

Specifying properties with multi-tenancy causes bug

Description

Server Setup Information

Related topics