401 Unauthorized When Anonymous Access Enabled

Support
1

Description

Hi we have a use case where we are creating thousands of tenants. When we initially did this Weaviate started choking because it was holding too many tenant files in memory. Given this we were instructed to move our tenant states out of the HOT status. So now when a new tenant comes through we will create them, flip their status to HOT save data into the tenant, and flip their status to COLD. When we need to query the tenant, which is sparingly we will flip the tenant to HOT, run our query, and then flip them back to COLD. However, we are now getting the following error message for a little over half of our tenants

Update classes tenants! Unexpected status code: 401, with response body: {'message': 'Unauthorized'}

Our setup anonymous access enabled so we should never see this error from Weaviate, and it only seems to happen when we are updating the status of multiple tenants in parallel. If I go back and retry the updating of the tenants one by one they will pass, but when it is happening at scale weaviate throws the error. Upon looking through the kubernetes pod logs it throws a shard not found error.

{"build_git_commit":"9069628","build_go_version":"go1.22.10","build_image_tag":"v1.28.0","build_wv_version":"1.28.0","cmd_class":"Contractexcellence","cmd_type":17,"cmd_type_name":"TYPE_UPDATE_TENANT","error":"updating schema: TYPE_UPDATE_TENANT: shard not found: [a5b556a195816880939bbc2d4bf9d37a24c3c654db564de633b648bfc16beaeb]","level":"error","log_index":54947,"log_name":"LogCommand","log_type":0,"msg":"apply command","time":"2025-01-24T06:07:13Z"}

Any suggestions or guidance on how to solve this issue or why it is happening at all would be greatly appreciated.

Server Setup Information

  • Weaviate Server Version: 1.28.0
  • Deployment Method: AWS EKS Fargate instance backed by EFS
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: python V3
  • Multitenancy?: Yes

Any additional Information

2

Hello, could you try again with v1.28.4? We recently refactored our error handling a bit and found a couple cases where a wrong error type was returned.

This will most likely not solve it, but return the correct error which might have more pointers