Weaviate shard in readonly during quantization 8bit

Support
,
1

Description

Tried enabling quantization on an existing collection via the API. However, for one shard, indexing switches to read-only mode, and queries through the Python client fail intermittently.

I also restored an older backup and repeated the same experiment in a sandbox environment, but observed the same behavior. What could be causing this?

Server Setup Information

  • Weaviate Server Version: 1.32
  • Deployment Method: Helm, Kubernetes
  • Multi Node: Yes, 3
  • Client Language and Version: Python , enabled quantization through REST
  • Multitenancy: No
  • Replicas: 2

Any additional Information

Time Event
13:42:48 Shard initialized and loaded in ~7.5 seconds
13:42:56 Shard marked as ready with 11,357,657 nodes
13:47:48 Tombstone cleanup started - processing 437,804 tombstones
13:47:48 - 13:48:04 Tombstone cleanup progressing (0 → 7M nodes processed)
13:56:48 Compaction halted - shard entered READONLY status
13:57:04-06 Multiple maintenance lock delays (>100ms waits)
14:09:12 Broadcast errors began - “read only” status
14:33:30-31 Continued broadcast failures
attaching the log file of weaivate-1 node where the shard is present Please read

image
image1894×814 89.3 KB

2

hi @Tibin !!

I see from logs you are using 1.32.0 :frowning:

The latest for this cut is 1.32.27 as I write: Tags · weaviate/weaviate · GitHub

Can you reproduce on that version and/or, better yet, on latest Release v1.35.3 - Bucket load limiter improvement, NaN average proplength Fix, multi2vec-voyageai module video modalilty support · weaviate/weaviate · GitHub ?

Thanks!