Weaviate shard becoming read-only due to disk usage threshold – unable to insert objects

Support
, ,
1

Description

We are facing an issue while inserting objects into Weaviate. When our backend tries to index document chunks, the insert operation fails and Weaviate returns a read-only error due to high disk usage.
The error occurs during the document indexing pipeline when uploading a PDF and inserting the generated chunks into the UserUploadedDocuments index.
Error message received:
put object: import into index useruploadeddocuments:
put local object: shard=“oy0cxqVCiJGt”: store is read-only due to:
disk usage too high. Set to read-only at 90.02%, threshold set to 90.00%
Because of this, the indexing fails and no chunks are stored in the vector database.
We would like guidance on:
How to safely recover from this read-only state
Whether write operations resume automatically after freeing disk space
Best practices for managing disk thresholds in production deployments

Server Setup Information

  • Weaviate Server Version: emitechnologies/weaviate:1.28.3
  • Deployment Method: Kubernetes
  • Multi Node? Number of Running Nodes: No
  • Client Language and Version: 1
  • Multitenancy?: No

Any additional Information

According to our analysis the weviate checks for the node disk space while running instead of the separate PVC space which we have provided to it and our node space whenever gets filled above 90% we face this error.

2

Hey @Manish_Jangid,

Welcome to our community so lovely to have you here with us!

Read-only mode because Weaviate is protecting itself when disk usage crosses a certain threshold. This is actually expected behavior. Weaviate marks all shards on the affected node as READONLY when disk usage goes over DISK_USE_READONLY_PERCENTAGE (which defaults to 90%).

Here’s how to recover:

  1. Increase the storage on the node/volume where Weaviate data is stored (like expanding the PVC).

  2. Once that’s done, use the Weaviate clients or API to set the shards back to READY.

  3. Repeat for each affected shard.

Here’s a Python snippet to update shard status:

# Update shards status
coll = client.collections.use("<COLLECTION_NAME>")

shards = coll.config.update_shards(
    status="READY",
    shard_names=["<SHARD_NAME>"]  # The names (List[str]) of the shard to update (or a shard name)
)
print(shards)

Once you explicitly change the status back to READY, you can continue writing to Weaviate normally.

Best practices for managing disk thresholds in production:

Weaviate has two disk-related thresholds you should know about:

  • DISK_USE_WARNING_PERCENTAGE (default: 80%)

  • DISK_USE_READONLY_PERCENTAGE (default: 90%)

Recommendations:

  • Tune thresholds for your environment, set the warning percentage low enough to give you time to react (like 70–80%). Set the read-only percentage to leave some headroom for compaction, WAL, and OS needs.

  • Example (Kubernetes env vars):

    env:
  - name: DISK_USE_WARNING_PERCENTAGE
    value: "75"
  - name: DISK_USE_READONLY_PERCENTAGE
    value: "85"

  • Monitor disk usage at both node and volume level.

  • Plan capacity and cleanup, make sure your PVC/storage supports expansion and have a process to grow it over time. Regularly delete unused data and old backups, and have a solid data retention strategy. Use Weaviate’s backup modules.

  • Use warnings to avoid surprises, configure both disk and memory warning percentages so your monitoring alerts you before hitting read-only mode.

One more thing, please be aware you’re on a very old version. I’d strongly recommend upgrading to the latest 1.36.x version, you’ll avoid bugs and get all the improvements since then!

Best,
Mohamed Shahin
Weaviate Support Engineer