Hi there,
Description
I am using WCS. I ran a batch job overnight, adding ~1 million objects into a collection, using the Python v4 client. Vector dimension per object is 768. I enabled PQ compression. I have 6 properties in the collection tokenized/indexed to support hybrid search.
The overnight batch job completed, and I can see by running collection.aggregate.over_all(total_count=True) that there are close to the expected 1 million objects in the collection. I figured out that the shard associated with my collection got set to READONLY during or upon completion of the batch job, likely due to high disk usage. I’ve set the shard status back to READY and confirmed that it is in READY status.
Just now, I tried adding a single object to my collection, again via the Python v4 client, and I get this error:
UnexpectedStatusCodeError: Object was not added! Unexpected status code: 500, with response body: {'error': [{'message': 'put object: import into index structured_metrics: put local object: shard="{shard_name}": store object in LSM store: upsert object data: write into commit log: write /var/lib/weaviate/{collection}/{shard_name}/lsm/objects/segment-1714753037834860823.wal: no space left on device'}]}.
How do I add space? How do I estimate how much space I need for 10 million objects?
Server Setup Information
- Weaviate Server Version: 1.24.10
- Deployment Method: WCS
- Multi Node? Number of Running Nodes: 1 node
- Client Language and Version: I’m using the python v4 client.