Description Hello. Currently, our Weaviate instance has a stable number of vectors which is ~10,5 kk and is not increasing. However we are hitting storage limits because something constantly eating free space in our nodes. At first glance it looks like some failures with operating HNSW commitlogs. …

Unfortunately in your case you need to manually fix this. I think this are the steps that you need to do: log into the node where this exception occurs and find the proplengths files which BucketedData is nil and delete those proplengths files restart this node with RECOUNT_PROPERTIES_AT_STARTUP …

[image] Grzegorz_Pasieka: At first glance it looks like some failures with operating HNSW commitlogs. Hi @Grzegorz_Pasieka , do you know which files are becoming bigger or created over time? Thanks!

We have had those problems since v1.24.2, that’s why we decided to upgrade to 1.24.14 to check if it helps somehow.

[image] jeronimo_irazabal: Hi @Grzegorz_Pasieka , do you know which files are becoming bigger or created over time? Thanks! I think main.hnsw.commitlog.d is becoming bigger in each shard

What is also worth mentioning is that one of my replicas constantly crashes because of goroutine panics: {"level":"error","msg":"Recovered from panic: assignment to entry in nil map","time":"2024-05-21T22:52:45Z"} goroutine 7255736 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/s…

@Grzegorz_Pasieka we have pushed fixes for hnsw combining logic so if you are using v1.24.14 you should have all of them already deployed. Do you observe that your main.hnsw.commitlog.d is becoming already smaller in size?

@Grzegorz_Pasieka we have pushed fixes for hnsw combining logic so if you are using v1.24.14 you should have all of them already deployed. Do you observe that your main.hnsw.commitlog.d is becoming already smaller in size? We are still observing this deployment however I can see another file siz…

Today we noticed storage usage drops in each replica. Surprisingly it was not because of commit logs which are still growing, but objects in the second shard. [image]

It’s great that the usage is decreasing and that the compaction works for objects. I think that I would be able to share with you a preview docker image with fixes that might help in your situation. I’m just now gathering all of it here internally, I can then share with you a preview image (that is …

we recorded another drop in storage usage in the first shard. We also performed migration process to 1.25.1 which I already described in another topic Panic after migration to 1.25.1

@Grzegorz_Pasieka I want to inform you also that we will be releasing tomorrow (the latest on Wednesday) new patch releases of v1.24 and v1.25 that will contain fixes that should improve your situation. We have identified those bugs and we are now pushing all of the fixes to our code base, so pleas…

Constant storage drain without increasing number of vectors

Support

Topic		Replies	Views
Production weaviate 24.6 crashed Support technical	1	815	March 8, 2025
Crashing 1.24.10 Support wcs , python	5	1070	May 7, 2024
Weaviate increasing memory usage after a successful backup daily Support developer-experience	1	636	June 7, 2024
Crashing weaviate 24.6 Support technical	1	670	March 8, 2025
Hnsw load commit log corruption error Support	8	1089	March 14, 2024

Constant storage drain without increasing number of vectors

Related topics