For some reason, weaviate decided to raise resource usage, but nothing is happening on the weaviate container(Nothing was seen in the log).
Lil bit of investigation later I found out this is happening during backups. Also Is this really how much resources the backup utilizes?
Hi!
What is the version you are running?
Also, how many objects? is it a single cluster? How is the backup being done?
does it get back after the backup?
Thanks!
It’s 1.22.5, 2.2mil objects text-ada embeddings on single docker swarm deployment. Backups are done using the built-in s3 backup module.
The server had 32GB of RAM and a CPU and memory spike would occur whenever a backup was triggered. This would cause the entire server to go down Increasing the resources on the server did resolve this but overall I’m still confused about the options like LIMIT_RESOURCES as in my local testing they don’t seem to do much at least for backup and restores
Oh, ok. A backup can indeed explain that.
does your local testing env have the specs and sizing of your prod?
No, My local setup didn’t have that much but I remember testing with smaller amount of sample data.