# Weaviate increasing memory usage after a successful backup daily **URL:** \ **Category:** Support\ **Tags:** developer-experience\ **Created:** [June 6, 2024, 1:55pm UTC](https://forum.weaviate.io/t/weaviate-increasing-memory-usage-after-a-successful-backup-daily/2633 "2024-06-06T13:55:38Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Divyansh\_Mishra](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/divyansh_mishra/32/1562_2.png) [@Divyansh\_Mishra](https://forum.weaviate.io/u/Divyansh_Mishra)\ **Post date:** [June 6, 2024, 1:55pm UTC](https://forum.weaviate.io/t/weaviate-increasing-memory-usage-after-a-successful-backup-daily/2633/1 "2024-06-06T13:55:38Z")
### Description We have the backup s3 module enabled and after every backup, the memory usage increases by 3-5GB daily; one day, the spike was 10GB. The memory usage normally for us is around 20GB and while taking a backup it goes up to around 40GB and comes back but the delta increase _(The memory at which it came back after the spike)_ is increasing daily. Currently, the weaviate uses 26GB of memory since it’s increasing daily after the backup process. ### Server Setup Information - **Weaviate Server Version** : 1.23.6 - **Deployment Method** : k8s - **Multi Node? Number of Running Nodes** : Single node - **Client Language and Version** : Python V3 - **Multitenancy** : enabled Spike in memory during the backup process: (It increases daily after the backup) ![image](https://canada1.discourse-cdn.com/flex027/uploads/weaviate/original/2X/7/79c1d7afc185dcae2224032ffc7cca5afd06a4a2.png) ### Logs when the backup process started > time=“2024-06-05T23:30:01Z” level=trace msg=“no segment eligible for compaction” action=lsm\_compaction class=Paragraph index=paragraph path=/var/lib/weaviate/paragraph/NiIyjUVQT/lsm/property\_source\_searchable shard=NiIyjUVQT > time=“2024-06-05T23:30:01Z” level=trace msg=“no segment eligible for compaction” action=lsm\_compaction class=Paragraph index=paragraph path=/var/lib/weaviate/paragraph/NiIyjUVQT/lsm/property\_content\_searchable shard=NiIyjUVQT BACKUP STARTED BELOW > time=“2024-06-05T23:30:01Z” level=debug msg=“received HTTP request” action=restapi\_request method=POST url=/v1/backups/s3 > time=“2024-06-05T23:30:02Z” level=info action=try\_backup backend=s3 backup\_id=05-06-2024\_23-30-01-k8s took=436.881717ms > time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit\_log\_file\_switched id=main new\_file\_name=1717630202 old\_file\_name=1717543806 old\_file\_size=0 > time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit\_log\_file\_switched id=main new\_file\_name=1717630202 old\_file\_name=1717543803 old\_file\_size=0 > time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit\_log\_file\_switched id=main new\_file\_name=1717630202 old\_file\_name=1717543803 old\_file\_size=0 > time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit\_log\_file\_switched id=main new\_file\_name=1717630202 old\_file\_name=1717543803 old\_file\_size=0 > time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit\_log\_file\_switched id=main new\_file\_name=1717630202 old\_file\_name=1717543807 old\_file\_size=0 > time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit\_log\_file\_switched id=main new\_file\_name=1717630202 old\_file\_name=1717543808 old\_file\_size=0 > time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit\_log\_file\_switched id=main new\_file\_name=1717630202 old\_file\_name=1717543804 old\_file\_size=0 > time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit\_log\_file\_switched id=main new\_file\_name=1717630202 old\_file\_name=1717543808 old\_file\_size=0 AFTER BACKUP > time=“2024-06-05T23:37:14Z” level=debug msg=“received HTTP request” action=restapi\_request method=GET url=/v1/meta > time=“2024-06-05T23:37:16Z” level=info msg=“finish uploading files” class=Paragraph > time=“2024-06-05T23:37:16Z” level=info msg=“start uploading meta data” > time=“2024-06-05T23:37:16Z” level=debug msg=starting class=Paragraph id=05-06-2024\_23-30-01-k8s op=release\_backup > time=“2024-06-05T23:37:16Z” level=info msg=“finish uploading meta data” > time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/xdoDw-Vfi/main.hnsw.commitlog.d/1703479198.condensed input\_second=/var/lib/weaviate/paragraph/xdoDw-Vfi/main.hnsw.commitlog.d/1717543802.condensed output=/var/lib/weaviate/paragraph/xdoDw-Vfi/main.hnsw.commitlog.d/1703479198 > time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/htBlqqM9C/main.hnsw.commitlog.d/1713163635.condensed input\_second=/var/lib/weaviate/paragraph/htBlqqM9C/main.hnsw.commitlog.d/1717543803.condensed output=/var/lib/weaviate/paragraph/htBlqqM9C/main.hnsw.commitlog.d/1713163635 > time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/LnVcb310\_/main.hnsw.commitlog.d/1707579378.condensed input\_second=/var/lib/weaviate/paragraph/LnVcb310\_/main.hnsw.commitlog.d/1717543807.condensed output=/var/lib/weaviate/paragraph/LnVcb310\_/main.hnsw.commitlog.d/1707579378 > time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/K2XWGL9oK/main.hnsw.commitlog.d/1708499782.condensed input\_second=/var/lib/weaviate/paragraph/K2XWGL9oK/main.hnsw.commitlog.d/1717543803.condensed output=/var/lib/weaviate/paragraph/K2XWGL9oK/main.hnsw.commitlog.d/1708499782 > time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/D8\_MRzWnm/main.hnsw.commitlog.d/1704735872.condensed input\_second=/var/lib/weaviate/paragraph/D8\_MRzWnm/main.hnsw.commitlog.d/1717543803.condensed output=/var/lib/weaviate/paragraph/D8\_MRzWnm/main.hnsw.commitlog.d/1704735872 > time=“2024-06-05T23:37:16Z” level=debug msg=“index grown from 1000 to 7044, took 16.7µs\n” action=hnsw\_grow\_index new\_size=7044 previous\_size=1000 took=“16.7µs” > time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/EIMe21t1Y/main.hnsw.commitlog.d/1707408396.condensed input\_second=/var/lib/weaviate/paragraph/EIMe21t1Y/main.hnsw.commitlog.d/1717543802.condensed output=/var/lib/weaviate/paragraph/EIMe21t1Y/main.hnsw.commitlog.d/1707408396 > time=“2024-06-05T23:37:16Z” level=debug msg=“index grown from 1000 to 3000, took 7.367µs\n” action=hnsw\_grow\_index new\_size=3000 previous\_size=1000 took=“7.367µs” > time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/3WN2TgubH/main.hnsw.commitlog.d/1704912524.condensed input\_second=/var/lib/weaviate/paragraph/3WN2TgubH/main.hnsw.commitlog.d/1717543802.condensed output=/var/lib/weaviate/paragraph/3WN2TgubH/main.hnsw.commitlog.d/1704912524 > time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 1000 to 3000, took 7.489µs\n” action=hnsw\_grow\_index new\_size=3000 previous\_size=1000 took=“7.489µs” > time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 3000 to 5000, took 12.781µs\n” action=hnsw\_grow\_index new\_size=5000 previous\_size=3000 took=“12.781µs” > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/0dj819\_9W/main.hnsw.commitlog.d/1717430326.condensed input\_second=/var/lib/weaviate/paragraph/0dj819\_9W/main.hnsw.commitlog.d/1717543804.condensed output=/var/lib/weaviate/paragraph/0dj819\_9W/main.hnsw.commitlog.d/1717430326 > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/MytQNGKCU/main.hnsw.commitlog.d/1717430329.condensed input\_second=/var/lib/weaviate/paragraph/MytQNGKCU/main.hnsw.commitlog.d/1717543807.condensed output=/var/lib/weaviate/paragraph/MytQNGKCU/main.hnsw.commitlog.d/1717430329 > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/jXEtkK6Lo/main.hnsw.commitlog.d/1705398870.condensed input\_second=/var/lib/weaviate/paragraph/jXEtkK6Lo/main.hnsw.commitlog.d/1717543805.condensed output=/var/lib/weaviate/paragraph/jXEtkK6Lo/main.hnsw.commitlog.d/1705398870 > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/KNsGbfcim/main.hnsw.commitlog.d/1715011941.condensed input\_second=/var/lib/weaviate/paragraph/KNsGbfcim/main.hnsw.commitlog.d/1717543804.condensed output=/var/lib/weaviate/paragraph/KNsGbfcim/main.hnsw.commitlog.d/1715011941 > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/5dvWnkeFj/main.hnsw.commitlog.d/1704142202.condensed input\_second=/var/lib/weaviate/paragraph/5dvWnkeFj/main.hnsw.commitlog.d/1717543808.condensed output=/var/lib/weaviate/paragraph/5dvWnkeFj/main.hnsw.commitlog.d/1704142202 > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/5-pDBLHuY/main.hnsw.commitlog.d/1710055918.condensed input\_second=/var/lib/weaviate/paragraph/5-pDBLHuY/main.hnsw.commitlog.d/1717543808.condensed output=/var/lib/weaviate/paragraph/5-pDBLHuY/main.hnsw.commitlog.d/1710055918 > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/NJ48e1XPK/main.hnsw.commitlog.d/1705074446.condensed input\_second=/var/lib/weaviate/paragraph/NJ48e1XPK/main.hnsw.commitlog.d/1717543808.condensed output=/var/lib/weaviate/paragraph/NJ48e1XPK/main.hnsw.commitlog.d/1705074446 > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/IPcExqqqs/main.hnsw.commitlog.d/1710786691.condensed input\_second=/var/lib/weaviate/paragraph/IPcExqqqs/main.hnsw.commitlog.d/1717543807.condensed output=/var/lib/weaviate/paragraph/IPcExqqqs/main.hnsw.commitlog.d/1710786691 > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/t0uC7EQO\_/main.hnsw.commitlog.d/1703449657.condensed input\_second=/var/lib/weaviate/paragraph/t0uC7EQO\_/main.hnsw.commitlog.d/1717543806.condensed output=/var/lib/weaviate/paragraph/t0uC7EQO\_/main.hnsw.commitlog.d/1703449657 > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/sMeBdLpr3/main.hnsw.commitlog.d/1710765984.condensed input\_second=/var/lib/weaviate/paragraph/sMeBdLpr3/main.hnsw.commitlog.d/1717543802.condensed output=/var/lib/weaviate/paragraph/sMeBdLpr3/main.hnsw.commitlog.d/1710765984 > time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 1000 to 3000, took 9.416µs\n” action=hnsw\_grow\_index new\_size=3000 previous\_size=1000 took=“9.416µs” > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/p-mHtzUkl/main.hnsw.commitlog.d/1703740170.condensed input\_second=/var/lib/weaviate/paragraph/p-mHtzUkl/main.hnsw.commitlog.d/1717543805.condensed output=/var/lib/weaviate/paragraph/p-mHtzUkl/main.hnsw.commitlog.d/1703740170 > time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 1000 to 3000, took 7.466µs\n” action=hnsw\_grow\_index new\_size=3000 previous\_size=1000 took=“7.466µs” > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/412nd32ou/main.hnsw.commitlog.d/1703449657.condensed input\_second=/var/lib/weaviate/paragraph/412nd32ou/main.hnsw.commitlog.d/1717543805.condensed output=/var/lib/weaviate/paragraph/412nd32ou/main.hnsw.commitlog.d/1703449657 > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/4JneooxUp/main.hnsw.commitlog.d/1704364995.condensed input\_second=/var/lib/weaviate/paragraph/4JneooxUp/main.hnsw.commitlog.d/1717543803.condensed output=/var/lib/weaviate/paragraph/4JneooxUp/main.hnsw.commitlog.d/1704364995 > time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw\_commit\_logger\_combine\_condensed\_logs id=main input\_first=/var/lib/weaviate/paragraph/PTZRqcs-2/main.hnsw.commitlog.d/1713163422.condensed input\_second=/var/lib/weaviate/paragraph/PTZRqcs-2/main.hnsw.commitlog.d/1717543805.condensed output=/var/lib/weaviate/paragraph/PTZRqcs-2/main.hnsw.commitlog.d/1713163422 > time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 1000 to 3000, took 8.497µs\n” action=hnsw\_grow\_index new\_size=3000 previous\_size=1000 took=“8.497µs” > time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 1000 to 3000, took 10.126µs\n” action=hnsw\_grow\_index new\_size=3000 previous\_size=1000 took=“10.126µs” > time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 3000 to 5000, took 20.506µs\n” action=hnsw\_grow\_index new\_size=5000 previous\_size=3000 took=“20.506µs” ---
**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\ **Post date:** [June 7, 2024, 6:28pm UTC](https://forum.weaviate.io/t/weaviate-increasing-memory-usage-after-a-successful-backup-daily/2633/2 "2024-06-07T18:28:33Z")
hi @Divyansh_Mishra ! It would be interesting to check if this still happens on latest 1.25.X version or at least at 1.24.7 (latest 1.24.X patch as I write). A lot has changed, since the version you are running. I even recall some memory leakage fix at some point, and this for sure smells like one 🙈 Let me know if this helps. Thanks!