Description
We have the backup s3 module enabled and after every backup, the memory usage increases by 3-5GB daily; one day, the spike was 10GB.
The memory usage normally for us is around 20GB and while taking a backup it goes up to around 40GB and comes back but the delta increase (The memory at which it came back after the spike) is increasing daily. Currently, the weaviate uses 26GB of memory since it’s increasing daily after the backup process.
Server Setup Information
- Weaviate Server Version: 1.23.6
- Deployment Method: k8s
- Multi Node? Number of Running Nodes: Single node
- Client Language and Version: Python V3
- Multitenancy: enabled
Spike in memory during the backup process: (It increases daily after the backup)
Logs when the backup process started
time=“2024-06-05T23:30:01Z” level=trace msg=“no segment eligible for compaction” action=lsm_compaction class=Paragraph index=paragraph path=/var/lib/weaviate/paragraph/NiIyjUVQT/lsm/property_source_searchable shard=NiIyjUVQT
time=“2024-06-05T23:30:01Z” level=trace msg=“no segment eligible for compaction” action=lsm_compaction class=Paragraph index=paragraph path=/var/lib/weaviate/paragraph/NiIyjUVQT/lsm/property_content_searchable shard=NiIyjUVQT
BACKUP STARTED BELOW
time=“2024-06-05T23:30:01Z” level=debug msg=“received HTTP request” action=restapi_request method=POST url=/v1/backups/s3
time=“2024-06-05T23:30:02Z” level=info action=try_backup backend=s3 backup_id=05-06-2024_23-30-01-k8s took=436.881717ms
time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit_log_file_switched id=main new_file_name=1717630202 old_file_name=1717543806 old_file_size=0
time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit_log_file_switched id=main new_file_name=1717630202 old_file_name=1717543803 old_file_size=0
time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit_log_file_switched id=main new_file_name=1717630202 old_file_name=1717543803 old_file_size=0
time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit_log_file_switched id=main new_file_name=1717630202 old_file_name=1717543803 old_file_size=0
time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit_log_file_switched id=main new_file_name=1717630202 old_file_name=1717543807 old_file_size=0
time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit_log_file_switched id=main new_file_name=1717630202 old_file_name=1717543808 old_file_size=0
time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit_log_file_switched id=main new_file_name=1717630202 old_file_name=1717543804 old_file_size=0
time=“2024-06-05T23:30:02Z” level=debug msg=“commit log switched forced” action=commit_log_file_switched id=main new_file_name=1717630202 old_file_name=1717543808 old_file_size=0
AFTER BACKUP
time=“2024-06-05T23:37:14Z” level=debug msg=“received HTTP request” action=restapi_request method=GET url=/v1/meta
time=“2024-06-05T23:37:16Z” level=info msg=“finish uploading files” class=Paragraph
time=“2024-06-05T23:37:16Z” level=info msg=“start uploading meta data”
time=“2024-06-05T23:37:16Z” level=debug msg=starting class=Paragraph id=05-06-2024_23-30-01-k8s op=release_backup
time=“2024-06-05T23:37:16Z” level=info msg=“finish uploading meta data”
time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/xdoDw-Vfi/main.hnsw.commitlog.d/1703479198.condensed input_second=/var/lib/weaviate/paragraph/xdoDw-Vfi/main.hnsw.commitlog.d/1717543802.condensed output=/var/lib/weaviate/paragraph/xdoDw-Vfi/main.hnsw.commitlog.d/1703479198
time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/htBlqqM9C/main.hnsw.commitlog.d/1713163635.condensed input_second=/var/lib/weaviate/paragraph/htBlqqM9C/main.hnsw.commitlog.d/1717543803.condensed output=/var/lib/weaviate/paragraph/htBlqqM9C/main.hnsw.commitlog.d/1713163635
time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/LnVcb310_/main.hnsw.commitlog.d/1707579378.condensed input_second=/var/lib/weaviate/paragraph/LnVcb310_/main.hnsw.commitlog.d/1717543807.condensed output=/var/lib/weaviate/paragraph/LnVcb310_/main.hnsw.commitlog.d/1707579378
time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/K2XWGL9oK/main.hnsw.commitlog.d/1708499782.condensed input_second=/var/lib/weaviate/paragraph/K2XWGL9oK/main.hnsw.commitlog.d/1717543803.condensed output=/var/lib/weaviate/paragraph/K2XWGL9oK/main.hnsw.commitlog.d/1708499782
time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/D8_MRzWnm/main.hnsw.commitlog.d/1704735872.condensed input_second=/var/lib/weaviate/paragraph/D8_MRzWnm/main.hnsw.commitlog.d/1717543803.condensed output=/var/lib/weaviate/paragraph/D8_MRzWnm/main.hnsw.commitlog.d/1704735872
time=“2024-06-05T23:37:16Z” level=debug msg=“index grown from 1000 to 7044, took 16.7µs\n” action=hnsw_grow_index new_size=7044 previous_size=1000 took=“16.7µs”
time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/EIMe21t1Y/main.hnsw.commitlog.d/1707408396.condensed input_second=/var/lib/weaviate/paragraph/EIMe21t1Y/main.hnsw.commitlog.d/1717543802.condensed output=/var/lib/weaviate/paragraph/EIMe21t1Y/main.hnsw.commitlog.d/1707408396
time=“2024-06-05T23:37:16Z” level=debug msg=“index grown from 1000 to 3000, took 7.367µs\n” action=hnsw_grow_index new_size=3000 previous_size=1000 took=“7.367µs”
time=“2024-06-05T23:37:16Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/3WN2TgubH/main.hnsw.commitlog.d/1704912524.condensed input_second=/var/lib/weaviate/paragraph/3WN2TgubH/main.hnsw.commitlog.d/1717543802.condensed output=/var/lib/weaviate/paragraph/3WN2TgubH/main.hnsw.commitlog.d/1704912524
time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 1000 to 3000, took 7.489µs\n” action=hnsw_grow_index new_size=3000 previous_size=1000 took=“7.489µs”
time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 3000 to 5000, took 12.781µs\n” action=hnsw_grow_index new_size=5000 previous_size=3000 took=“12.781µs”
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/0dj819_9W/main.hnsw.commitlog.d/1717430326.condensed input_second=/var/lib/weaviate/paragraph/0dj819_9W/main.hnsw.commitlog.d/1717543804.condensed output=/var/lib/weaviate/paragraph/0dj819_9W/main.hnsw.commitlog.d/1717430326
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/MytQNGKCU/main.hnsw.commitlog.d/1717430329.condensed input_second=/var/lib/weaviate/paragraph/MytQNGKCU/main.hnsw.commitlog.d/1717543807.condensed output=/var/lib/weaviate/paragraph/MytQNGKCU/main.hnsw.commitlog.d/1717430329
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/jXEtkK6Lo/main.hnsw.commitlog.d/1705398870.condensed input_second=/var/lib/weaviate/paragraph/jXEtkK6Lo/main.hnsw.commitlog.d/1717543805.condensed output=/var/lib/weaviate/paragraph/jXEtkK6Lo/main.hnsw.commitlog.d/1705398870
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/KNsGbfcim/main.hnsw.commitlog.d/1715011941.condensed input_second=/var/lib/weaviate/paragraph/KNsGbfcim/main.hnsw.commitlog.d/1717543804.condensed output=/var/lib/weaviate/paragraph/KNsGbfcim/main.hnsw.commitlog.d/1715011941
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/5dvWnkeFj/main.hnsw.commitlog.d/1704142202.condensed input_second=/var/lib/weaviate/paragraph/5dvWnkeFj/main.hnsw.commitlog.d/1717543808.condensed output=/var/lib/weaviate/paragraph/5dvWnkeFj/main.hnsw.commitlog.d/1704142202
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/5-pDBLHuY/main.hnsw.commitlog.d/1710055918.condensed input_second=/var/lib/weaviate/paragraph/5-pDBLHuY/main.hnsw.commitlog.d/1717543808.condensed output=/var/lib/weaviate/paragraph/5-pDBLHuY/main.hnsw.commitlog.d/1710055918
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/NJ48e1XPK/main.hnsw.commitlog.d/1705074446.condensed input_second=/var/lib/weaviate/paragraph/NJ48e1XPK/main.hnsw.commitlog.d/1717543808.condensed output=/var/lib/weaviate/paragraph/NJ48e1XPK/main.hnsw.commitlog.d/1705074446
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/IPcExqqqs/main.hnsw.commitlog.d/1710786691.condensed input_second=/var/lib/weaviate/paragraph/IPcExqqqs/main.hnsw.commitlog.d/1717543807.condensed output=/var/lib/weaviate/paragraph/IPcExqqqs/main.hnsw.commitlog.d/1710786691
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/t0uC7EQO_/main.hnsw.commitlog.d/1703449657.condensed input_second=/var/lib/weaviate/paragraph/t0uC7EQO_/main.hnsw.commitlog.d/1717543806.condensed output=/var/lib/weaviate/paragraph/t0uC7EQO_/main.hnsw.commitlog.d/1703449657
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/sMeBdLpr3/main.hnsw.commitlog.d/1710765984.condensed input_second=/var/lib/weaviate/paragraph/sMeBdLpr3/main.hnsw.commitlog.d/1717543802.condensed output=/var/lib/weaviate/paragraph/sMeBdLpr3/main.hnsw.commitlog.d/1710765984
time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 1000 to 3000, took 9.416µs\n” action=hnsw_grow_index new_size=3000 previous_size=1000 took=“9.416µs”
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/p-mHtzUkl/main.hnsw.commitlog.d/1703740170.condensed input_second=/var/lib/weaviate/paragraph/p-mHtzUkl/main.hnsw.commitlog.d/1717543805.condensed output=/var/lib/weaviate/paragraph/p-mHtzUkl/main.hnsw.commitlog.d/1703740170
time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 1000 to 3000, took 7.466µs\n” action=hnsw_grow_index new_size=3000 previous_size=1000 took=“7.466µs”
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/412nd32ou/main.hnsw.commitlog.d/1703449657.condensed input_second=/var/lib/weaviate/paragraph/412nd32ou/main.hnsw.commitlog.d/1717543805.condensed output=/var/lib/weaviate/paragraph/412nd32ou/main.hnsw.commitlog.d/1703449657
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/4JneooxUp/main.hnsw.commitlog.d/1704364995.condensed input_second=/var/lib/weaviate/paragraph/4JneooxUp/main.hnsw.commitlog.d/1717543803.condensed output=/var/lib/weaviate/paragraph/4JneooxUp/main.hnsw.commitlog.d/1704364995
time=“2024-06-05T23:37:17Z” level=info msg=“successfully combined previously condensed commit log files” action=hnsw_commit_logger_combine_condensed_logs id=main input_first=/var/lib/weaviate/paragraph/PTZRqcs-2/main.hnsw.commitlog.d/1713163422.condensed input_second=/var/lib/weaviate/paragraph/PTZRqcs-2/main.hnsw.commitlog.d/1717543805.condensed output=/var/lib/weaviate/paragraph/PTZRqcs-2/main.hnsw.commitlog.d/1713163422
time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 1000 to 3000, took 8.497µs\n” action=hnsw_grow_index new_size=3000 previous_size=1000 took=“8.497µs”
time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 1000 to 3000, took 10.126µs\n” action=hnsw_grow_index new_size=3000 previous_size=1000 took=“10.126µs”
time=“2024-06-05T23:37:17Z” level=debug msg=“index grown from 3000 to 5000, took 20.506µs\n” action=hnsw_grow_index new_size=5000 previous_size=3000 took=“20.506µs”
