I'm creating incremental backups for a database. The size is around 8 GB per dump.
I have a backup creation policy, that looks like this: (don't worry, I already avoided collisions)
- An increment every hour, every day. (saved in
/hourly) - An increment every day, at midnight. (saved in
/daily) - An increment every Sunday, at midnight. (saved in
/weekly) - An increment every 1st of the month, at midnight. (saved in
/monthly)
All of these backups are saved in a S3 bucket.
The thing is, i need to maximize efficiency when creating and restoring to duplicity.
For example:
A dump takes approximately 30/40 minutes to restore. This is because my backup chain is almost a week old, and it has to download all the chain, decrypt and decompress each of the increment, one by one.
I tried doing the same with the backups that are done daily, and since less than a week happened, duplicity only had to download a few files, do the rest (decompress, decrypt), and voilà, it took 5 minutes, which is acceptable.
My idea is to create a new backup chain every week, but the last few days of the week, will have the most increments, thus, the slowest restore time.
I need some help to improve this time as much as it's possible, or at least, just improve it. While reading the man page, i came across some interesting flags, such as --copy-blocksize, --full-if-older-than (to create a new chain every week), --max-blocksize, --s3-multipart-chunk-size, --s3-multipart-max-procs, --s3-unencrypted-connection (maybe?).
What I'm asking is for advice to improve the time to restore a file with duplicity. Currently, I'm doing this, to restore the last backup available.
PASSPHRASE="<redacted>" duplicity restore s3://<redacted>/daily/ database-daily.sql
Also, an important fact is the hardware that I'm running this on:
Server Type: ccx43 (ID: 99)
ID: 99
Name: ccx43
Description: CCX43 Dedicated CPU
Cores: 16
CPU Type: dedicated
Memory: 64 GB
Disk: 160 GB
Storage Type: local
OS flavor: debian
OS version: 12
