September 7, 2022 16:20
Note
Some operations will need a lot power or disk space, so consult man borg
for detailed information!
Make sure to have at least 1G+ free disk space for larger archives (bigger repository means more free space needed)!
To disable backup deletion see: ‘https://borgbackup.readthedocs.io/en/stable/usage/notes.html#append-only-mode'
You can add to nearly to any command:
-s
Show statistics at finish-p
Little information while working…--dry-run
Well, you know…
Legend for this
TARGET
Means the folders path (sometimes mentioned as repository) or an ssh-connection-folder (e.g.ssh://([USER]@)[IP(:PORT)]/~/[PATH_IN_USERDIR](/$HOSTNAME)
)NAME
Means the archives name - maybe a$(date)
is useful…[?]
MUST filled(?)
CAN filled
Init repository
Create a borg folder/repository
borg init -e repokey TARGET
-e [MODE]
Specifies the encryption; ‘keyfile’ is good (it will ask for the password every time) - you have the key under ~/.config/borg/keys/, ‘repokey’ is DANGEROUS WITHOUT PASSWORD (it will still ask for the password every time) - the key will be saved (with password encrypted) inside the repositories configuration, ’none’ is… yeah…--append-only
Prohibits deletion of archives--storage-quota [QUOTA]
Set storage quota of the new repository (e.g. 5G, 1.5T) - useful to make sure to be able to delete afterwards…
First Backup
Create an archive
borg create -C none TARGET::NAME [NOW MULTIPLE FOLDERS TO INCLUDE]
-C [MODE]
Sets compression level;none
~;zlib
is medium speed and compression;lzma
is slow but best;zlib,[COMPRESSIONLEVEL]
andlzma,[COMPRESSIONLEVEL]
are available too[0,9]
--lock-wait [SECONDS]
Maybe inside a script which fires multiple creations - to proccess the timeout on connection losses
Commands…
borg check TARGET::NAME
Rechecks the archives integrity - useful to determine the size the dataloss after a drive failureborg check --repair TARGET::NAME
ONLY IF NECCESSARY… THIS WILL MINIMIZE DATA LOSS ON DAMAGED FILES BUT NOT FIXborg list TARGET
Lists all available archivesborg info TARGET::NAME
Archive information…borg prune TARGET::NAME
Removes archiveborg prune TARGET --keep-daily=7 --keep-weekly=4 --keep-monthly=6 --keep-yearly 2
Cleans the repository up…borg change-passphrase TARGET
Changes keyfiles passwordborg mount TARGET::NAME [DIR]
Mounts the archive for easier operations…borg umount [dir]
Unmounts the archive…borg key export TARGET [PATH]
Backup the encryption key of the repositoryborg key import TARGET [PATH]
Restores the encryption key of the repository (useful with keyfile encrytion)borg break-lock TARGET
In case borg cant finish the backup, you’ll need to release the lock manuallyborg extract TARGET::NAME [PATH]
Extracts the path from the archive to the current working directory
Useful…
export BORG_PASSPHRASE='[PASSWORD]'
Prevents the passwords request - BUT BE CAREFUL! THAT WILL BE SAVED E.G. TO THE BASH_HISTORYexport BORG_PASSCOMMAND='cat $HOME/.borg-passphrase'
Same as above, but reads the password e.g. from a file (or e.g. from zenity –password). ‘~’ doesn’t work.export BORG_KEY_FILE='[KEYFILE_PATH]'
Specifies the path for the keyfile, if the key is stored locally (if not using the repokey)export BORG_RSH='ssh -i [SSH_KEYFILE_PATH]'
Maybe neccessary, if ssh fails to authenticate automatically with the keyfiles under ~/.ssh
Universal backup script
The example is here ssh://server/./subdir
- meaning a remote repository. If you plan to use a local solution, you may omit the SSH parts.
The script expects a folder called backup_scripts
inside your home (you should run this as root anyways). Inside this folder you have to create a subfolder containing the backup target yaml-configuration (see below).
Prepare the repository
Lets call this example repository remote
located at ssh://server/./subdir
(as noted before).
# Remote only: Create a new ssh key
ssh-keygen -q -N '' -a 4096 -f ~/backup_scripts/remote/SSHKey.pem
# Remote only: Prepare the ssh-env-vars for repository creation
export BORG_RSH='ssh -i ~/backup_scripts/remote/SSHKey.pem'
# Remote only: View the public key and MAKE SURE TO INSTALL IT NOW on the target server!
cat ~/backup_scripts/remote/SSHKey.pem.pub
# Create a new encryption key
openssl rand -base64 4096 > ~/backup_scripts/remote/BORGKeyPassword.file
# Prepare the env-vars for repository creation
export BORG_PASSPHRASE=`cat ~/backup_scripts/remote/BORGKeyPassword.file`
# Create the repo!
borg init -e keyfile ssh://server/./subdir
# And make sure to have a key backup!!!
borg key export ssh://server/./subdir ~/backup_scripts/remote/BORGKey.bak
Prepare the config(s)
Put this into ~/backup_scripts/remote/config.yaml
:
backup:
- $HOME/backup_scripts/
target: 'ssh://server/./subdir'
options:
backup: '-C lzma,9'
cleanup: '--lock-wait 60'
check: ''
cleanup:
daily: 7
weekly: 4
monthly: 6
yearly: 2
tries:
backup: 3
cleanup: 3
check: 3
Finally the script…
…place it somewhere into the home folder - just make sure only the user can read and execute it (chmod 700
)!
#!/usr/bin/python3
import os, sys, yaml, time, logging, argparse, subprocess
logger = logging.getLogger(__name__)
# Base variables
configDirPath = os.path.join(os.path.expanduser('~'), 'backup_scripts')
# Parse args
parser = argparse.ArgumentParser()
parser.add_argument('--config', help='Set the configuration directory name (located at ' + configDirPath + ').', type=str, required=True)
parser.add_argument('--shell', help='Execute a shell with all environment variables set.', action='store_true')
parser.add_argument('--debug', help='Debug mode! Also add "-v" o the borg-command.', action='store_true')
parser.add_argument('--output', help='Do not capture the borg-command outputs.', action='store_true')
parser.add_argument('--nocreate', help='Skip archive creation.', action='store_true')
parser.add_argument('--nocleanup', help='Skip repository cleanup.', action='store_true')
parser.add_argument('--nocheck', help='Skip repository checks.', action='store_true')
parser.add_argument('--progress', help='Show progress by adding "-p" o the borg-command.', action='store_true')
args = parser.parse_args()
if args.debug:
logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', level=logging.DEBUG)
else:
logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', level=logging.INFO)
if args.debug and args.progress and not args.shell and not args.output:
logging.warning('Please note, that you won\'t see any progress (even in "--debug" mode) as the output is still captured. Use "--output" instead.')
# Make sure config dir is available
configDirPath = os.path.join(configDirPath, args.config)
if not os.path.isdir(configDirPath):
logger.critical('Config dir (' + configDirPath + ') not available!')
sys.exit(1)
# Read config vars
configFilePath = os.path.join(configDirPath, 'config.yaml')
if not os.path.isfile(configFilePath):
logger.critical('Config file (' + configFilePath + ') not available!')
sys.exit(2)
with open(configFilePath, 'r') as configFile:
configDict = yaml.safe_load(configFile)
configBackupThis = [os.path.expandvars(x) for x in configDict['backup']]
configBackupOptions = configDict['options']['backup'].split(' ') if configDict['options']['backup'] != '' else []
configCleanupOptions = configDict['options']['cleanup'].split(' ') if configDict['options']['cleanup'] != '' else []
configCheckOptions = configDict['options']['check'].split(' ') if configDict['options']['check'] != '' else []
if args.progress:
configBackupOptions.append('-p')
configCleanupOptions.append('-p')
configCheckOptions.append('-p')
if args.debug:
configBackupOptions.append('-v')
configCleanupOptions.append('-v')
configCheckOptions.append('-v')
# Make sure every backup target exists
if not args.nocreate:
for p in configBackupThis:
if not os.path.exists(p):
if not not args.shell:
logger.warning('Backup path (' + p + ') not available!')
else:
logger.critical('Backup path (' + p + ') not available!')
sys.exit(3)
# Prepare environment variables
os.environ['THIS_TARGET'] = configDict['target']
targetIsRemote = configDict['target'].startswith('ssh://')
if targetIsRemote:
sshKeyPath = os.path.join(configDirPath, 'SSHKey.pem')
os.environ['BORG_RSH'] = 'ssh -i ' + sshKeyPath
if not os.path.isfile(sshKeyPath):
logger.critical('SSH key (' + sshKeyPath + ') not available!')
sys.exit(4)
os.environ['BORG_KEY_FILE'] = os.path.join(configDirPath, 'BORGKey.bak')
# I tried to load the file in the BORG_PASSPHRASE variable, but Python does not seem to handle new lines consistently (sometimes they are just becoming spaces)
os.environ['BORG_PASSCOMMAND'] = 'cat "' + os.path.join(configDirPath, 'BORGKeyPassword.file') + '"'
os.environ['BACKUP_THIS'] = ' '.join(configBackupThis)
# Should we fire of a shell?
if args.shell:
logger.info('Following environment variables are set for you:')
vars = ['THIS_TARGET', 'BORG_PASSCOMMAND', 'BACKUP_THIS', 'BORG_KEY_FILE']
if targetIsRemote:
vars.append('BORG_RSH')
for v in sorted(vars):
logger.info('\t' + v)
logger.info('Try to execute "borg info $THIS_TARGET" to test the connection.')
logger.info('Starting shell...')
os.system(os.environ['SHELL'])
logger.info('Bye!')
else:
# Run other stuff - add your own backup commands here!
# Note this: https://borgbackup.readthedocs.io/en/stable/internals/frontends.html
# Therefore we are not importing borg here, but instead use the command line interface!
def runBorgCommand(cmnd, tries):
for tryNum in range(1, tries+1):
logger.debug(cmnd)
if args.output:
proc = subprocess.Popen(cmnd)
else:
proc = subprocess.Popen(cmnd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
stdout, _ = proc.communicate()
if proc.returncode != 0:
# Whoops!
logger.error('Something went wrong (try {} out of {}):'.format(tryNum, tries))
logger.error(stdout)
time.sleep(10)
else:
logger.debug(proc.returncode)
if not args.output:
logger.info(stdout)
break
if tryNum == tries:
logger.error('Cleanup failed.')
return tryNum != tries
# Create new archive
createOK = args.nocreate
if not args.nocreate:
cmnd = ['borg', 'create', '-s']
cmnd += configBackupOptions
cmnd += [configDict['target'] + '::' + str(time.time())]
cmnd += configBackupThis
logger.info('Backup started...')
createOK = runBorgCommand(cmnd, configDict['tries']['backup'])
# Cleanup
cleanOK = args.nocleanup
if not args.nocleanup:
cmnd = ['borg', 'prune', '-s']
cmnd += configCleanupOptions
cmnd += [configDict['target'], '--keep-daily=' + str(configDict['cleanup']['daily']), '--keep-weekly=' + str(configDict['cleanup']['weekly']), '--keep-monthly=' + str(configDict['cleanup']['monthly']), '--keep-yearly=' + str(configDict['cleanup']['yearly'])]
logger.info('Cleanup started...')
cleanOK = runBorgCommand(cmnd, configDict['tries']['cleanup'])
# Check (max-duration requires borgbackup version >= 1.2.0)
checkOK = args.nocheck
if not args.nocheck:
cmnd = ['borg', 'check', '--max-duration=' + str(60 * 60)] # Run archive checks for up to one hour
cmnd += configCheckOptions
cmnd += [configDict['target']]
logger.info('Cleanup started...')
checkOK = runBorgCommand(cmnd, configDict['tries']['check'])
if createOK and cleanOK and checkOK:
logger.info('Jobs successful finished.')
else:
logger.warning('At least one job failed.')
sys.exit(5)
The last lines should allow your Sieve-Filters to highlight any failed executions easily.
You should also make sure to install this script into your crontab (daily is recommended):
0 2 * * * /usr/bin/python3 "$HOME/backup_scripts/backup.py" --config remote