OK, here is what I do now. Crappy as it is.
- A daily cron job lists all snapshots on the repository server and emails output to myself. This way I can spot it if a backup job is no longer running (the dates don’t increase) which I can see in the blink of an eye every day or so.
- Daily cron job rclones the whole # to both MSFT and BackBlaze. Reused a simple script from someone else to encapsulate the rclone action.
root@repositoryserver:~# cat /etc/cron.daily/kopia
#!/bin/bash
# crappy backup report temporarily
kopia snapshot list --max-results=1 --all | grep -v -B1 `date "+%Y-%m-%d"` | mail --subject="kopiarepoadminuser@kopia backup report" -a "From: root at repositoryserver <my.source@email.address>" my.destination@email.address
sample output:
user@host1:C:\Users
2021-12-08 18:28:14 CET k29590b6b403353d2820205c43d95ea1c3cee1ee8c6693f5b2 76.3 GB dr-xr-xr-x files:39944 dirs:7768 errors:1 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)
user@host2:/boot/grub
2021-12-08 21:00:00 CET k2f5fabe81720f5349ee3a6b4feb5740e7d58c0a6b3c308f21 6.8 MB drwxr-xr-x files:293 dirs:4 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)
[...]
For backblaze b2 (note the additional options to save cost and increase performance)
#!/bin/bash
##############################################################################
# An rclone backup script by Jared Males (jaredmales@gmail.com)
#
# Copyright (C) 2018 Jared Males <jaredmales@gmail.com>
#
# This script is licensed under the terms of the MIT license.
# https://opensource.org/licenses/MIT
#
# Runs the 'rclone sync' command. Designed to be used as a cron job.
#
# 1) Backup Source
# Edit the SRC variable below to point to the directory you want to backup.
#
# 2) Backup Destination
# Edit the DEST variable to point to the remote and location (see rclone docs).
#
# 3) Excluding files and directories
# Edit the EXCLUDEFILE variable below to point to a file listing files and directories to exclude.
# See the rclone docs for the format.
#
# Also, any directory can be excluded by adding an '.rclone-ignore' file to it without editing the exclude file.
# This file can be empty. You can edit the name of this file with EXIFPRESENT below.
#
# 4) You can change the bandwidth limits by editing BWLIMT, which includes a timetable facility.
# See rclone docs for more info.
#
# 5) Logs:
# -- The output of rclone is written to the location specified by LOGFILE. This is rotated with savelog.
# The details of synclog can be edited.
# -- The log rotation, and start and stop times of this script, are written to the location specified by CRONLOG.
# This isn't yet rotated, probably should be based on size.
#
##############################################################################
#### rclone sync options
RCLONEDATA=/opt/bcp/rclone
SRC=/opt/bcp/kopia
#---- Edit this to the desired destination
DEST=bucketname:/folderinbucket
#---- This is the path to a file with a list of exclude rules
EXCLUDEFILE=$RCLONEDATA/excludes
#---- Name of exclude file
# NOTE: you need "v1.39-036-g2030dc13β" or later for this to work.
EXIFPRESENT=rclone-ignore
#---- The bandwidth time table https://rclone.org/docs/#bwlimit-bandwidth-spec
BWLIMIT="07:00,5M 23:30,off Sat-00:01,5M"
#---- Don't sync brand new stuff, possible partials, etc.
MINAGE=15m
#---- [B2 Specific] number of transfers to do in parallel. rclone docs say 32 is recommended for B2.
TRANSFERS=32
#---- Additional options
# Note: bucket storage specific (S3, B2, GCS, Swift, Hubic] uses fewer transactions (pay less) at the cost of more memory https://rclone.org/docs/#fast-list
OPTIONS="--fast-list"
#---- Location of sync log [will be rotated with savelog]
LOGFILE=$RCLONEDATA/rclone-sync.log
LOGS='-vv --log-file='$LOGFILE
#---- Location of cron log
CRONLOG=$RCLONEDATA/rclone-cron.log
###################################################
## Locking Boilerplate from https://gist.github.com/przemoc/571091
## Included under MIT License:
###################################################
## Copyright (C) 2009 Przemyslaw Pawelczyk <przemoc@gmail.com>
##
## This script is licensed under the terms of the MIT license.
## https://opensource.org/licenses/MIT
#
# Lockable script boilerplate
### HEADER ###
LOCKFILE="/tmp/`basename $0`"
LOCKFD=99
# PRIVATE
_lock() { flock -$1 $LOCKFD; }
_no_more_locking() { _lock u; _lock xn && rm -f $LOCKFILE; }
_prepare_locking() { eval "exec $LOCKFD>\"$LOCKFILE\""; trap _no_more_locking EXIT; }
# ON START
_prepare_locking
# PUBLIC
exlock_now() { _lock xn; } # obtain an exclusive lock immediately or fail
exlock() { _lock x; } # obtain an exclusive lock
shlock() { _lock s; } # obtain a shared lock
unlock() { _lock u; } # drop a lock
###################################################
# End of locking code from Pawelczyk
###################################################
#make a log entry if we exit because locked
exit_on_lock() { echo $(date)" | `basename $0` already running." >> $CRONLOG; exit 1; }
#Now check for lock
exlock_now || exit_on_lock
#We now have the lock.
#Rotate logs.
savelog -n -c 7 $LOGFILE >> $CRONLOG
#Log startup
echo $(date)" | `basename $0` starting . . . " >> $CRONLOG
#Now do the sync!
rclone sync $SRC $DEST $OPTIONS --transfers $TRANSFERS --bwlimit "$BWLIMIT" --min-age $MINAGE --exclude-from $EXCLUDEFILE --exclude-if-present $EXIFPRESENT --delete-excluded $LOGS
#log success
echo $(date)" | `basename $0` completed." >> $CRONLOG
# Mail today's loglines
TODAY="$(date +'%a %d %b %G')"
grep "$TODAY" $CRONLOG | grep -v Rotated | mail --subject="reposerver $TODAY rclone sync $SRC $DEST" -a "From: root at reposerver <my.source@email.address>" my.destination@email.address
#release the lock
unlock
exit
Similar for onedrive:
#!/bin/bash
##############################################################################
# An rclone backup script by Jared Males (jaredmales@gmail.com)
#
# Copyright (C) 2018 Jared Males <jaredmales@gmail.com>
#
# This script is licensed under the terms of the MIT license.
# https://opensource.org/licenses/MIT
#
# Runs the 'rclone sync' command. Designed to be used as a cron job.
#
# 1) Backup Source
# Edit the SRC variable below to point to the directory you want to backup.
#
# 2) Backup Destination
# Edit the DEST variable to point to the remote and location (see rclone docs).
#
# 3) Excluding files and directories
# Edit the EXCLUDEFILE variable below to point to a file listing files and directories to exclude.
# See the rclone docs for the format.
#
# Also, any directory can be excluded by adding an '.rclone-ignore' file to it without editing the exclude file.
# This file can be empty. You can edit the name of this file with EXIFPRESENT below.
#
# 4) You can change the bandwidth limits by editing BWLIMT, which includes a timetable facility.
# See rclone docs for more info.
#
# 5) Logs:
# -- The output of rclone is written to the location specified by LOGFILE. This is rotated with savelog.
# The details of synclog can be edited.
# -- The log rotation, and start and stop times of this script, are written to the location specified by CRONLOG.
# This isn't yet rotated, probably should be based on size.
#
##############################################################################
#### rclone sync options
RCLONEDATA=/opt/bcp/rclone
SRC=/opt/bcp/kopia
#---- Edit this to the desired destination
DEST=email_outlook_com:/kopia
#---- This is the path to a file with a list of exclude rules
EXCLUDEFILE=$RCLONEDATA/excludes
#---- Name of exclude file
# NOTE: you need "v1.39-036-g2030dc13β" or later for this to work.
EXIFPRESENT=rclone-ignore
#---- The bandwidth time table https://rclone.org/docs/#bwlimit-bandwidth-spec
BWLIMIT="07:00,5M 23:30,off Sat-00:01,5M"
#---- Don't sync brand new stuff, possible partials, etc.
MINAGE=15m
#---- [B2 Specific] number of transfers to do in parallel. rclone docs say 32 is recommended for B2.
TRANSFERS=32
#---- Additional options
# Note: bucket storage specific (S3, B2, GCS, Swift, Hubic] uses fewer transactions (pay less) at the cost of more memory https://rclone.org/docs/#fast-list
OPTIONS="--fast-list"
#---- Location of sync log [will be rotated with savelog]
LOGFILE=$RCLONEDATA/rclone-sync.log
LOGS='-vv --log-file='$LOGFILE
#---- Location of cron log
CRONLOG=$RCLONEDATA/rclone-cron.log
###################################################
## Locking Boilerplate from https://gist.github.com/przemoc/571091
## Included under MIT License:
###################################################
## Copyright (C) 2009 Przemyslaw Pawelczyk <przemoc@gmail.com>
##
## This script is licensed under the terms of the MIT license.
## https://opensource.org/licenses/MIT
#
# Lockable script boilerplate
### HEADER ###
LOCKFILE="/tmp/`basename $0`"
LOCKFD=99
# PRIVATE
_lock() { flock -$1 $LOCKFD; }
_no_more_locking() { _lock u; _lock xn && rm -f $LOCKFILE; }
_prepare_locking() { eval "exec $LOCKFD>\"$LOCKFILE\""; trap _no_more_locking EXIT; }
# ON START
_prepare_locking
# PUBLIC
exlock_now() { _lock xn; } # obtain an exclusive lock immediately or fail
exlock() { _lock x; } # obtain an exclusive lock
shlock() { _lock s; } # obtain a shared lock
unlock() { _lock u; } # drop a lock
###################################################
# End of locking code from Pawelczyk
###################################################
#make a log entry if we exit because locked
exit_on_lock() { echo $(date)" | `basename $0` already running." >> $CRONLOG; exit 1; }
#Now check for lock
exlock_now || exit_on_lock
#We now have the lock.
#Rotate logs.
savelog -n -c 7 $LOGFILE >> $CRONLOG
#Log startup
echo $(date)" | `basename $0` starting . . . " >> $CRONLOG
#Now do the sync!
rclone sync $SRC $DEST $OPTIONS --transfers $TRANSFERS --bwlimit "$BWLIMIT" --min-age $MINAGE --exclude-from $EXCLUDEFILE --exclude-if-present $EXIFPRESENT --delete-excluded $LOGS
#log success
echo $(date)" | `basename $0` completed." >> $CRONLOG
# Mail today's loglines
TODAY="$(date +'%a %d %b %G')"
grep "$TODAY" $CRONLOG | grep -v Rotated | mail --subject="reposerver $TODAY rclone sync $SRC $DEST" -a "From: root at reposerver <my.source@email.address>" my.destination@email.address
#release the lock
unlock
exit
Sample output:
repositoryserver Thu 09 Dec 2021 rclone sync /opt/bcp/kopia bucketname:/bucketfolder
Sample output:
kopiarepoadmin@kopia backup report
Sun 21 Nov 2021 11:28:08 PM CET | rclone-user-outlook-com starting . . .
Sun 21 Nov 2021 11:44:36 PM CET | rclone-user-outlook-com completed.
Sun 21 Nov 2021 11:44:36 PM CET | rclone-b2 starting . . .
Sun 21 Nov 2021 11:46:08 PM CET | rclone-b2 completed.
