Introduction
Your Linux servers are running smoothly? Fine.
Now if something incidentally gets wrong you need also to prepare an emergency plan.
And even when everything is going fine, a backup that is directly stored on the same server can still be useful:
- for example when you need to see what was changed on a specific file,
- or if you want to find the list of files that were installed or modified after the installation of an application
This article is then intended to show you how you can set up an open-source solution, using the magic of the famous ‘rsync’ tool and some shell-scripting, to deploy a backup system without the need of investing into expensive proprietary software.
Another advantage of a shell-script is that you can easily adapt it according to your specific needs, for example for you DRP architecture.
The proposed shell-script is derivate from the great work of Mikes Handy’s rotating-filesystem-snapshot utility (cf. http://www.mikerubel.org/computers/rsync_snapshots).
It creates backups of your full filesystem (snapshots) with the combined advantages of full and incremental backups:
- It uses as little disk space as an incremental backup because all unchanged files are hard linked with existing files from previous backups; only the modified files require new inodes.
- Because of the transparency of hard links, all backups are directly available and always online for read access with your usual programs; there is no need to extract the files from a full archive and also no complicate replay of incremental archives is necessary.
It is capable of doing local (self) backups and it can also be run from a remote backup server to centralize all backups to a safe place and therefore avoid correlated physical risks.
‘rsync’ features tremendous optimizations of bandwidth usage and transfers only the portions of a file that were changed thanks to its brilliant algorithms, created by Andrew Tridgell. (cf. http://bryanpendleton.blogspot.ch/2010/05/rsync-algorithm.html)
‘rsync’ is also using network encryption via ‘ssh’.
The script let you achieve:
- local or remote backups with extremely low bandwidth requirement
- file level deduplication between backups using hard links (also across servers on the remote backup server)
- specify a bandwidth limit to moderate the network and I/O load on production servers
- backup retention policy:
- per server disk quota restrictions: for example never exceed 50GB and always keep 100GB of free disk
- rotation of backups with non-linear distribution, with the idea that recent backups are more useful than older, but that sometimes you still need a very old backup
- filter rules to include or exclude specific patterns of folders and files
- integrity protection, the backups have a ‘chattr’ read-only protection and a MD5 integrity signature can also be calculated incrementally
Installation
The snapshot backups are saved into the ‘/backup’ folder.
You can also create a symbolic link to point to another partition with more disk space, for example:
ln -sv mnt/bigdisk /backup
Then create the folders:
mkdir -pv /backup/snapshot/{$(hostname -s),rsync,md5-log}
[ -h /backup/snapshot/localhost ] || ln -vs $(hostname -s) /backup/snapshot/localhost
Now create the shell-script ‘/backup/snapshot/rsync/rsync-snapshot.sh’ (download rsync-snapshot.sh):
#!/bin/bash
# ----------------------------------------------------------------------
# created by francois scheurer on 20070323
# derivate from mikes handy rotating-filesystem-snapshot utility
# see http://www.mikerubel.org/computers/rsync_snapshots
# ----------------------------------------------------------------------
#rsync note:
# 1) rsync -avz /src/foo /dest => ok, creates /dest/foo, like cp -a /src/foo /dest
# 2) rsync -avz /src/foo/ /dest/foo => ok, creates /dest/foo, like cp -a /src/foo/. /dest/foo (or like cp -a /src/foo /dest)
# 3) rsync -avz /src/foo/ /dest/foo/ => ok, same as 2)
# 4) rsync -avz /src/foo/ /dest => dangerous!!! overwrite dest content, like cp -a /src/foo/. /dest
# solution: remove trailing / at /src/foo/ => 1)
# minor problem: rsync -avz /src/foo /dest/foo => creates /dest/foo/foo, like mkdir /dest/foo && cp -a /src/foo /dest/foo
# main options:
# -H --hard-links
# -x --one-file-system
# -a equals -rlptgoD (no -H,-A,-X)
# -r --recursive
# -l --links
# -p --perms
# -t --times
# -g --group
# -o --owner
# -D --devices --specials
# useful options:
# -S --sparse
# -n --dry-run
# -I --ignore-times
# -c --checksum
# -z --compress
# -bwlimit=X limit disk IO to X kB/s
# other options:
# -v --verbose
# -y --fuzzy
# --stats
# -h --human-readable
# --progress
# -i --itemize-changes
# quickcheck options:
# the default behavior is to skip files with same size & mtime on destination
# mtime = last data write access
# atime = last data read access (can be ignored with noatime mount option or with chattr +A)
# ctime = last inode change (write access, change of permission or ownership)
# note that a checksum is always done after a file synchronization/transfer
# --modify-window=X ignore mtime differences less or equal to X sec
# --size-only skip files with same size on destination (ignore mtime)
# -c --checksum skip files with same MD5 checksum on destination (ignore size & mtime, all files are read once, then the list of files to be resynchronized is read a second time, there is a lot of disk IO but network trafic is minimal if many files are identical, log includes only different files)
# -I --ignore-times never skip files (all files are resynchronized, all files are read once, there is more network trafic than with --checksum but less disk IO, log includes all files)
# --link-dest does the quickcheck on another reference-directory and makes hardlinks if quickcheck succeeds
# (however, if mtime is different and --perms is used, the reference file is copied in a new inode)
# see also this link for a rsync tutorial: http://www.thegeekstuff.com/2010/09/rsync-command-examples/
# ------------- the help page ------------------------------------------
if [ "$1" == "-h" ] || [ "$1" == "--help" ]
then
cat << "EOF"
Version 2.00 2012-08-31
USAGE: rsync-snapshot.sh HOST [--recheck]
PURPOSE: create a snapshot backup of the whole filesystem into the folder
'/backup/snapshot/HOST/snapshot.001'.
If HOST is 'localhost' it is replaced with the local hostname.
If HOST is a remote host then rsync over ssh is used to transfer the files
with a delta-transfer algorithm to transfer only minimal parts of the files
and improve speed; rsync uses for this the previous backup as reference.
This reference is also used to create hard links instead of files when
possible and thus save disk space. If original and reference file have
identical content but different timestamps or permissions then no hard link
is created.
A rotation of all backups renames snapshot.X into snapshot.X+1 and removes
backups with X>512. About 10 backups with non-linear distribution are kept
in rotation; for example with X=1,2,3,4,8,16,32,64,128,256,512.
The snapshots folders are protected read-only against all users including
root using 'chattr'.
The --recheck option forces a sync of all files even if they have same mtime
& size; it is can verify a backup and fix corrupted files;
--recheck recalculates also the MD5 integrity signatures without using the
last signature-file as precalculation.
Some features like filter rules, MD5, chattr, bwlimit and per server retention
policy can be configured by modifying the scripts directly.
FILES:
/backup/snapshot/rsync/rsync-snapshot.sh the backup script
/backup/snapshot/rsync/rsync-list.sh the md5 signature script
/backup/snapshot/rsync/rsync-include.txt the filter rules
Examples:
(nice -5 ./rsync-snapshot.sh >log &) ; tail -f log
cd /backup/snapshot; for i in $(ls -A); do nice -10 /backup/snapshot/rsync/rsync-snapshot.sh $i; done
EOF
exit 1
fi
# ------------- tuning options, file locations and constants -----------
SRC="$1" #name of backup source, may be a remote or local hostname
OPT="$2" #options (--recheck)
HOST_PORT=22 #port of source of backup
SCRIPT_PATH="/backup/snapshot/rsync"
SNAPSHOT_DST="/backup/snapshot" #destination folder
NAME="snapshot" #backup name
LOG="rsync.log"
MIN_MIBSIZE=5000 # older snapshots (except snapshot.001) are removed if free disk <= MIN_MIBSIZE. the script may exit without performing a backup if free disk is still short.
OVERWRITE_LAST=0 # if free disk space is too small, then this option let us remove snapshot.001 as well and retry once
MAX_MIBSIZE=20000 # older snapshots (except snapshot.001) are removed if their size >= MAX_MIBSIZE. the script performs a backup even if their size is too big.
#old: SPEED=5 # 1 is slow, 100 is fast, 100000 faster and 0 does not use slow-down. this allows to avoid rsync consuming too much system performance
BWLIMIT=100000 # bandwidth limit in KiB/s. 0 does not use slow-down. this allows to avoid rsync consuming too much system performance
BACKUPSERVER="rembk" # this server connects to all other to download filesystems and create remote snapshot backups
MD5LIST=0 #to compute a list of md5 integrity signatures of all backuped files, need 'rsync-list.sh'
CHATTR=1 # to use 'chattr' command and protect the backups again modification and deletion
DU=1 # to use 'du' command and calculate the size of existing backups, disable it if you have many backups and it is getting too slow (for example on BACKUPSERVER)
SOURCE="/" #source folder to backup
HOST_LOCAL="$(hostname -s)" #local hostname
#HOST_SRC="${SRC:-${HOST_LOCAL}}" #explicit source hostname, default is local hostname
if [ -z "${SRC}" ] || [ "${SRC}" == "localhost" ]
then
HOST_SRC="${HOST_LOCAL}" #explicit source hostname, default is local hostname
else
HOST_SRC="${SRC}" #explicit source hostname
fi
if [ "${HOST_LOCAL}" == "${BACKUPSERVER}" ] #if we are on BACKUPSERVER then do some fine tuning
then
MD5LIST=1
MIN_MIBSIZE=35000 #needed free space for chunk-file tape-arch.sh
MAX_MIBSIZE=12000
DU=0 # NB: 'du' is currently disabled on BACKUPSERVER for performance reasons
elif [ "${HOST_LOCAL}" == "${HOST_SRC}" ] #else if we are on a generic server then do other some fine tuning
then
if [ "${HOST_SRC}" == "ZRHSV-TST01" ]; then MIN_MIBSIZE=500; CHATTR=0; DU=0; MD5LIST=0; fi
fi
# ------------- initialization -----------------------------------------
shopt -s extglob #enable extended pattern matching operators
OPTION="--stats
--recursive
--links
--perms
--times
--group
--owner
--devices
--hard-links
--numeric-ids
--delete
--delete-excluded
--bwlimit=${BWLIMIT}"
# --progress
# --size-only
# --stop-at
# --time-limit
# --sparse
if [ "${HOST_SRC}" != "${HOST_LOCAL}" ] #option for a remote server
then
SOURCE="${HOST_SRC}:${SOURCE}"
OPTION="${OPTION}
--compress
--rsh="ssh -p ${HOST_PORT} -i /root/.ssh/rsync_rsa -l root"
--rsync-path="/usr/bin/rsync""
fi
if [ "${OPT}" == "--recheck" ]
then
OPTION="${OPTION}
--ignore-times"
elif [ -n "${OPT}" ]
then
echo "Try rsync-snapshot.sh --help ."
exit 2
fi
# ------------- check conditions ---------------------------------------
echo "$(date +%Y-%m-%d_%H:%M:%S) ${HOST_SRC}: === Snapshot backup is created into ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.001 ==="
STARTDATE=$(date +%s)
# make sure we're running as root
if (($(id -u) != 0))
then
echo "Sorry, must be root. Exiting..."
echo "$(date +%Y-%m-%d_%H:%M:%S) ${HOST_SRC}: === Snapshot failed. ==="
exit 2
fi
# make sure we have a correct snapshot folder
if [ ! -d "${SNAPSHOT_DST}/${HOST_SRC}" ]
then
echo "Sorry, folder ${SNAPSHOT_DST}/${HOST_SRC} is missing. Exiting..."
echo "$(date +%Y-%m-%d_%H:%M:%S) ${HOST_SRC}: === Snapshot failed. ==="
exit 2
fi
# make sure we do not have started already rsync-snapshot.sh or rsync process (started by rsync-cp.sh or by a remote rsync-snapshot.sh) in the background.
if [ "${HOST_LOCAL}" != "${BACKUPSERVER}" ] #because BACKUPSERVER need sometimes to perform an rsync-cp.sh it must disable the check of "already started".
then
RSYNCPID=$(pgrep -f "/bin/bash .*rsync-snapshot.sh")
if ([ -n "${RSYNCPID}" ] && [ "${RSYNCPID}" != "$$" ]) || pgrep -x "rsync"
then
echo "Sorry, rsync is already running in the background. Exiting..."
echo "$(date +%Y-%m-%d_%H:%M:%S) ${HOST_SRC}: === Snapshot failed. ==="
exit 2
fi
fi
# ------------- remove some old backups --------------------------------
# remove certain snapshots to achieve an exponential distribution in time of the backups (1,2,4,8,...)
for b in 512 256 128 64 32 16 8 4
do
let a=b/2+1
let f=0 #this flag is set to 1 when we find the 1st snapshot in the range b..a
for i in $(eval echo $(printf "{%.3d..%.3d}" "${b}" "${a}"))
do
if [ -d "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i}" ]
then
if [ "${f}" -eq 0 ]
then
let f=1
else
echo "Removing ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i} ..."
[ "${CHATTR}" -eq 1 ] && chattr -fR -i "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i}"
rm -rf "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i}"
fi
fi
done
done
# remove additional backups if free disk space is short
remove_snapshot() {
local MIN_MIBSIZE2=$1
local MAX_MIBSIZE2=$2
for i in {512..001}
do
if [ -d "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i}" ] || [ ${i} -eq 1 ]
then
let d=0 #disk space used by snapshots and free disk space are ok
echo -n "$(date +%Y-%m-%d_%H:%M:%S) Checking free disk space... "
FREEDISK=$(df -B M ${SNAPSHOT_DST} | tail -1 | sed -e 's/ */ /g' | cut -d" " -f4 | sed -e 's/M*//g')
echo -n "${FREEDISK} MiB free. "
if [ ${FREEDISK} -ge ${MIN_MIBSIZE2} ]
then
echo "Ok, bigger than ${MIN_MIBSIZE2} MiB."
if [ "${DU}" -eq 0 ]
then #avoid slow 'du'
break
else
echo -n "$(date +%Y-%m-%d_%H:%M:%S) Checking disk space used by ${SNAPSHOT_DST}/${HOST_SRC} ... "
USEDDISK=$(du -B 1048576 -s "${SNAPSHOT_DST}/${HOST_SRC}/" | cut -f1)
echo -n "${USEDDISK} MiB used. "
if [ ${USEDDISK} -le ${MAX_MIBSIZE2} ]
then
echo "Ok, smaller than ${MAX_MIBSIZE2} MiB."
break
else
let d=2 #disk space used by snapshots is too big
fi
fi
else
let d=1 #free disk space is too small
fi
if [ ${d} -ne 0 ] #we need to remove snapshots
then
if [ ${i} -ne 1 ]
then
echo "Removing ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i} ..."
[ "${CHATTR}" -eq 1 ] && chattr -fR -i "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i}"
rm -rf "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i}"
else #all snapshots except snapshot.001 are removed
if [ ${d} -eq 1 ] #snapshot.001 causes that free space is too small
then
if [ "${OVERWRITE_LAST}" -eq 1 ] #last chance: remove snapshot.001 and retry once
then
OVERWRITE_LAST=0
echo "Warning, free disk space will be smaller than ${MIN_MIBSIZE} MiB."
echo "$(date +%Y-%m-%d_%H:%M:%S) OVERWRITE_LAST enabled. Removing ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.001 ..."
rm -rf "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.001"
else
for j in ${LNKDST//--link-dest=/}
do
if [ -d "${j}" ] && [ "${CHATTR}" -eq 1 ] && [ $(lsattr -d "${j}" | cut -b5) != "i" ]
then
chattr -fR +i "${j}" #undo unprotection that was needed to use hardlinks
fi
done
echo "Sorry, free disk space will be smaller than ${MIN_MIBSIZE} MiB. Exiting..."
echo "$(date +%Y-%m-%d_%H:%M:%S) ${HOST_SRC}: === Snapshot failed. ==="
exit 2
fi
elif [ ${d} -eq 2 ] #snapshot.001 causes that disk space used by snapshots is too big
then
echo "Warning, disk space used by ${SNAPSHOT_DST}/${HOST_SRC} will be bigger than ${MAX_MIBSIZE} MiB. Continuing anyway..."
fi
fi
fi
fi
done
}
# perform an estimation of required disk space for the new backup
while : #this loop is executed a 2nd time if OVERWRITE_LAST was ==1 and snapshot.001 got removed
do
OOVERWRITE_LAST="${OVERWRITE_LAST}"
echo -n "$(date +%Y-%m-%d_%H:%M:%S) Testing needed free disk space ..."
mkdir -p "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.test-free-disk-space"
chmod -R 775 "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.test-free-disk-space"
cat /dev/null >"${SNAPSHOT_DST}/${HOST_SRC}/${LOG}"
LNKDST=$(find "${SNAPSHOT_DST}/" -maxdepth 2 -type d -name "${NAME}.001" -printf " --link-dest=%p")
for i in ${LNKDST//--link-dest=/}
do
if [ -d "${i}" ] && [ "${CHATTR}" -eq 1 ] && [ $(lsattr -d "${i}" | cut -b5) == "i" ]
then
chattr -fR -i "${i}" #unprotect last snapshots to use hardlinks
fi
done
eval rsync
--dry-run
${OPTION}
--include-from="${SCRIPT_PATH}/rsync-include.txt"
${LNKDST}
"${SOURCE}" "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.test-free-disk-space" >>"${SNAPSHOT_DST}/${HOST_SRC}/${LOG}"
RES=$?
if [ "${RES}" -ne 0 ] && [ "${RES}" -ne 23 ] && [ "${RES}" -ne 24 ]
then
echo "Sorry, error in rsync execution (value ${RES}). Exiting..."
echo "$(date +%Y-%m-%d_%H:%M:%S) ${HOST_SRC}: === Snapshot failed. ==="
exit 2
fi
let i=$(tail -100 "${SNAPSHOT_DST}/${HOST_SRC}/${LOG}" | grep 'Total transferred file size:' | cut -d " " -f5)/1048576
echo " ${i} MiB needed."
rm -rf "${SNAPSHOT_DST}/${HOST_SRC}/${LOG}" "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.test-free-disk-space"
remove_snapshot $((${MIN_MIBSIZE} + ${i})) $((${MAX_MIBSIZE} - ${i}))
if [ "${OOVERWRITE_LAST}" == "${OVERWRITE_LAST}" ] #no need to retry
then
break
fi
done
# ------------- create the snapshot backup -----------------------------
# perform the filesystem backup using rsync and hard-links to the latest snapshot
# Note:
# -rsync behaves like cp --remove-destination by default, so the destination
# is unlinked first. If it were not so, this would copy over the other
# snapshot(s) too!
# -use --link-dest to hard-link when possible with previous snapshot,
# timestamps, permissions and ownerships are preserved
echo "$(date +%Y-%m-%d_%H:%M:%S) Creating folder ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000 ..."
mkdir -p "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000"
chmod 775 "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000"
cat /dev/null >"${SNAPSHOT_DST}/${HOST_SRC}/${LOG}"
echo -n "$(date +%Y-%m-%d_%H:%M:%S) Creating backup of ${HOST_SRC} into ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000"
if [ -n "${LNKDST}" ]
then
echo " hardlinked with ${LNKDST//--link-dest=/} ..."
else
echo " not hardlinked ..."
fi
eval rsync
-vv
${OPTION}
--include-from="${SCRIPT_PATH}/rsync-include.txt"
${LNKDST}
"${SOURCE}" "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000" >>"${SNAPSHOT_DST}/${HOST_SRC}/${LOG}"
RES=$?
if [ "${RES}" -ne 0 ] && [ "${RES}" -ne 23 ] && [ "${RES}" -ne 24 ]
then
echo "Sorry, error in rsync execution (value ${RES}). Exiting..."
echo "$(date +%Y-%m-%d_%H:%M:%S) ${HOST_SRC}: === Snapshot failed. ==="
exit 2
fi
for i in ${LNKDST//--link-dest=/}
do
if [ -d "${i}" ] && [ "${CHATTR}" -eq 1 ] && [ $(lsattr -d "${i}" | cut -b5) != "i" ]
then
chattr -fR +i "${i}" #undo unprotection that was needed to use hardlinks
fi
done
mv "${SNAPSHOT_DST}/${HOST_SRC}/${LOG}" "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000/${LOG}"
gzip -f "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000/${LOG}"
# ------------- create the MD5 integrity signature ---------------------
# create a gziped 'find'-list of all snapshot files (including md5 signatures)
if [ "${MD5LIST}" -eq 1 ]
then
echo "$(date +%Y-%m-%d_%H:%M:%S) Computing filelist with md5 signatures of ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000 ..."
OWD="$(pwd)"
cd "${SNAPSHOT_DST}"
# NOW=$(date "+%s")
# MYTZ=$(date "+%z")
# let NOW${MYTZ:0:1}=3600*${MYTZ:1:2}+60*${MYTZ:3:2} # convert localtime to UTC
# DATESTR=$(date -d "1970-01-01 $((${NOW} - 1)) sec" "+%Y-%m-%d_%H:%M:%S") # 'now - 1s' to avoid missing files
DATESTR=$(date -d "1970-01-01 UTC $(($(date +%s) - 1)) seconds" "+%Y-%m-%d_%H:%M:%S") # 'now - 1s' to avoid missing files
REF_LIST="$(find ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.001/ -maxdepth 1 -type f -name 'snapshot.*.list.gz' 2>/dev/null)"
if [ -n "${REF_LIST}" ] && [ "${OPT}" != "--recheck" ]
then
REF_LIST2="/tmp/rsync-reflist.tmp"
gzip -dc "${REF_LIST}" >"${REF_LIST2}"
touch -r "${REF_LIST}" "${REF_LIST2}"
${SCRIPT_PATH}/rsync-list.sh "${HOST_SRC}/${NAME}.000" 0 "${REF_LIST2}" | sort -u | gzip -c >"${HOST_SRC}/${NAME}.${DATESTR}.list.gz"
rm "${REF_LIST2}"
else
${SCRIPT_PATH}/rsync-list.sh "${HOST_SRC}/${NAME}.000" 0 | sort -u | gzip -c >"${HOST_SRC}/${NAME}.${DATESTR}.list.gz"
fi
touch -d "${DATESTR/_/ }" "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${DATESTR}.list.gz"
cd "${OWD}"
[ ! -d "${SNAPSHOT_DST}/${HOST_SRC}/md5-log" ] && mkdir -p "${SNAPSHOT_DST}/${HOST_SRC}/md5-log"
cp -al "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${DATESTR}.list.gz" "${SNAPSHOT_DST}/${HOST_SRC}/md5-log/${NAME}.${DATESTR}.list.gz"
mv "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${DATESTR}.list.gz" "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000/${NAME}.${DATESTR}.list.gz"
touch -d "${DATESTR/_/ }" "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000"
fi
# ------------- finish and clean up ------------------------------------
# protect the backup against modification with chattr +immutable
if [ "${CHATTR}" -eq 1 ]
then
echo "$(date +%Y-%m-%d_%H:%M:%S) Setting recursively immutable flag of ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000 ..."
chattr -fR +i "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.000"
fi
# rotate the backups
if [ -d "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.512" ] #remove snapshot.512
then
echo "Removing ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.512 ..."
[ "${CHATTR}" -eq 1 ] && chattr -fR -i "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.512"
rm -rf "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.512"
fi
[ -h "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.last" ] && rm -f "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.last"
for i in {511..000}
do
if [ -d "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i}" ]
then
let j=${i##+(0)}+1
j=$(printf "%.3d" "${j}")
echo "Renaming ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i} into ${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${j} ..."
[ "${CHATTR}" -eq 1 ] && chattr -i "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i}"
mv "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${i}" "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${j}"
[ "${CHATTR}" -eq 1 ] && chattr +i "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.${j}"
[ ! -h "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.last" ] && ln -s "${NAME}.${j}" "${SNAPSHOT_DST}/${HOST_SRC}/${NAME}.last"
fi
done
# remove additional backups if free disk space is short
OVERWRITE_LAST=0 #next call of remove_snapshot() will not remove snapshot.001
remove_snapshot ${MIN_MIBSIZE} ${MAX_MIBSIZE}
echo "$(date +%Y-%m-%d_%H:%M:%S) ${HOST_SRC}: === Snapshot backup successfully done in $(($(date +%s) - ${STARTDATE})) sec. ==="
exit 0
#eof
Then create the file ‘/backup/snapshot/rsync/rsync-include.txt’ (download rsync-include) that contains the include and exclude patterns:
#created by francois scheurer on 20120828 # #note: # -be careful with trailing spaces, '- * ' is different from '- *' # -rsync stops at first matched rule and ignore the rest # -rsync descends iteratively the folders-arborescence # -'**' matches also zero, one or several '/' # -get the list of all root files/folders # pdsh -f 1 -w server[1-22] 'ls -la / | sed -e "s/ */ /g" | cut -d" " -f9-' | cut -d" " -f2- | sort -u # -include all folders with '+ */' (missing this rule implies that '- *' will override all the inclusions of any subfolders) # -exclude all non explicited files with '- *' # -exclude everything except /etc/ssh: '+ /etc/ssh/** + */ - *' # -exclude content of /tmp but include foldername: '- /tmp/* + */' # -exclude content and also foldername /tmp: '- /tmp/ + */' # -exclude content of each .ssh but include foldername: '- /**/.ssh/* + */' # #include everything except /tmp/: #- /tmp/ #same but include /tmp/ as an empty folder: #- /tmp/* #include only /var/www/: #+ /var/ #+ /var/www/ #+ /var/www/** #- * #same but also include folder structure: #+ /var/www/** #+ */ #- * #pattern list for / (include by default): + / - /lost+found/* - /*.bak* - /*.old* #- /backup/* #- /boot/* #- /etc/ssh/ssh_host* #- /home/* - /media/* - /mnt/*/* #- /opt/* - /opt/fedora*/data/* - /opt/fedora*/lucene/* - /opt/fedora*/tomcat*/logs/* - /opt/fedora*/tomcat*/temp/* - /opt/fedora*/tomcat*/work/* - /postgresql/*/main/pg_log/* - /postgresql/*/main/pg_xlog/* - /postgresql/*/main/postmaster.opts - /postgresql/*/main/postmaster.pid #- /postgresql/*/main/*/* - /proc/* - /root/old/* #- /root/.bash_history - /root/.mc/* #- /root/.ssh/*openssh* - /root/.viminfo - /root/tmp/* #- /srv/* - /sys/* - /tmp/* #- /usr/local/franz/logstat/logstat.log - /var/cache/* - /var/lib/mysql/* - /var/lib/postgresql/*/main/*/* - /var/log/* #- /var/spool/* - /var/tmp/* #pattern list for /backup/ (exclude by default): + /backup/ - /backup/lost+found/* - /backup/*.bak* - /backup/*.old* + /backup/snapshot/ + /backup/snapshot/rsync/ + /backup/snapshot/rsync/** - /backup/snapshot/* - /backup/* + /mnt/ + /mnt/*/ + /mnt/*/backup/ + /mnt/*/backup/snapshot/ + /mnt/*/backup/snapshot/rsync/ + /mnt/*/backup/snapshot/rsync/** - /mnt/*/backup/snapshot/ - /mnt/*/backup/ #pattern list for /boot/ (include by default): + /boot/ - /boot/lost+found/* - /boot/*.bak* - /boot/*.old* + /boot/** #pattern list for /home/ (include by default): + /home/ - /home/lost+found/* - /home/*.bak* - /home/*.old* - /home/xen/* + /home/** #include folder structure by default: #+ */ #include everything by default: + * #exclude everything by default: #- * #eof
And finally create the optional shell-script ‘/backup/snapshot/rsync/rsync-list.sh’ (download rsync-list.sh) that calculates the MD5 integrity signatures:
#!/bin/bash
# created by francois scheurer on 20081109
# this script is used by rsync-snapshot.sh,
# it recursively prints to stdout the filelist of folder $1 and computes md5 signatures
# it deals correctly with special filenames with newlines or ''
# note1: the script assumes that a file is unchanged if its size and ctime are unchanged;
# this assumption has a very small risk of being wrong:
# it could be wrong if two files with different contents but same filename and size are created in the same second in two directories;
# if the first directory is then removed and the second is renamed as the first, the file is not detected as changed.
# note2: ctime checking can be replaced by mtime checking if CTIME_CHK=0;
# this is needed by rsync-snapshot.sh (because of hard links creation that do not preserve ctime).
# ------------- the help page ---------------------------------------------
if [ "$1" == "-h" ] || [ "$1" == "--help" ]
then
cat << "EOF"
Version 1.6 2009-06-19
USAGE: rsync-list.sh PATH/DIR CTIME_CHK [REF_LIST]
PURPOSE: recursively prints to stdout the filelist of folder PATH/DIR and computes md5 integrity signatures.
It deals correctly with special filenames with newlines or ''.
If a ref_list is provided, it is used to avoid the re-calculation of md5 on files
with unchanged filename and ctime.
A ref_list is a file containing the output of a previously execution of this shell-script.
The script assumes that a file is unchanged if its size and ctime are unchanged.
The ref_list_mtime is used to force a md5 re-calculation of all files with newer ctime:
-if file_ctime > ref_list_mtime then re-calculate md5
-if file_ctime = ref_file_ctime then use ref_list
CTIME_CHK can be 1 to base the algorithm on ctime or 0 to base it on mtime.
NOTE: the script assumes that all processes avoid all file modifications in PATH/DIR during the script's execution,
you should read following remarks if this assumption cannot be guaranted:
-a recent ref_list_mtime (>= date_of_first_write_to_ref_list) causes the script
to miss all files with: ref_list_mtime >= file_ctime > ref_file_ctime
solution: 'touch' ref_file_mtime with date_of_first_write_to_ref_list - 1 second
-an old ref_list_mtime (< date_of_last_write_to_ref_list) causes the script
to double all files with: ref_list_mtime < file_ctime = ref_file_ctime
solution: pipe the output to 'sort -u'
EXAMPLE:
DATESTR=$( date -d "1970-01-01 UTC $(( $( date +%s ) - 1 )) seconds" "+%Y-%m-%d_%H:%M:%S" ) # 'now - 1s' to avoid missing files
REF_LIST="/etc.2008-11-23_10:00:00.list.gz"
REF_LIST2="/tmp/rsync-reflist.tmp"
gzip -dc "${REF_LIST}" >"${REF_LIST2}"
touch -r "${REF_LIST}" "${REF_LIST2}"
./rsync-list.sh "/etc/" 1 "${REF_LIST2}" | sort -u | gzip -c >"/etc.${DATESTR}.list.gz" # 'sort -u' to avoid doubling files
rm "${REF_LIST2}"
touch -d "${DATESTR/_/ }" "/etc.${DATESTR}.list.gz"
EOF
exit 1
elif [ $# -ne 2 ] && [ $# -ne 3 ]
then
echo "Sorry, you must provide 2 or 3 arguments. Exiting..."
exit 2
fi
# ------------- file locations and constants ---------------------------
SRC="$1" #name of source of backup, remote or local hostname
CTIME_CHK=$2 #1 for ctime checking, 0 for mtime checking
if [ "$CTIME_CHK" -eq 1 ]
then
CTIME_STAT="%z"
CTIME_FIND="-cnewer"
else
CTIME_STAT="%y"
CTIME_FIND="-newer"
fi
REF="$3" #filename of optional reference list
SCRIPT_PATH="/backup/snapshot/rsync"
FINDSCRIPT="$SCRIPT_PATH/rsync-find.sh.tmp" # temporary shell-script to calculate filelist
# ------------- using reference list to to reduce md5 calculation time -
if [ -n "$REF" ] #we have a previous md5 list
then
if ! [ -s "$REF" ] #invalid reference list
then
echo "Error: $REF is not a valid reference list. Exiting..."
exit 2
fi
touch /tmp/testsystime.tmp
if ! [ /tmp/testsystime.tmp -nt "$REF" ] #if system time is incorrect then exit
then
echo "Error: system time is older than mtime of $REF. Exiting..."
rm /tmp/testsystime.tmp
exit 2
fi
rm /tmp/testsystime.tmp
cat "$REF" | while read -r LINE #consider all previous files that still exist now with same ctime and size and print their already calculated md5
do
SIZE_AND_CTIME="${LINE#* md5sum=* * * * }" #extract size and ctime from reference list
SIZE_AND_CTIME="${SIZE_AND_CTIME% `*}"
LINE2="${LINE%% md5sum=*}" #1) keep only the filename part of the line
LINE2="${LINE2//n/
}" #2) replace 'n' with newline, the problem now is that 'n' is replaced, too (following is not a solution because it removes previous char LINE2="${LINE2//[^]n/newline}")
LINE2="${LINE2//
/n}" #3) replace ''+newline with 'n', fixing the problem of 2)
LINE2="${LINE2///}" #4) replace '' with ''
if [ -a "$LINE2" ] || [ -h "$LINE2" ] #check if file still exists
then
SIZE_AND_CTIME2=$( stat -c"%s $CTIME_STAT" "$LINE2" )
SIZE_AND_CTIME2="${SIZE_AND_CTIME2#* md5sum=* * * * }" #get size and ctime from current file
SIZE_AND_CTIME2="${SIZE_AND_CTIME2% `*}"
if [ "$SIZE_AND_CTIME" == "$SIZE_AND_CTIME2" ] #current file unchanged (see above note), so print the already calculated md5
then
echo "$LINE"
elif [ "${SIZE_AND_CTIME#* }" == "${SIZE_AND_CTIME2#* }" ] #size is different but ctime is same: update current file's ctime to force md5's recalculation (see below)
then
if [ "$CTIME_CHK" -eq 1 ]
then
chmod --reference="$LINE2" "$LINE2" #update ctime (note: system time is assumed to be correct)
else
touch -m "$LINE2" #update mtime (note: system time is assumed to be correct)
fi
fi
fi #else the file has been either deleted or modified (different ctime) and reference list is here useless
done
CNEWER_REF="$CTIME_FIND $REF" #prepare 'find' -cnewer option
else
CNEWER_REF=""
fi
# ------------- calculation of md5 sums --------------------------------
#this 1st method is not slow but fails on filenames with newlines or ''
find "${SRC}" $CNEWER_REF ! ( -path "*
*" -o -path "**" -o -path " *" -o -path "* " ) | while read LINE
do
LINE2="$LINE"
if ! [ -h "$LINE" ] && [ -f "$LINE" ]
then
RES=$( md5sum "$LINE" )
LINE2="$LINE2 md5sum=${RES%% *}"
else
LINE2="$LINE2 md5sum=-"
fi
RES=$( echo $( stat -c"%A %U %G %s $CTIME_STAT `%F'" "$LINE" ) )
echo -E "$LINE2 $RES"
done
#this 2nd method is slow but works on filenames with newlines or ''
( cat << "EOF"
#!/bin/bash
LINE="$1"
# LINE2="${LINE///}" # replace with
LINE2="${LINE///}" # replace with
LINE2="${LINE2//
/n}" # replace newline with n
if ! [ -h "$LINE" ] && [ -f "$LINE" ]
then
RES=$( md5sum "$LINE" )
LINE2="$LINE2 md5sum=${RES%% *}"
else
LINE2="$LINE2 md5sum=-"
fi
RES=$( echo $( stat -c"%A %U %G %s $CTIME_STAT `%F'" "$LINE" ) )
echo -E "$LINE2 $RES"
EOF
) >"$FINDSCRIPT"
chmod +x "$FINDSCRIPT"
find "${SRC}" $CNEWER_REF ( -path "*
*" -o -path "**" -o -path " *" -o -path "* " ) -print0 | xargs --replace --null "$FINDSCRIPT" "{}"
rm "$FINDSCRIPT"
#eof
Set the ownerships and permissions:
chown -cR root:root /backup/snapshot/rsync/ chmod 700 /backup/snapshot/rsync/rsync-*.sh chmod 600 /backup/snapshot/rsync/rsync-include.txt
Usage
When you call the script ‘rsync-snapshot.sh’ without parameters or with the hostname of the server itself (or localhost), the script performs a self-snapshot of the complete filesystem ‘/’.
You can and should use filter rules to exclude things like ‘/proc/*’ and ‘/sys/*’. For this you need to edit the configuration file ‘/backup/snapshot/rsync/rsync-include.txt’.
A description of the filter rules syntax is written as comments in the file itself.
The snapshot backup is created into ‘/backup/snapshot/HOST/snapshot.001′, where ‘HOST’ is your server’s hostname. If the folder ‘snapshot.001′ exists already it is rotated to ‘snapshot.002′ and so on, up to ‘snapshot.512′, thereafter it is removed. So if you create one backup per night, for example with a cronjob, then this retention policy gives you 512 days of retention. This is useful but this can require to much disk space, that is why we have included a non-linear distribution policy. In short, we keep only the oldest backup in the range 257-512, and also in the range 129-256, and so on. This exponential distribution in time of the backups retains more backups in the short term and less in the long term; it keeps only 10 or 11 backups but spans a retention of 257-512 days.
In the following table you can see on each column the different steps of the rotation, where each column shows the current set of snapshots (limited from snapshot.1 to snapshot.16 in this example):
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
3 3 3 3 3 3 3 3
4 4 4 4 4 4 4 4 4
5 5 5 5
6 6 6 6
7 7 7 7
8 8 8 8 8
9 9
10 10
11 11
12 12
13 13
14 14
15 15
16 16 16
To save more disk space, ‘rsync’ will make hard links for each file of ‘snapshot.001′ that already existed in ‘snapshot.002′ with identical content, timestamps and ownerships.
For example, the following example creates a backup and then use commands to let you see the used disk space of the 4 existing backups:
root@server05:~# /backup/snapshot/rsync/rsync-snapshot.sh 2012-09-11_19:07:43 server05: === Snapshot backup is created into /backup/snapshot/server05/snapshot.001 === 2012-09-11_19:07:43 Testing needed free disk space ... 0 MiB needed. 2012-09-11_19:07:45 Checking free disk space... 485997 MiB free. Ok, bigger than 5000 MiB. 2012-09-11_19:07:45 Checking disk space used by /backup/snapshot/server05 ... 11011 MiB used. Ok, smaller than 20000 MiB. 2012-09-11_19:07:46 Creating folder /backup/snapshot/server05/snapshot.000 ... 2012-09-11_19:07:46 Creating backup of server05 into /backup/snapshot/server05/snapshot.000 hardlinked with /backup/snapshot/server05/snapshot.001 ... 2012-09-11_19:07:52 Setting recursively immutable flag of /backup/snapshot/server05/snapshot.000 ... Renaming /backup/snapshot/server05/snapshot.003 into /backup/snapshot/server05/snapshot.004 ... Renaming /backup/snapshot/server05/snapshot.002 into /backup/snapshot/server05/snapshot.003 ... Renaming /backup/snapshot/server05/snapshot.001 into /backup/snapshot/server05/snapshot.002 ... Renaming /backup/snapshot/server05/snapshot.000 into /backup/snapshot/server05/snapshot.001 ... 2012-09-11_19:07:55 Checking free disk space... 485958 MiB free. Ok, bigger than 5000 MiB. 2012-09-11_19:07:55 Checking disk space used by /backup/snapshot/server05 ... 11050 MiB used. Ok, smaller than 20000 MiB. 2012-09-11_19:07:56 server05: === Snapshot backup successfully done in 13 sec. === ----------------------------- root@server05:~# du -chslB1M /backup/snapshot/localhost/snapshot.* | column -t 10901 /backup/snapshot/localhost/snapshot.001 10901 /backup/snapshot/localhost/snapshot.002 10901 /backup/snapshot/localhost/snapshot.003 10901 /backup/snapshot/localhost/snapshot.004 0 /backup/snapshot/localhost/snapshot.last 43602 total ----------------------------- root@server05:~# du -chsB1M /backup/snapshot/localhost/snapshot.* | column -t 10898 /backup/snapshot/localhost/snapshot.001 40 /backup/snapshot/localhost/snapshot.002 45 /backup/snapshot/localhost/snapshot.003 45 /backup/snapshot/localhost/snapshot.004 0 /backup/snapshot/localhost/snapshot.last 11026 total -----------------------------
We can see that the 4 snapshot backups use 10.9 GB each, so without hard links they would sum to 43 GB; the last command shows on the contrary that the real used size is only 11 GB, thanks to the hard links.
BTW, the following command can be very useful to replace all duplicate files with hard links to the first file in each set of duplicates, even if they have different name or path:
chattr -fR -i /backup/snapshot/localhost/snapshot.* fdupes -r1L /backup/snapshot/localhost/snapshot.*
A good tutorial on how to use the ‘rsync’ command is available here:
http://www.thegeekstuff.com/2010/09/rsync-command-examples/
When called with a remote hostname as parameter, the script performs a snapshot backup via the network. This can be very useful for a DRP (Disaster Recovery Plan), in order to have a servers’ farm replicated every night to a secondary site. In addition to that you could implement a continuous replication of the databases for example. The ‘BWLIMIT’ can then be changed inside the shell-script to limit here the network bandwidth usage and the disk I/O overhead; it can help so to moderate the performance impact and avoid any slow down on critical production servers.
Other variables can also be modified at the beginning of the script, either as a global setting or specific tuning for some servers; a ‘BACKUPSERVER’ section is already provided for this purpose and let you tune specific settings for the remote central backup server:
HOST_PORT=22 #port of source of backup
SCRIPT_PATH="/backup/snapshot/rsync"
SNAPSHOT_DST="/backup/snapshot" #destination folder
NAME="snapshot" #backup name
LOG="rsync.log"
MIN_MIBSIZE=5000 #older snapshots (except snapshot.001) are removed if free disk <= MIN_MIBSIZE. the script may exit without performing a backup if free disk is still short.
OVERWRITE_LAST=0 #if free disk space is too small, then this option let us remove snapshot.001 as well and retry once
MAX_MIBSIZE=20000 #older snapshots (except snapshot.001) are removed if their size >= MAX_MIBSIZE. the script performs a backup even if their size is too big.
BWLIMIT=100000 #bandwidth limit in KiB/s. 0 does not use slow-down. this allows to avoid rsync consuming too much system performance
BACKUPSERVER="rembk" #this server connects to all other to download filesystems and create remote snapshot backups
MD5LIST=0 #to compute a list of md5 integrity signatures of all backuped files, need 'rsync-list.sh'
CHATTR=1 #to use 'chattr' command and protect the backups again modification and deletion
DU=1 #to use 'du' command and calculate the size of existing backups, disable it if you have many backups and it is getting too slow (for example on BACKUPSERVER)
SOURCE="/" #source folder to backup
if [ "${HOST_LOCAL}" == "${BACKUPSERVER}" ] #if we are on BACKUPSERVER then do some fine tuning
then
MD5LIST=1
MIN_MIBSIZE=35000 #needed free space for chunk-file tape-arch.sh
MAX_MIBSIZE=12000
DU=0 # NB: 'du' is currently disabled on BACKUPSERVER for performance reasons
elif [ "${HOST_LOCAL}" == "${HOST_SRC}" ] #else if we are on a generic server then do other some fine tuning
then
if [ "${HOST_SRC}" == "ZRHSV-TST01" ]; then MIN_MIBSIZE=500; CHATTR=0; DU=0; MD5LIST=0; fi
fi
To make the backup server able to connect via ‘ssh’ to the target servers without interactive entering of a password, you should create a ‘ssh’ host-key with empty passphrase ‘/root/.ssh/rsync_rsa’ and copy the public key to the target servers:
#on each targetserver:
mkdir -p ~root/.ssh/
chown root:root ~root/.ssh/
chmod 700 ~root/.ssh/
touch ~root/.ssh/authorized_keys
chown root:root ~root/.ssh/authorized_keys
chmod 600 ~root/.ssh/authorized_keys
#update manually /etc/ssh/sshd_config to have 'AllowUsers root'
service ssh reload
#on the backupserver, create the key with an empty passphrase:
ssh-keygen -f ~/.ssh/rsync_rsa
#and upload the public key to the targetserver:
MYIP=$(hostname -i) #assign here the backupserver's external IP if necessary
echo "from="${MYIP%% *}",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="rsync ${SSH_ORIGINAL_COMMAND#* }" $(ssh-keygen -yf ~/.ssh/rsync_rsa)" | ssh targetserver "cat - >>~/.ssh/authorized_keys"
Note that the ‘command=’ restriction (http://larstobi.blogspot.ch/2011/01/restrict-ssh-access-to-one-command-but.html) will not apply if ‘/etc/sshd_config’ has already a ‘ForceCommand’ directive.
This central backup server could also be used to centralize the administration of all other servers via pdsh/ssh (LINK).
Because the script does not freeze the filesystem during its operation, there is no guaranty that the snapshot backup will be a strict snapshot, in other words the files will not be copied at the exact same moment. This is usually not an issue, except for databases. In order to keep the consistency of a database, you should follow the instructions of http://www.postgresql.org/docs/9.1/static/continuous-archiving.html and http://www.anchor.com.au/blog/documentation/better-postgresql-backups-with-wal-archiving/.
The following example applies for PostgreSQL 9.1 on Debian:
#sudo -u postgres mkdir /var/lib/postgresql/9.1/main/wal_archive
#sudo -u postgres chmod 700 /var/lib/postgresql/9.1/main/wal_archive
#vi /etc/postgresql/9.1/main/postgresql.conf
# - Archiving -
#archive_mode = on # allows archiving to be done
# # (change requires restart)
#archive_command = 'test ! -f /var/lib/postgresql/9.1/main/backup_in_progress || (test ! -f /var/lib/postgresql/9.1/main/wal_archive/%f && cp %p /var/lib/postgresql/9.1/main/wal_archive/%f)' # command to use to archive a logfile segment
#archive_timeout = 0 # force a logfile segment switch after this
# # number of seconds; 0 disables
#check if postgresql is running:
if sudo -u postgres /usr/lib/postgresql/9.1/bin/pg_ctl -D /var/lib/postgresql/9.1/main/ status
then
touch /var/lib/postgresql/9.1/main/backup_in_progress
#freeze posgresql writing (all writes will go only in pg_xlog WAL-files), in order to make a clean backup at filesystem-level:
sudo -u postgres psql -c "SET LOCAL synchronous_commit TO OFF; SELECT pg_start_backup('rsync-snapshot', true);"
fi
#perform the backup
/backup/snapshot/rsync/rsync-snapshot.sh
#check if postgresql is running:
if sudo -u postgres /usr/lib/postgresql/9.1/bin/pg_ctl -D /var/lib/postgresql/9.1/main/ status
then
#unfreeze posgresql writing:
sudo -u postgres psql -c "SET LOCAL synchronous_commit TO OFF; SELECT pg_stop_backup();"
rm /var/lib/postgresql/9.1/main/backup_in_progress
chattr -R -i /backup/snapshot/localhost/snapshot.001/var/lib/postgresql/9.1/main &>/dev/null
mv /var/lib/postgresql/9.1/main/wal_archive/* /backup/snapshot/localhost/snapshot.001/var/lib/postgresql/9.1/main/pg_xlog/
rm /backup/snapshot/localhost/snapshot.001/var/lib/postgresql/9.1/main/{backup_in_progress,backup_label}
chattr -R +i /backup/snapshot/localhost/snapshot.001/var/lib/postgresql/9.1/main &>/dev/null
fi
#NB:
# -If you need to re-create a standby server while transactions are waiting, make sure that the commands to run pg_start_backup() and pg_stop_backup() are run in a session with synchronous_commit = off, otherwise those requests will wait forever for the standby to appear.
# -see also 'pg_dump' and 'pg_basebackups' commands.
It is even possible to freeze an ext3/ext4 filesystem before backuping, but this is quite dangerous, because all processes that try to write on it will get frozen until you unfreeze the filesystem.
You should therefore avoid using this on production server! But for the sake of information here are the steps to freeze the filesystem mounted on ‘/mnt/folder’ during 30 seconds on Debian:
wget ftp://ftp.kernel.org/pub/linux/utils/util-linux/v2.22/util-linux-2.22-rc2.tar.gz tar xfz util-linux-2.22-rc2.tar.gz cd xfz util-linux-2.22-rc2 aptitude install ncurses-dev libncurses5-dev mkcramfs cramfsprogs zlib1g-dev libpam-dev libpam0g-dev ./configure make make clean man -l sys-utils/fsfreeze.8 ./fsfreeze -f /mnt/folder && sleep 30 && ./fsfreeze -u /mnt/folder
I hope this ‘rsync-snapshot.sh’ script can be useful to you! ^_^
Another script will be posted on the blog soon to show you how to archive those snapshot backups on tapes using ‘tar’ with encrypted split chunks of data.