rsync Backups on OS X

Clones across the network

With a little bit of work you can have rsync cloning a primary server to a backup server. If the primary server ever fails, all you have to do is reboot the backup from the cloned system and away you go.

Ed. Note: This is a very nice step-by-step for getting rsync working across the network. This is not necessarily hard to setup. However, I would strongly urge you to do this on a test system before deploying. Rsync, in whatever flavor you use, can be rather cryptic in configuration and use.

rsync (http://rsync.samba.org) is a favorite tool of many UNIX sysadmins for doing snapshot-style backups of hosts across the network. It’s more useful than other utilities like tar or scp because it’s smart about only copying the differences between files across a connection. rsync also has the ability to run as a server, making it possible to sync many slaves up to a master or vice-versa. We recently migrated the majority of our services (mail, web, directory, etc) from Linux to Mac OS X and I was surprised at just how tricky it was to get rsync working properly on OS X.

Matthew Phillips has a pretty good tutorial on using rsync for backups on OS X at http://www.labf.org/~egon/mac_backup/, but he describes the process with rsyncX (http://archive.macosxlabs.org/rsyncx/rsyncx.html), which suffers from a number of bugs that make it unreliable.

I’ve put together a little tutorial here on how I got snapshot backups working.

Getting Started

For this tutorial, we’ll be creating a snapshot fail-over server on an older G4 which we want to mirror a Xserve. In the event of a failure on our Xserve, we’d like to be able to change the startup disk on our G4 to the partition mirroring our Xserve system disk, reboot and be back up on our feet with last night’s configuration. This is an ideal solution for a small business with a tight budget but high-availability requirements.

On the xserve, we have the following partitions which need to be mirrored:

/dev/disk0s3 on / (local, journaled)
/dev/disk1s10 on /Volumes/Users (NFS exported, local, journaled)
/dev/disk1s12 on /Volumes/Space (NFS exported, local, journaled)

We’ve put a 120GB and a 40GB hard drive in the G4 and created the following partitions:

/dev/disk1s10 on / (local, journaled)
/dev/disk1s12 on /Volumes/System (local, journaled)
/dev/disk1s14 on /Volumes/Users (local, journaled)
/dev/disk0s5 on /Volumes/Space (local)

Having separate partitions is necessary for the quick failover to work and it keeps things neat on the reboot with Share Points and whatnot. On the Xserve, our / partition is labeled System and on the G4 it’s labeled Backup. This should keep confusion to a minimum.

Getting rsync to work on OS X

Once your machines are configured and the OS is installed the next trick is getting rsync to work properly with HFS+ filesystems. There are some issues which make this a little complicated. The first, of course, is the resource forks. rsync_hfs or rsyncX address this issue by bundling up the resource fork with the data fork and tossing both across the wire to a (hopefully) resource fork-aware rsync on the other side. This is good (and it comes with a neat GUI), but there are two problems with rsyncX. The first is that rsync (designed on Linux) requires a Linux-friendly lchown system call for changing the permissions on a symbolic link and gives us a “File not found” error when running under Darwin. This issue is documented by Ulrich Hoffmann at http://www.honkbude.org/article.php?story=20040801233916583&mode=print (with patch to boot!). The second issue is that the latest release of rsyncX is 2.1, which suffers from a nasty double-free bug in the bundled z-lib. Not only does this create a security issue; it also causes backups to fail when rsync bombs out.

Ed. Note: I use rsyncX a lot on a local basis, without going across the network, with no issues. Of course local-only is much less complex than a network synch, and it’s certainly true true that rsyncX is rather crusty in it’s base version of rysnc. I’ve found that for admins new to rsync, using the rsyncX GUI’s script generator is a decent way to get a feel of what’s going on. From there you can move on to writing the scripts yourself with whatever flavor of rsync you choose.

To fix the HFS+ problem, D Andrew Reynhout has proposed a neat solution at http://www.quesera.com/reynhout/misc/rsync+hfsmode/. He separates the data and resource fork into AppleDouble format before transmission. Using his code has two benefits. First off, his source code is easy to get to and he keeps his diffs up to date with the latest release of rsync proper, which makes it easy for us to get around the z-lib issue, which was patched in 2.6.3. Second off, since we’re splitting the files into AppleDouble format, we can receive the files on non-forked filesystems, like UFS, ext3, and Xsan.

Reynhout releases patched rsync binaries on his site, but as far as I know, they don’t include the lchown patch. So, we’ll build from scratch. First, grab all the source we need:

• rsync http://rsync.samba.org/download.html
• Hoffman’s patch http://www.honkbude.org/filemgmt_data/files/rsync_hfs-lchown.patch
• Reynhout’s patch http://www.quesera.com/reynhout/misc/rsync+hfsmode/

Unpack the rsync sources and apply the patches as best as you can. Hoffman’s patch is against an earlier version of rsync than 2.6.3 and I had to manually edit rsync.c. It’s not too tricky though, he just bails out of the chown process if the file is a symbolic link. Reynhout’s patch applies cleanly to the current sources. When compiling rsync on OS X, you have to specify LDFLAGS="-framework CoreServices" – autoconf doesn’t pick that up. rsync‘s Makefile supports the DESTDIR environment variable, so making a package with PackageMaker is a snap. Just set DESTDIR to some friendly location like /tmp/rsync/Files and use that as your “Files” directory in the package. Make a /tmp/rsync/Resources directory, throw in a copy of the README and the GPL for good measure, and you’re done. Once you’ve made the package, you can use Remote Desktop to push it out to your machines or install it the old fashioned way.

The package I made is available here:

Setting up SSH

Once we’ve gotten past the rsync hurdle, we need to be able to to SSH between the machines as root without a password (unless you intend to stand at the console and type in your root password every morning at 3:00AM). There are some security concerns any time you set up a login process like this, but chances are good that if someone has root on one of your boxes, they can get root on the others pretty easily. There are two ways to allow SSH between machines without a password, using host-based authentication and using authorized keys. Host-based is a real pain in the neck so we’ll use authorized keys, which works out of the box on OS X.

$ sudo passwd root

# ssh-keygen -t dsa -f ~/.ssh/id_dsa

# scp ~/.ssh/id_dsa.pub [email protected]:
# ssh [email protected]
# mkdir -p ~/.ssh
# cat id_dsa.pub >> ~/.ssh/authorized_keys
# rm id_dsa.pub
# chmod 700 .ssh

# rsync -e ssh --archive --update --delete --verbose ${MYDIR} [email protected]:${MYDIR}

for dir in System Users Space/admin
do
  rsync -e ssh --archive --update --delete --verbose /Volumes/${dir}/ [email protected]:/Volumes/${dir}
done

tmp/*
proc
Network/*
Volumes/*
cores/*
*/.Trash
dev/*
afs/*
automount/*
private/tmp/*
private/var/vm/*
private/var/run/*
private/var/spool/postfix/private/*
private/var/spool/postfix/public/*
private/var/imap/socket/*

RSYNC="/usr/local/bin/rsync"
RSYNCOPTS="-e ssh --archive --update --delete --verbose --hfs-mode=appledouble"
EXCLUDEFILE="/Library/Scripts/Sync/sync_exclude"
PUSH="System Users Space/admin"
LOCALPREFIX="/Volumes"
REMOTEPREFIX="[email protected]:/Volumes"

for dir in $PUSH
do
  $RSYNC $RSYNCOPTS --exclude-from=$EXCLUDEFILE --rsync-path=$RSYNC $LOCALPREFIX/$dir/ $REMOTEPREFIX/$dir
done

Then we’ll run the script by hand on the Xserve, make sure that everything looks good on the G4, change the Startup Disk on the G4, remember to unplug the ethernet cable from it so we don’t get an IP conflict, and reboot. It should come up looking just like the Xserve. Woot!

Ed. Note: For this to work correctly, since the remote G4 is a live system, you’ll want to have a System volume with a different name than the volumes that you are synching over. In case of disaster on the primary volume, you would set the boot volume on the secondary server to the cloned copy of the primary server and reboot it. At this point the secondary would be a spitting image of the primary from the last time the rsync was run.

Running the script from cron

Once we have a good sync script, we’ll want to run it every night from cron. The easiest way to do this is to create a daily.local script and drop it in our /etc directory. It should run after all the other daily tasks are finished. If you already have an /etc/daily.local, just add the running of the script to it. Be sure to roll any logs that you create in your daily script so that you can keep track of what happened while you weren’t looking. If you want some fancy execution times, go ahead and edit /etc/crontab or root‘s crontab. Just remember that the script will have to run with root permissions to keep the permissions on the target host correct.

Ed. Note: Do keep in mind that while this article has you synchronizing the entire system from one machine to another, you don’t have to do it all or nothing. Instead you can just synch data folders or other pieces to another system if that’s all you need. However, a remote, up to date, bootable clone of your production server is quite a sexy thing to have around in case things go bad.

msolberg

More Posts - Website