Faking Filesystem Snapshots on Mac OS X Server
Most other operating systems on the market allow you to make a "snapshot" of a filesystem. Using these snapshots you can provide easy access, and in many cases self-service access, to instant backups of the files and folders. This functionality is usually part of the filesystem and one that HFS, in its now numerous forms, does not support. We can however fake it, and provide easy, self-servicing access backups to our users on some shares, in some cases. We call it the GhettoSnap.Read on for more...
Filesystem Snapshots in a Nutshell
How snapshots (shadow copies to Windows admins.) works can be a bit mystical, so I'll break it down in a very simplified way. When other OSes make a snapshot they all follow a procedure that goes something like this:
1. The system flushes any pending writes to the disk
2. A snapshot file is created. This contains the state of all the files on the filesystem
3. When writes resume the bits to be changed are moved to a snapshot directory that corresponds to the snapshot file
4. When the snapshot filesystem is read the OS references the snapshot file and composites the changed file parts in the snapshot directory against the source directory.
Generally the snapshot algorithm is pretty smart and only syncs the binary file parts of what has changed. That way if a user changes 300K of a 2GB database each day your storage requirements only go up by 300K a day.
To the user this is indistinguishable from magic as you can now present them with snapshot folders, commonly named for days of the week, that look like you backed up the entire filesystem each day. This way when they need access to a file that is inside your snapshot retention timeframe they can just go and get the file and not bother you.
Sounds great huh? Too bad HFS and Mac OS X don't support filesystem snapshots! There is a way though.
shadowy Tricks
Apple's intensely robust disk imaging frameworks do support snapshots of a sort through shadow files. These are used all the time to allow apparent changes to read-only disk images and are key to making Netboot usable. They work like this:
1. A disk image is mounted with a shadow file attached, this is created as needed
2. When writes occur they are redirected to shadow file that is attached to the disk image
3. When the filesystem is read the OS composites the changed file parts in the shadow file against the unchanging source directory on the disk image.
Sound familiar? The big difference is that we are redirecting the writes instead of backing up the original bits from the source.
Apple has always said that disk images are just part of the regular filesystems that Mac OS X can use. Let's put that to the test.
Plan of Attack
Here is what we can do to fake filesystem snapshots on Mac OS X Server. As it turns out you can mount a disk image multiple times with different shadow files. This gives us the snapshot functionality that we are looking for as I can have a shadow for each day of the week. In this example we are going to make three directories named "Images", "shadows", and "Snapshots". Inside the Snapshots we will create a directory for each day of the work week and we will then use these as the mountpoints for our daily snapshots. The entire Snapshots folder will then be symlinked back into the sharepoint as a directory named "Backups" to allow the users to retrieve their own backups.
Each night we will stop sharing the sharepoint, unmount the current image, copy the current shadow file with the appropriate name for that day, remount the original image with shadow and re-share it, and then mount that day's snapshot into the corresponding Snapshot directory.
Ready? Here we go.
Setup
The first thing we need to do is setup our directories on our data volume. Since I'm doing this on a remote server I'll show it via SSH, but you can make them anyway you want.
cd /Volumes/Data/
mkdir shadows Images Snapshots
cd shadows/
mkdir Monday Tuesday Wednesday Thursday FridayWith that out of the way we need to create our initial image. If you are creating a new share you can just make a new, empty image. If you are converting an existing directory then we need to create an image from that folder. In our case we are going to convert an existing directory, which will form our initial snapshot.
cd ../Images/
hdiutil create -srcfolder ~/Documents/AFP548 AFP548_ShareNow we need to make sure that the image has enough room to hold new files. To do this we are going to convert it to a sparse image and then resize the filesystem inside it. We can examine the existing freespace with hdiutil.
hdiutil hfsanalyze AFP548_Share.dmg | grep free space
current free space 0x0056E000 (5693440) bytes (5560.0 Kb/5.4 MB/0.0 GB)Now convert to a sparse and resize. I'll use 1GB as the filesystem size just as an example.
hdiutil convert -format UDSP -o AFP548_share AFP548_share.dmg
...
created: /Volumes/Data/Images/AFP548_share.sparseimage
hdiutil resize -size 1g AFP548_share.sparseimage
hdiutil hfsanalyze AFP548_share.sparseimage | grep free space
current free space 0x3C1AF000 (1008398336) bytes (984764.0 Kb/961.7 MB/0.9 GB)The nice thing about using sparse images is that they don't take up any more space than they need. Compare the file size difference between our initial image that has 5.4 MB free and the new sparse image that has 961.7 MB free.
ls -lh
total 212672
-rw-r--r-- 1 josh admin 38M Mar 3 07:35 AFP548_Share.dmg
-rw-r--r-- 1 josh admin 65M Mar 3 07:41 AFP548_share.sparseimagePretty cool huh? The sparse files grow as needed, but in this case it will never get a chance as we are going to mount it with a shadow. Since all writes are directed to the shadow file it is what will grow as needed instead. I picked 1 GB as a fairly arbitrary size in this example, really you can make it just about anything you want. You could even size it larger than the physical filesystem and then add storage as you approached capacity. This is a common practice with places such as .Mac or Gmail, but it requires constant monitoring, trending of data usage, and a filesystem that allows for expansion.
If I had wanted to make a new empty share I could just create a sparse directly.
hdiutil create -volname BigShare -type SPARSE -fs HFSX -size 10g BigShare
ls -lh
...
18M Mar 3 07:50 BigShare.sparseimageHere I was able to create a filesystem that will present 10 GB to the users while only consuming the amount of disk that the data actually takes up. I was able to do this even though the volume only has 1.7 GB free on it as well, so make sure you watch out for low disk conditions.
Next we need to mount our initial image with a shadow file. You can remove the original, pre-sparse conversion image if you like as well. It isn't needed anymore and is just wasting space.
rm AFP548_Share.dmg
hdiutil mount AFP548_share.sparseimage -shadow ../shadows/current
...
/dev/disk1s2 Apple_HFS /Volumes/AFP548Now the disk is mounted and we can see that it shows 962MB of free space.
df -h | grep AFP
/dev/disk1s2 1.0G 62M 962M 6% /Volumes/AFP548If we took a look in /Volumes, we would see that our image looks and feels like a regular volume, and it is.
ls /Volumes/AFP548
ACL Paper
AFP548.com Swag Shop.pdf
AFPCard Josh.psd
AppleScript Article
Backup.pdf
Bartosh revies.rtf
Bug Article
...You can share this volume just as you would any other in Workgroup Manager. I generally don't share the root of a filesystem, but in this specific case I would. This disk image represents the sharepoint we wish to create, not a volume to create new shares on. In the case of a new blank image I would create my shares under the root of the volume like normal.
We should go ahead and create our symlink from the share back to the Snapshots folder. This link will allow users to retrieve their own backups.
cd /Volumes/AFP548/
ln -s /Volumes/Data/Snapshots Backups
ls -l Backups
lrwxr-xr-x 1 josh josh 23 Mar 7 07:57 Backups -> /Volumes/Data/SnapshotsThe stage is now set.
Manual run
So we've got a mounted image that we can share that is backed by a shadow file. What next? We need to unmount the image, copy the current shadow --renaming it in the process--, remount the original image, and then wrap up by mounting the image again with the daily snapshot shadow. Let's pretend it's 12:01 AM on Wednesday morning and we need to generate the Tuesday snapshot.
hdiutil eject /Volumes/AFP548
"disk2" unmounted.
"disk2" ejected.cp /Volumes/Data/shadows/current /Volumes/Data/shadows/tuesday
hdiutil mount /Volumes/Data/Images/AFP548_share.sparseimage -shadow /Volumes/Data/shadows/current
...
/dev/disk2s2 Apple_HFS /Volumes/AFP548Now that we have created our Tuesday shadow file we need to mount it, but there are a few things that we need to think about first.
1. We don't want the users to modify this filesystem, so it needs to be read only.
2. We want to mount it in the Snapshot directory under the correct day.
3. We don't want it to appear on the Desktop as things get confusing quickly with 6 volumes with the same name showing up there.
4. We need to mount it with the correct shadow file.
Luckily hdiutil has options for all of these. (To make the command short enough to put here I've moved into the /Volumes/Data directory.)
hdiutil mount Images/AFP548_share.sparseimage -readonly -mountroot Snapshots/Tuesday -shadow shadows/tuesday -nobrowse
...
/dev/disk3s2 Apple_HFS /Volumes/Data/Snapshots/Tuesday/AFP548Now you can navigate the symlink on the main volume to the Tuesday snapshot and see the data as it existed that day. Let's try it out.
cd /Volumes/AFP548
ls
ACL Paper
AFP548.com Swag Shop.pdf
AFPCard Josh.psd
AppleScript Article
Backup.pdf
Backups
Bartosh revies.rtf
Bug Article
...OK, see that ACL Paper directory? That's my ACL whitepaper that you can get from the site. It's important to me as it contains all of my notes, screens, and the Pages file that is the source of the paper. Since it's important we can pretend we are a user for a moment...
rm -r ACL Paper/Aggghhhhhhhhhhh!!!! Not to worry though, we have a snapshot of the filesystem from yesterday and can even go get the backup on our own.
cd Backups/Tuesday/AFP548/
ls -l
total 33272
drwxr-xr-x 25 josh josh 850 Jan 25 19:21 ACL Paper
-rw-r--r-- 1 josh josh 58715 Jan 2 11:39 AFP548.com Swag Shop.pdf
-rw-r--r-- 1 josh josh 1366871 May 13 2005 AFPCard Josh.psd
drwxr-xr-x 7 josh josh 238 May 13 2005 AppleScript Article
...There it is! The ownership is even correct! Bless that wonderful sysadmin!
cp -r ACL Paper /Volumes/AFP548/
cd ../../../
ls -l | more
total 33272
drwxr-xr-x 25 josh josh 850 Mar 8 07:49 ACL Paper
-rw-r--r-- 1 josh josh 58715 Jan 2 11:39 AFP548.com Swag Shop.pdf
-rw-r--r-- 1 josh josh 1366871 May 13 2005 AFPCard Josh.psd
drwxr-xr-x 7 josh josh 238 May 13 2005 AppleScript Article
...Things fall apart
Now I get to disappoint you. I really don't like to say this, but actually deploying this requires some trickery to the level that I really wouldn't be comfortable with it in a production environment. Test environment sure, but not in production. As a policy we avoid posting anything that could cause data loss, or even just cause trouble, with our reader's servers. Because of this policy I'm not going to be walking you through all of the steps to implement this particular solution. Once I explain I think you will agree.
1. It's tricky to mount an image at the right time during boot. At first I was hopeful that I would be able to use a launchd job to call hdik in order to mount the image very early on. (hdik is the lightweight, in-kernel, image mounting tool and it requires less in the way of an OS around it to get an image mounted.) This was not to be the case though and I couldn't get the image mounted with launchd. You can use a startup item to mount the images fairly easily, but the problem then is that you need to have the image mounted before the file services try to share it. The best idea I had here was to make the image mounting script call serveradmin to start the service rather than rely on the service to start itself. This should work, but it's a level of chicanery that I don't like to resort to.
2. You can't do any of this with AFP. If you want to see an AFP server/client combo freak out just try having it follow a symlink. The good news is that you can use SMB or NFS or FTP just fine for this setup, with SMB providing the closest experience to AFP. The big problem with this would come from trying to create snapshots of existing legacy data. You would need to re-stream all of it from AFP to SMB or otherwise split the forks and meta off with SplitForks so that the clients would be able to see them when they connect. If you want to make a design group unhappy, lose the resource forks off all of their Quark files.
3. You get to a point where the shadows will be bigger than the source image. This solution makes it so that EVERYTHING is shadowed forever and ever. My thought at first was that you could simply merge the current shadow into the base at the end of your shadow retention period. This however makes you start back at zero with the daily snaps. If you had a snapshot policy that said that the self-service backups would vanish each weekend and be rebuilt during the week then it's OK, but I don't know of many places that would call that a good policy.
4. It's simply a complicated solution. You will have the same image mounted at least 6 times with different shadow files. Once automated you can forget about it, but setting it up has all sorts of room for error.
Summing up
So much like Joel's 10.3 managed mcx article, this one ends up being more of a parlor trick than anything else. If anyone wants to deploy this somewhere and give it a workout though, I've provided more than enough info for anyone with the necessary skills to pull it off. I just don't want to get anyone in over their head. If anything, you've learned more about the disk imaging frameworks in Mac OS X and there are certainly many uses for shadow files that are much less scary, and more practical, than this.
None of this stuff is particularly hard to implement but it does have a lot that has to be done just right, at the right time, and in the correct order. Like I said, it's tricky. I don't play fast and loose with my production servers and neither should you.
Snapshot support is generally something that is built-in at a filesystem level and HFS+ just doesn't have it yet. If Apple does pursue ZFS as an option then we might get snapshots sooner that later, but I don't know how realistic of a milestone that would be in the future. Here's to hoping that Apple does get snapshot support into the OS. It's a great tool that can help free sysadmins from the job of backup operator by providing self-service file recovery.
As always, have fun and read the man pages.