How Xsan changes absolutely everything you ever thought you knew about storage on your servers.
First off it may take some time for you to fully wrap your brain around what Xsan is. Storage area networks are very similar in this manner to the first time you begin thinking about LDAP or Kerberos. A four word description of Xsan makes sense, easy enough, but the four paragraph description can take you months to fully digest and understand the implications that it brings to your network.
Don't worry about it. It's really not your fault. This truly is something that takes some time to fully sink in.
History
While Xsan is new as a product for Apple, the actual filesystem behind it is not. Apple is leveraging the StoreNext filesystem which is primarily utilized by ADIC in their products which have been available for quite some time.
Apple has been heavily beta testing Xsan since mid-2004 at a number of different sites tweaking the StoreNext filesystem and the Xsan GUI for OS X.
While Apple is very much targeting Xsan towards the video space, the enterprise data center can easily come along for the ride on this one.
What it is
In two words Xsan is a "storage network". That was easy, right?
At the heart of it, this means that Xsan allows for multiple machines to directly access the same storage without the need for a fileserver in the middle. Think of this like attaching a firewire drive to three different Xserves at the same time and having all three be able to read and write to the same files on that firewire drive simultaneously.
Now, take that firewire drive and add another dozen firewire drives. Group 5 of the drives into a RAID 5. Take four of the remaining drives and stripe them together. Finally take the last four and pair them off mirroring the drives in each pair together.
Take this huge mess of drives and lump them into one massive volume. Then create a "safe" and a "fast" folder on that volume, assiging the "safe" to the mirrored drives and the "fast" folder to the stripped drives.
Whoa, that's a lot of work with Disk Utilty!
Now all three Xserves get hooked up through a mess of redundant firewire cables, utilizing a number of firewire hubs to prevent any single point of failure. Each server can see and concurrently read and write to the total volume of firewire drives, however files put into either the "safe" or the "fast" folders get specifically stored on their respective RAID sets.
Finally throw a firewire tape drive into the mix which can pull directly from the firewire drives themselves without needing to be directly connected to any server.
You are beginning to scratch the surface of what Xsan can do for you. Take a few minutes to think about this.
What it can do
So now that you have an idea of concept behind storage area networks, we can talk about Xsan.
Xsan builds volumes out of Fibre Channel (FC) storage devices. Coincidentally enough, the Xserve RAID is a Fibre Channel storage device! Note the "e" on the spelling of Fibre. They say this is to distinguish Fibre Channel from Fiber optics, but I think it's to sound pretentious. However, please please don't pronounce them differently and attempt to make Fibre sound French. They are pronounced the same.
The FC devices present RAID sets, or LUNs, to the FC network. Xsan groups these LUNs together into storage pools of similar sizes and types of storage. These pools are then grouped together into volumes that show up on your desktop.
When data gets written to the volume, if it doesn't have a specific storage pool associated with it, like the "fast" and "safe" folders in the example above, the data is stripped across all of the pools.
As an Xsan admin you have the ability to make your storage as safe, as fast, and as redundant as you want by combining LUNs and pools into the most optimal solution for your needs.
Of course once you have the volume setup you still need a way to control who can write to what when. For this purpose Xsan uses a metadata controller (MDC). No, this is not like the metadata on your digital photos nor does it have anything to do with what Spotlight in Tiger will do.
Instead this MDC keeps track of who's writing to what and where the file actually live across the dozens of disks that may make up your Xsan volume. It's the MDC that allows an application to get an exclusive lock on a file when the volume might be mounted by up to 64 different systems and really utilize the functions of the SAN.
To account for any single point of failure, the MDC is backed up by one or more secondary metadata controllers which automatically take over if the primary goes down.
Also, Xsan can take advantage of Fibre Channel multipathing to allow for redudandent connections to redundant Fibre Channel switches so the failure of any Fibre Channel cable or switch will not result in an interruption of service as long as you have another FC connection to fall back on.
Other operating systems
Unix/Linux and Windows are happy to play along in an Xsan environment. ADIC makes client software for non-OSX systems which is entirely compatible with Xsan. However, do take care to hold onto your hind quarters when getting quotes on this. In most cases it's cheaper to buy Xsan, and an Xserve to use it on, then it is to purchase a client license for the other operating systems.
Gotchas
Xsan shines the most on transfers of very large files. Due to the overhead of the metadata system, and the large minimum block sizes of multi-TB volumes it's not going to be as efficient with collections of really small files.
For example, you probably won't see much in the way of performance gains on mail servers, as these are primarily comprised of millions of files that are a few k each. This is not to say that these small files shouldn't be used with Xsan, just that you won't see the speeds that you would with larger files.
Also database servers are not a particular good match with Xsan. Unless you've paid a massive amount of money on your database solution it isn't designed to have multiple hosts writing to the database file at the same time, so the Xsan is pretty much lost on this. Read up on your database to see what the vendor recommends for this situation.