WOW! Ask and you shall receive. They set up an Xsan forum...cool.
Greetings,
I will begin by saying I am not a UBER-Geek. I more of a psudo-Geek. I don't have any Apple certifacations (yet), I'm just the guy at my company who takes care of all the Macs. That being said, I think there might be some people out there like me. So if you are thinking of setting up an Xsan, or are trying to set one up, this might help a bit.
I will give a couple of sentences on our setup. I work for a broadcast TV/radio station. We have three Avid Media Composers on an Avid Unity (Avid's Xsan). We also have a tricked-out HD Final Cut Pro suite, that is doing 1080i uncompressed. Lastly, add in three graphics stations, an audio station, and a few print guys on macs.
First off, When we bought the Xsan, we spent $20K. This was wildly under what we should have. We bought one Xserve (to be the P. MDC), One 7TB Xserve RAID, an Emulex Fibre Switch (the cheapest on the Apple site), three Apple Fibre cards, and the cables. AFP548 has done a couple of articles on Setting up an Xsan, so I won't cover what they did.
I will say, If you are used to the Avid Unity SAN, then you are under the impression that if your MDC dies, all that happens is you loose access to the SAN until a MDC comes back up. Avid stores the metadata on the same arrays as the user data. This is NOT the case of Xsan. You see, where as the metadata is stored on the SAN, the MDC keeps a very small folder located at /Library/Filesystems/Xsan/config. This folder contains a few .plist files that if go missing or get trashed will cause you to loose ALL data on the Xsan. Good to keep this in mind. Apple will tell you that this is why you should have a second MDC. When I was looking over the afp548 stuff, and all the other documentation on Xsan, I just assumed they kept putting in a second MDC to prevent any downtime. That is one reason, the other is the MDC keep files locally that affect the WHOLE SAN. So mirror those local OS drives on your MDCs (as a minimum)
Now this second little tidbit that I've learned will keep most who visit this site laughing, but I offer it to those who don't know. Nowhere in any documentation have I read the following. NEVER CHANGE THE IP OF AN XSERVE ONCE YOU HAVE SET IT UP. As I stated in the begining of this post, I'm not an UBER-Geek. I have never had OS X server before buying the Xserve we bought for our Xsan. Although OS X Server has a System Preferences pain just like OS X, and in that sys prefs there IS a NETWORK option that looks the exact same as OS X client. You should NEVER change the settings.
One of Andrea's things during her ACL slide show at MWSF was "plan, plan, plan, and then plan some more" and this is very true about your Xsan. Once your installed, and have created a volume, with the LUNs, and the storage pools, and the yadda-yadda. DON'T change that IP. Now, the reason I did change it is because I wanted to move to a region of IP addresses that wouldn't have any other communication. When I did this, it changed the files located in the config folder I spoke of earlier, and as I said, all info on the SAN went south.
I had to restore the files, but then I started chasing ghosts all afternoon due to the fact that lots of little things kept acting strange. I finally had to open each of the files in PICO to make sure the IP address was correct. The Apple guys kept telling me just to reformat, and reinstall. The reason I think I'm safe is due to the fact that I don't have any other OS X Server Services turned on my MDC. (I hear giggles in the background, but I'm not proud, we all learn by doing)
Wow, this is WAY longer than I wanted it to be. I'll finish by saying I don't recomment the Emulex switch. Go QLogic. If you see an Xsan at a trade show it will most likely be using the QLogic 5600 (and yes there was an Xsan shoved into a dark corner of the Apple booth at MWSF, and yes it was using a QLogic). It is more expensive, but your fibre switch is the backbone of the whole SAN. Don't skimp.
Also, They don't really tell you this unless you do a WHOLE lot of reading. But to do HD uncompressed using an Xsan you have to have a minumum of three Xserve RAIDs. Xsan only achieves speed by splitting the read/writes across RAID controllers. HD 8-bit uncompressed required 160MB/sec sustained, and that requires more XSRs.
Lastly, (I swear) If you are planning on finally setting up that OD server you've always wanted, don't plan on using network HOME folders if you need to run FCP. FCP wants local Home folders. You can still set up an OD server (on your secondary MDC) for user authintication I THINK (I'll let the masters here correct me). I havn't gotten to try setting up an OD service, so I'll have to report on that as time goes by.
Thanks AFP548 for the opportunity to have a Xsan forum!