I’m just starting a new Xsan, and here is what I’ve learned

[Osglith]

WOW! Ask and you shall receive. They set up an Xsan forum…cool.

Greetings,

I will begin by saying I am not a UBER-Geek. I more of a psudo-Geek. I don’t have any Apple certifacations (yet), I’m just the guy at my company who takes care of all the Macs. That being said, I think there might be some people out there like me. So if you are thinking of setting up an Xsan, or are trying to set one up, this might help a bit.

I will give a couple of sentences on our setup. I work for a broadcast TV/radio station. We have three Avid Media Composers on an Avid Unity (Avid’s Xsan). We also have a tricked-out HD Final Cut Pro suite, that is doing 1080i uncompressed. Lastly, add in three graphics stations, an audio station, and a few print guys on macs.

First off, When we bought the Xsan, we spent $20K. This was wildly under what we should have. We bought one Xserve (to be the P. MDC), One 7TB Xserve RAID, an Emulex Fibre Switch (the cheapest on the Apple site), three Apple Fibre cards, and the cables. AFP548 has done a couple of articles on Setting up an Xsan, so I won’t cover what they did.

I will say, If you are used to the Avid Unity SAN, then you are under the impression that if your MDC dies, all that happens is you loose access to the SAN until a MDC comes back up. Avid stores the metadata on the same arrays as the user data. This is NOT the case of Xsan. You see, where as the metadata is stored on the SAN, the MDC keeps a very small folder located at /Library/Filesystems/Xsan/config. This folder contains a few .plist files that if go missing or get trashed will cause you to loose ALL data on the Xsan. Good to keep this in mind. Apple will tell you that this is why you should have a second MDC. When I was looking over the afp548 stuff, and all the other documentation on Xsan, I just assumed they kept putting in a second MDC to prevent any downtime. That is one reason, the other is the MDC keep files locally that affect the WHOLE SAN. So mirror those local OS drives on your MDCs (as a minimum)

Now this second little tidbit that I’ve learned will keep most who visit this site laughing, but I offer it to those who don’t know. Nowhere in any documentation have I read the following. NEVER CHANGE THE IP OF AN XSERVE ONCE YOU HAVE SET IT UP. As I stated in the begining of this post, I’m not an UBER-Geek. I have never had OS X server before buying the Xserve we bought for our Xsan. Although OS X Server has a System Preferences pain just like OS X, and in that sys prefs there IS a NETWORK option that looks the exact same as OS X client. You should NEVER change the settings.

One of Andrea’s things during her ACL slide show at MWSF was “plan, plan, plan, and then plan some more” and this is very true about your Xsan. Once your installed, and have created a volume, with the LUNs, and the storage pools, and the yadda-yadda. DON’T change that IP. Now, the reason I did change it is because I wanted to move to a region of IP addresses that wouldn’t have any other communication. When I did this, it changed the files located in the config folder I spoke of earlier, and as I said, all info on the SAN went south.

I had to restore the files, but then I started chasing ghosts all afternoon due to the fact that lots of little things kept acting strange. I finally had to open each of the files in PICO to make sure the IP address was correct. The Apple guys kept telling me just to reformat, and reinstall. The reason I think I’m safe is due to the fact that I don’t have any other OS X Server Services turned on my MDC. (I hear giggles in the background, but I’m not proud, we all learn by doing)

Wow, this is WAY longer than I wanted it to be. I’ll finish by saying I don’t recomment the Emulex switch. Go QLogic. If you see an Xsan at a trade show it will most likely be using the QLogic 5600 (and yes there was an Xsan shoved into a dark corner of the Apple booth at MWSF, and yes it was using a QLogic). It is more expensive, but your fibre switch is the backbone of the whole SAN. Don’t skimp.

Also, They don’t really tell you this unless you do a WHOLE lot of reading. But to do HD uncompressed using an Xsan you have to have a minumum of three Xserve RAIDs. Xsan only achieves speed by splitting the read/writes across RAID controllers. HD 8-bit uncompressed required 160MB/sec sustained, and that requires more XSRs.

Lastly, (I swear) If you are planning on finally setting up that OD server you’ve always wanted, don’t plan on using network HOME folders if you need to run FCP. FCP wants local Home folders. You can still set up an OD server (on your secondary MDC) for user authintication I THINK (I’ll let the masters here correct me). I havn’t gotten to try setting up an OD service, so I’ll have to report on that as time goes by.

Thanks AFP548 for the opportunity to have a Xsan forum!

[MDhaliwal]

Cool post. Always nice to see people putting real world experiences to help out other folks. Its the only way to really learn!

So, yeah, a second MDC is really what you need. You can also use cvgather to grab up that Xsan config. It’ll tar it up nicely for you. This is exceptionally valuable when troubleshooting an Xsan deployment.

Qlogic is a very nice switch company to work with. I’ve always liked the 5200 for small to medium sized Xsan deployments as they include the 10Gb uplink, for stacking. This way, for smaller shops, your not blowing all of your allocated budget on a switch that you won’t fill for years to come.

Ideally, leave your OD installation on its own. If you don’t have the cash for extra Xserves, you can deploy and OD on the MDC. I usually suggest doing opposites. Take the secondary MDC and make it the ODM. Take the high priority MDC and make that the ODR. This way you have high availability to all services you need for the Xsan. As long as your not planning on plugging large amounts of users into that LDAP directory, you should be OK. Usually, the OD is just being used for Xsan users in this case.

Just my $0.02.

[maccanada]

Yes, Mike is right – cvgather is your friend – run it – run it regularly and keep a copy of the tar file somewhere other than on your MDC.

IP addressing – yes there are several files living in /Libray/Filesystems/Xsan/config with IP addresses in them. Changeip was never designed for an Xsen environment so changing IPs can get messy. Like you say, best to make sure the initial IP is a permanent one.

It’s good practice to use cvgather, along with saving all your configs from serveradmin, and from your FC switch should the worst happen.

The Emulex is not a true Fabric switch and as such won’t give you the same performance and scalability that comes with one.

Three XSRs sounds like overkill for uncompressed 8-bit (AJA reckon 1920 x1080 8-bit @ 29.97 fps to be around 125MB/s and 10-bit to be around the 165MB/s mark) – 3 controllers, maybe; but 3 whole RAIDs will give you (assuming one controller for metadata) 5 controllers – that will give you something in the region of 400-500MB/s. Of course, it all depends on how many workstations you have wanting to work simultaneously.

As Mike said, in an idea setup, you’d have 3 Xserves – 1 Primary MDC. 1 Backup MDC and OD Replica, and 1 OD Master (which can also be another backup MDC). Final Cut can work with network homes (traditional direct-attached RAID, no Xsan in the mix), but it’s not recommended and I’ve not even tried in an Xsan setup. OD with local homes works just fine with Xsan. If you want to have different access rights for different editors, and to enforce quotas for them, you pretty much have to use OD. (To clarify – I’m talking about FC attached clients for a video environment Xsan)

Hopefully that hasn’t confused things any further 🙂

~Ian

[matx]

I can’t count all the times my neck was saved by having a dedicated Xserve for my MD backup. Especially when we switched to tiger, and we used xsan tiger beta in a production environment (crazy!). The MD master and backup kept passing control back and forth every minute sometimes.

Nice to know about cvgather, and maybe I’ll set up an OD replica too (i have a seperate xsserve OD master now).

-x

[rhcw]

[QUOTE BY= maccanada] IP addressing – yes there are several files living in /Libray/Filesystems/Xsan/config with IP addresses in them. Changeip was never designed for an Xsen environment so changing IPs can get messy. Like you say, best to make sure the initial IP is a permanent one.
~Ian[/QUOTE]

Are we saying that changeip will work but will do you no good if it is an MDC on which you are changing the IP after the fact?

On a related subject, lets say you set the IP to 10.0.0.2 and use a DNS that is good for the off-site location where you are pre-building. What bad things happen when you finally install on site and leave the IP alone but have to change the DNS to the one that is correct for the new network?

[Osglith]

Thanks to everyone for adding so much. I checked out cvgather today and it rocks. Now I just have to geek myself up to write a CRON script to copy the tarball to a thumb drive once a week, and I’ll impress myself.

Anyway, From what I’ve been told by some Apple reps (and I’ll let the really knowledgeable people concur/squash me here) Xsan doesn’t record the DNS in any of it’s config files, but does require DNS only in the since that UNIX does. UNIX seems to constantly refer to the DNS to make sure "all is right with itself, and the world". As long as you have a DNS server there to say "yes your fine, go forth and prosper", then it shouldn’t wig XSan too much (if you properly shut down the Xsan and it’s components **see K-Base article # 301787)

Anybody want to pummel me with their superior intellect? (shouldn’t be too hard to do)

[maccanada]

changeip doesn’t update any of the config files for Xsan, so if you run it and change the IP address of a machine, you will have to hand-edit all the Xsan config files to reflect the change.

If you miss one out, or mis-type something you could be chasing your tail trying to find it. It really is best to make sure your IPs aren’t going to be changing. To that end, don’t even think about using DHCP 🙂

~Ian

[MDhaliwal]

[QUOTE BY= rhcw] [QUOTE BY= maccanada] IP addressing – yes there are several files living in /Libray/Filesystems/Xsan/config with IP addresses in them. Changeip was never designed for an Xsen environment so changing IPs can get messy. Like you say, best to make sure the initial IP is a permanent one.
~Ian[/QUOTE]

Are we saying that changeip will work but will do you no good if it is an MDC on which you are changing the IP after the fact?

On a related subject, lets say you set the IP to 10.0.0.2 and use a DNS that is good for the off-site location where you are pre-building. What bad things happen when you finally install on site and leave the IP alone but have to change the DNS to the one that is correct for the new network?[/QUOTE]

changeip hasn’t been expanded to change the Xsan config files, just the server itself.

Out of curiosity, why build the MDC offsite? Just wondering. Ideally, you don’t want to move it.

[rhcw]

One might build off site if the client wanted to see the SAN running off site for acceptance tests.

[MDhaliwal]

Fair enough.

As long as the server can resolve itself properly in DNS, its usually OK. You can use something like scutil to statically assign the FQDN to the server. If it can use that same name in both locations, then your limiting any configuration issues that might arise.

The average Xsan probably won’t want to be moved, though. Multiple Xserves, multiple Xserve RAID units, fibre switches, gigabit switches, wiring, etc…that’s quite a bit to move around multiple times!

Ideally, you’d probably want to talk to an Apple rep who could provide you with a test deployment to see, like at one of the Apple offices. This way you could kick the tires on the Xsan without having to set it up multiple times, etc.

[three18bill]

[QUOTE BY= MDhaliwal] Out of curiosity, why build the MDC offsite? Just wondering. Ideally, you don’t want to move it. [/QUOTE]

we’ve built them offsite for a few clients, for a variety of reasons… The best way to do it is just configure a full test environment using all the same IPs. (ie, build the primary and back-up MDCs, OD masters, and Xraids in our office)

It totally depends on what the client’s specific situation is, and we try to cater to that.

-bill

[matx]

I’ll just add that yes, DNS should be setup correctly. But I have setup fully functioning Xsan without it. Of course, I used hosts files on all clients instead. A proper DNS server is better, but not necessary. All clients should have (must have!) static IPs, but everyone just has to know who and where everyone else is (hence the host files). Just adding my real world experience (DNS works and hosts files do too).

[Author]

Posts