A True Tale of an Xserve install

Notes of the install process for the new AFP548 Xserve

A discussion of some best practices for installing servers. Specifically those bound for co-locations facilities, but good ideas for any install.These are the notes I took while setting up the new Xserve for AFP548.com. A lot of people have asked for a step-by-step installation of OS X Server, so I figured this was as good of an opportunity as I was going to get to do this.

Your setup will almost certainly be different than this. However this document should still give you a good idea about what to think about when setting up a machine.

Purchase – 2 weeks

I’d put in the order for the Xserve in late June. Ingram was listing a mid-July deliver date, so I was pretty amazed when it showed up the first week of July. I was in Santa Barbara teaching at a Mac retreat when it showed up, so my neighbor, a Mac user, had to endure a few days of looking at a new Xserve, that he couldn’t touch, sitting in his front hall.

The Xserve box is big, and unlike PowerBooks it doesn’t ship in a plain brown wrapper. I got the base-line single processor G5 Xserve. I can’t imagine that serving out AFP548 and some other projects is going to make a 2 Ghz. Xserve sweat.

I unwrapped the box. Threw the manual to the side and grabbed the power cord and the server serial number from the pile of documentation. The server came with one 80 GB drive and I had two other 80 GB modules that had arrived earlier for a total of 240 GB of storage. I put in the other drives and powered on the machine.

Install – 1 hour

Now there are a number of ways that you can install a new server that doesn’t have a video card. By far the hardest is to use the CDs. Instead, my preferred method is to restore a freshly installed OSXS image onto the drive. So I used the front panel mode of the Xserve to get the server booted up into target disk mode. At which point I connected it by the FireWire port on the front to my laptop. Using Disk Utility I created a mirror RAID out of the drives in bays 1 and 2. I then partitioned the remaining drive into a 60 GB and a 20 GB partition. The RAID will be used for booting and the other drive will keep a copy of the RAID volume, or at least 20 GB of it. I’ll put another bootable system and incremental backups on the remaining 60 GB, just in case.

After the RAID was built, I used Disk Utility to restore a new OSXS 10.3.3 image onto it. I use this image, and the corresponding OSX client one a lot. I originally created it months ago by installing OSXS onto a new drive. Before the server had a chance to reboot after the install was done I booted off another drive and imaged the un-configured drive using Disk Utility. After the image was done I used Disk Utility to Scan for Restore on the image.

Whenever system updates get released I mount the disk image, convert it to a read/write image and run the update against the disk image. Once the update has completed I close the disk image and then rescan it in Disk Utility. This way you’ll always have a current system image. Take a look at Josh’s article on disk imaging for more info.

Since the image was scanned for restore, using Disk Utility to restore it onto the RAID set on the Xserve takes only a few minutes. This is the difference between a file copy and a block copy. If you don’t scan you get a file copy which might take 20-30 minutes.

Once this is completed make sure there is an ethernet cable in the bottom ethernet port on the Xserve and boot it. While this is going on, install the OSXS Admin Tools onto your admin machine. You can’t just drag copy the Server folder from an OSXS install for the remote installations, so make sure you use the package installer.

Once the server comes back up you can use the Server Assistant in /Applications/Server to configure it. The assistant will use Rendezvous, err OpenTalk, to find your installed, but unconfigured, server. Log onto it using the first 8 characters of the serial number as the password. Capitalization matters here.

From there it’s a pretty straight-forward configuration. My only advice is to always set your server to be a Stand Alone machine at the initial configuration. Not only does this seem to smooth things out, but you’re probably going to be using your own server to run DNS. Since your KDC won’t start up right if DNS isn’t working, making your server an OD Master at this stage will be problematic.

Update – 15 mins

Before going further, update your system. You can do this through Server Admin (SA) although I find it a bit easier to just use the softwareupdate command over ssh. Either way, get your OS up to the latest and run the security patches.

DNS – 15 mins

If you already have good DNS hosted on another machine, you can skip this section.

Otherwise the GUI for DNS seems to have become less problematic in recent updates. I do find it best to be patient with it, though. So create a new zone file using your server as the NS, nameserver record. Then add an A record in for your server. Select the box to automatically create a reverse record and you should be done with as much as you need to do for right now.

If you have other machines, or plan to host mail on this server, it would be a good idea to put the other records into your server, but you can do that later if you need to.

You can test your records using “dig @127.0.0.1 ” before putting 127.0.0.1 into your network preferences as the primary name server. To actually make the change to use yourself for DNS you can use the networksetup command from the command line.

Open Directory – 0 mins

My server is going into a co-lo and I did the set up at my office. As such the IP was going to be changing shortly after the install. Because of that I waited to do any work with Open Directory. However, if you have no intention of moving the machine, go ahead and set yourself up as an Open Directory Master. That is after you have triple checked that your DNS works first.

Apple Remote Desktop – 10 mins

It’s about this time that I set up ARD. I didn’t use ARD 1.x much. It was decent enough, and I liked to demo it for new admins, but it just wasn’t robust enough for what I wanted to do a lot of the time.

All that has changed with version 2.0 though. It’s a .0 product, so be careful with it, but so far it’s made me a believer. I like that I can send shell scripts to the remote boxes. I like that I can scan the network much more reliably. And I really like that I can build a custom package installer for the client software that I can install from an ssh session.

The client installer gives you the option of creating a new user for the ARD connection. Go ahead and do this. It keeps things nice and organized. Choose a nice hard password since once you save it into your admin console you’ll never have to use it again.

Once you install this package from the command line, ARD is immediately useable. Score! Now you really didn’t need to buy that $99 video card for the Xserve.

Developer’s Tools – 20 mins

You’ll need them on your server to add in all the cool stuff, so use either ARD or the installer command from the CLI to get these on your system. OS X Server should come with a copy of the Dev Tools CD in the box. If you can’t find it or don’t have one, you can download the installer from Apple’s Developer site.

SSL – 20 mins

I like to roll my own, mostly cause I’m cheap and it’s fun to do. However, SSL certs are getting real inexpensive. Check out Instant SSL for a cheap price. Shop around though, since there are other companies that might be lower.

Either way, it’s good to get this out of the way early so you don’t have to worry about it later. Plus as you’re configuring the other services you’ll have the appropriate SSL cert ready to go.

Mail – 3.5 hrs

Mail is something close to my heart and I’m certainly rather particular about it. I really like the Amavisd/SpamAssassin/ClamAV combo that you can find in my article. This takes a bit of installing but I find it well worth the effort.

After that I added cyradm, which no self-respecting Cyrus admin should be without. If you don’t have this installed on your Cyrus box you might as well be running just a POP server. Cyradm, which I’m declaring should be pronounced “Sir-Adam” for ease of use, will allow you to change IMAP permissions on your mailboxes and set quotas. A little-known fact is that if you set the quotas for the mail box using Server Admin they are only valid for the user’s Inbox, not the other folders that user might have. Cyradm allows you to change that. I’m working up an update to my article on cyradm, so that should be posted soon.

With cyradm working you can also set up shared spam and ham mailboxes to allow SpamAssassin to automatically learn from it’s mistakes.

Next I installed Sieve. Sieve lets me make about a dozen folders in my IMAP box and filter all my list server mail through to the right box. Plus I have a stupid-simple, and mailing list aware, vacation system for when I go to the islands.

Set up SSL for SMTP/POP/IMAP if you going in that direction, and you should.

Finally, you should at least think about setting up an automated backup of your mail system. More info here.

MySQL – 10 mins

If you need this, the easiest way to get it going is to use the simple little Apple GUI in /Applications/Server/. This will get it running and automatically edit /etc/hostconfig so that MySQL will always start up.

After starting it you need to set the root MySQL password. Do this by

mysql -u root -p 'supersecretpassword'

Hit return when it asks you for your current password, because that’s what it is, a big fat blank just waiting to get hacked. Obviously swap “supersecretpassword” for a password of your choosing.

Once you have MySQL up and running you can use the mysql command, phpmyadmin, or CocoaMySQL to do the rest of your work.

Web – 30 mins

This actually took me about two days because I had to setup Geeklog and migrate over the old bbs. However, if you already have your html done, which really isn’t part of a “server software” install, the web configuration should not take you long.

I used Server Admin to do all of this, because there isn’t too much you should need outside of the Apple Tools.

As a rule of thumb I turn off the performance cache anyplace that I see it. I also make sure that the php and ssl modules are enabled.

If you are going to use WebMail, which you should, please, please, please wrap it in SSL. Otherwise your mail passwords, and e-mail, are going across the web wrapped up in a nice “hack me now, please” ribbon. It’s simple to do, so do it. Don’t allow WebMail access on a non-ssl site. This should not be an optional feature.

I was tempted to install the updated php package from entropy.ch here. However, Apple has been good lately about keeping php up to date and fairly well full-featured. So I’ll hang off on that until I know I need it.

VPN – 20 mins

Here’s where I started to get a bit sly. My co-lo is only going to give me 2 IPs. That’s fine, since I’ll probably only have 2 Xserves in the rack (I’m thinking about putting in a second one for testing purposes). However, when you start doing VPNs you’ll need some IPs that are local to the server for the VPN clients. Well, you certainly don’t want to fork over real money to your co-lo firm for an IP that you’ll only use once in a while.

Instead alias a stub network onto your server. I gave mine a 10.99.37.0/24 address. It’s non-routable and I was pretty sure no one else at the co-lo was going to be using that range (although I did ping it to make sure before I started using it). Now in the Server Admin GUI you can set up a client IP ranges inside the 10.99.37.x network and not have to worry about buying more IPs.

You’ve got two flavors of VPN: L2TP/IPSec and PPTP. The former is a bit more secure, but as long as you keep PPTP set to 128 bit encryption either is good enough for what you’re going to do with it. I like to set both up, because PPTP has a tendency to work better with older network equipment. So try L2TP first and fall back if necessary.

Also set up a private network in the GUI that just encompasses your stub network. That way only your traffic going to the server will go through the VPN, instead of everything coming out of your client machine.

Firewall – 20 mins

Now that you have your VPN setup raise your shields.

I prefer hardware firewall appliances, but for one server in a co-lo, dropping a thousand dollars on a good firewall seemed a bit of overkill. Instead use Server Admin to block all but the very necessary services from coming in over the internet. Lock down the public IP heavily. Leave ssh, and your VPN ports open in addition to mail and web if you are running those services. The GUI has a fairly exhaustive list. Please, if you spent the time to set up SSL for mail services, only allow POP and IMAP connections if they are using SSL. So block 110 and 143 at the firewall.

Remember how excited I was about ARD 2.0? Block it at the firewall. The actual ARD session is unencrypted so force yourself to get a VPN connection up and running before you use it. I prefer to force the Server Admin tools to run over the VPN too. Not that they really need it, but more just cause I’m anal that way.

A sneaky trick I’ve learned when doing firewall work remotely. Set yourself up a deadman’s switch. Before you turn on the firewall set up a sleeper command, either through a cron job, or just by writing a short script that uses the sleep command. For example, when working with ipfw commands remotely I set up an “ipfw flush” command in the crontab for an hour or so in the future. If you bone the config, just grab a beer and spend some time thinking about how totally hosed you would be if you didn’t have that sleeper in there.

Granted your pants will be down for a bit when the command kicks in. But better that than not being able to use your box until you can get out to the co-lo.

UPS – 10 mins

Buy a UPS. Now.

I live in tornado country. With tornadoes come storms and power outages. I have had to reformat my test server every few weeks because the power had gone out 4 times that month and the drive just clicked when I tried to boot it up.

Buy a UPS with a USB connection. Throw away the vendor software that comes with the UPS and just plug the USB cable into your server. OSXS will automatically shut itself down when the UPS runs out. You can use the Energy Saver preference pane to set this up.

Backup – 1 hr

Pick your poison: Retrospect, BRU or Rsyncx. If you are using tape Retro or BRU are pretty much your only options. BRU is great cause it works from the command line, so you can run your backups from the ssh client on your fancy phone. Retrospect has a better GUI and has been much improved in the last version. I personally lean towards BRU, but try them both.

However, for the AFP548 server I’m not using a tape drive. Instead I’m using Rsyncx to synch the boot mirror RAID to the 20 GB partition on the remaining drive in the box. Since there isn’t that much actual data on the drive, I tar up the mail and web server files and put them into daily folders on the 60 GB partition. Of these I Rsyncx a remote copy of the web files to my home machine. Never can be too careful.

Gratuitously open the Xserve in front of the other co-lo users – 20 mins

It’s such a thing of beauty inside that metal shell, that you really should show it off.

I picked up a Gig of RAM on the way to the co-lo just so I could have an excuse to slide it out on the rails and look at all the pretty lights on the inside.

Install in the co-lo – 10 mins

There’s nothing more enjoyable than sliding a G5 Xserve into a rack full of butt ugly HP servers. Well nothing except for the look on the co-lo sysadmin’s face when he asks you where you’d like to set up the monitor to do the configuration, and you reply “Monitor, hell son, it don’t even have a video card. This ain’t no game machine, what do you need graphics for?”

I’ll remember that look for a while.

Finish up the install by using ARD from your laptop or ssh to make sure that the new public IP address is first on the list in the Network Preferences. Then run the changeip command to make sure that everything has been changed that needs it.

Open Directory – 5 mins

Now is when you can make your server an Open Directory Master. Well, after you have double checked the DNS again.

You should be at your final IP, so use slapconfig from the CLI or Server Admin to promote yourself to being a master.

Watching the sysadmins at the co-lo call their friends to come over and see the new Xserve – 10 mins

Ahhh, it’s the little things.