SSL certificate replacement gone badly awryNeed help.

[HEMI]

My first post, and it’s bad. 🙁

I administer a pile of Macs and an OS X Server machine. It’s a dual 1.33GHz XServe running Server 10.4.9. We use the XServe for OD, AFP, mail, our database (MySQL), web, etc., most of these services using SSL encryption. In short, we put all our eggs in one basket…Not my planning, but whatever. Anyway, to the problem at hand…

We’re a federal agency that is allied with a major university. Our lab is on the university campus, our address space and DNS is done through the university, etc. We obtain our SSL certificate from them, since they re-sell for Thawte. Our existing certificate expires on 18 May 2007. Last year, I manually generated a (passphrase-protected) key using openssl, then generated a csr, also using openssl. I sent the csr to certmgr@ and waited. They sent me back our shiny new crt, and after removing the old crt in Server Admin, the new one imported fine. That was last year.

I followed the exact same procedure this year: new passphrase-encrypted key, new csr, new crt. I deleted the old cert using Server Admin, but Server Admin refused to import the new one. I successfully imported the old cert, which I’d backed up beforehand. Of course, the procedure I used last year doesn’t work this time. Good job, Apple. After looking at the system log, I get an error -29225 (IIRC), which seems to indicate some sort of duplicate entry or conflict with something else in the keychain.

So, I head upstairs and inform my boss of the situation. He says to keep working on it. I come back downstairs, verify that I’m using the correct password by decrypting the key, then remove the old cert and try the import again. No dice. I figured something was wrong with the cert and I’ll just re-import the old cert until I get it sorted out. I wish I were that lucky. I tried to import the old cert, but it fails. Now I’m in trouble, and here’s where the mess starts to snowball out of control.

I start digging for info. I find several pages talking about certtool, Keychain Admin, etc. I try a few things, like making sure there’s no remnants of the old cert in any keychains, etc., but nothing works. At this point apache won’t start, no one can log in to their accounts that auth against OD, e-mail is being bounced and my day just got really bad.

I followed some suggestions to replace individual services’ certs with the old certs. I create a dir, /etc/certs-2006, then copy the old crt/(passphrase-protected) key/chcrt in to there. I point Apache at the cert files in /etc/certs-2006 and, after starting the service, the site’s back up. I finally caught a break, but the fun was just beginning.

I decide to get OD going again, so I point OD at the cert files in /etc/certs-2006. OD doesn’t seem to have any provisions for being restarted via Server Admin, or at least it wasn’t available to me at that point, so I decided to reboot the machine. This was probably my first major mistake.

The box stays down, for all intents and purposes. I can’t get to it via ARD or ssh. Great. So, I stroll up to the server room, along with my boss, who has remained incredibly calm about the whole situation. The machine’s hung at the last part of the loading bar; the bar itself isn’t even scrolling any more. So, we reboot it in to single-user mode, run a fsck, reboot in to safe mode, clear some caches, reboot it again and it finally gets to the login window. I try logging in as root, and it just drags. Finally, it lets me log in.

After poking around for a bit, we decide to fall back to our backup server, with is an almost-exact duplicate of the working machine, up till about a few hours ago (yay rsync and bless). I can get everything back in sync once we get all of this sorted out. The backup server is up and running now, working well enough that people can still be productive, but I’d really like to get this sorted out.

I found a page that details how to renew a cert…I really didn’t want to renew it, just replace the existing cert with a new cert that’s for the same FQDN.

So, I realize I did some things wrong, first and foremost of which being the assumptin that Apple wouldn’t change major things between non-major releases. This worked fine last year under 10.4; it doesn’t now. Second, trying to force OD to use the new cert was probably a mistake. The case could be made that I should have researched this better, but coming from my *nix (primarily FreeBSD) background I assume that what worked for the same minor version will continue to work, so I didn’t bother looking in to it.

I need some help. This has been a total disaster beyond anything I’ve had to deal with in my *nix admin days. I’m really not quite sure where to go here. The current state of things is the real server is turned off, takes forever to let root log in to the console, none of the crucial services run and no one else can log in to their machines since all of their account info lives in OD and their homedirs live on AFP. There’s also no certs—either new or old—installed system-wide as far as Server Admin is concerned. The backup server is humming (more like crawling) along, but I fear for our backups. The backup server has the old cert installed and it still works fine.

If anyone could give me some advice, some pointers or any help whatsoever I would greatly appreciate it. Should I create a non-passphrase-protected key and try importing that along with the new crt?

[HEMI]

Anyone know which keychain/keystore/whatever this thing would be conflicting with?

I’m to the point where I’m just going to restore from a pre-SSL-song-and-dance backup, then see about following the listed procedure on the page I linked to…

I can’t believe there’s no solution for this.

[Author]

Posts