So I decided it would be a good idea to update my server since it had been a while. I think my first mistake was using the ServerAdmin app instead of VNC'ing into the machine. So I ran the 10.3.8 update no problem. Then I rebooted and did the 002 update no problem. Then I rebooted and did the 003 update. Upon rebooting after that, my server was dead. Every time it boots it seems to completely lockup, I can VNC in and see a login screen but i cna't type. ssh and http ports are open but I can't connect to anything. I have them (the people at the data center, its colocated 1000 miles away) help me make it boot off my backup drive. I ran every test I could think of, Disk Utility, Wisk Warrior, TechTool. I tried it all, even checked all the prefs files and the fonts. No really major issues were found, the minor ones were all repaired. So I rebooted off the server drive and again I get nothing. So as of right now, lean versions of all my web sites are running off my damn cable modem at my house, and they are going to ship my server back to me. What an incredible pain in the ass. Oh yeah and another intersting thing, when I did boot off the Backup HD, it told me that my main drive (a two drive mirror) was out of sync and it had to rebuild it.
Anyone else have anything like this happen? I really can't even guess how it happened, nor can I think of anything I did wrong that could have caused it. I have a pretty tight firewall, so the only conclusion I can come to is that, to prevent this, I'll never run any more software updates for the life of the server. Its goona run 10.3.8 sec update 003 until it is retired. Luckily I haven't put any paid customers on this server yet but I have several lined up!