Kernel Panic caused whole system to cave in

Hi everyone,

During the week we’ve had weird issues with our network of around 120 computers and couple of servers (Xserve 10.5.5 and Windows 2003 server). We’d get huge spikes in ping response and/or 20 seconds of drops, inside or outside our network, and then things would come back to normal again. I checked through all sorts of logs, couldn’t figure out what it was.

Last night our entire system went down. No DHCP, no DNS, complete loss of packets. Hooking my laptop into the servers directly proved the servers were working fine. Plugged into our managed switches and bingo, no network. Removed bottom tier, same issue. Removed top tier, everything went back to normal. Our litmus test was to open Remote Desktop, which essentially pings everything on our network to see if anything responds. We kept pinging a known static IP on the network, and it would fail every time we opened it. We strongly suspected switches at first, but we removed cable after cable to see if it was a singular device.

We eventually got to the last couple of cables, removed one and tried our test. No issues. I suspected this could be a case of the straw breaks the camels back, so we plugged in everything but that cable and tested again. No issues, network going strong. Traced cable back to find a MacBook Pro in kernel panic.

What had happened was that the MacBook Pro kernel panicked on logout, right when ARD client changes status. It may have had a piece of code looping in memory, possibly sending out masses of incomplete packets whenever pinged by Remote Desktop. Apparently this computer was kernel panicking every time it logged out (quite often). As an admin, I leave Remote Desktop open at all times. This explains the spikes. But for this computer to bring down our entire network is just crazy! I’m in the process of gathering logs etc, so can post those later on.

What are your thoughts on this? Any idea on why a kernel panic bought our system to its knees?