Solution!

Hi all, I finally got it to work. It turns out that this was not specifically a Sonicwall problem. Instead it looks like might be a Racoon implementation issue. If you remember, the problem was that the Racoon client was behaving as if it wasn't receiving the phase 1 response from the Sonicwall. However what I found via tcpdump was that the Sonicwall was in fact sending the response. It's just that the packet got fragmented because it was larger than the MTU (which on my Sonicwall is set to the maximum 1500 bytes). On a hunch I tried to work around the problem by reducing the packet size and eliminate the fragmentation. By going from a 2048 to 1024 bit certificate and by removing some v3 extensions (which get incorporated into the cert as a side effect of our CA implementation), I got the Sonicwall's response packet size below 1500 bytes. So after making that change (and messing around with the SA definition till I got it right ;-)), I was able to connect from a Mac OS X client to the Sonicwall using certificate based authentication. Very cool! I honestly don't know enough about either IPSEC or the Racoon implementation to determine if this behavior with fragmented negotiation packets is a Racoon bug or not. But I posted to the Racoon mailing list in the hopes that the authors will take a look at the issue. Best regards, -- Allen Cronce