Self-Signed SSL Certificate = Slooooooooooow Sending

I use Self Signed SSL Certs on my server. I'm looking for encryption, not 'authentication'. So finally in 10.6, Mail.app started to REMEMBER when you checked the box to always trust a given certificate. So I've been using my certs and it's been working great with one issue. It takes a very long time to send mail. It used to be instant, now it's slooooooooow. I'm assuming this is somehow due to the SSL certificate. Its a real problem. Sometimes you wake a computer up from sleep, to find that an email you sent before you put your Mac to sleep, hasn't sent you. I was thinking about just buying a real cert. But then I figured, WTF. I don't need a real cert. What I need is a way to figure out exactly why it's taking longer to send, and fix it! My server is very fast, and significantly under-loaded, so I should be able to pull of near instant sending!