SSL only on port 80?

I've set up my SSL cert on my server, If I call it with https://secure.mydomain.com:80/ it will work. If I call it with https://secure.mydomain.com/ it won't work. I have 443 open on my firewall, I have Listen 443 in my http.conf (not the _macosxserver.conf). Is there something I'm missing?