Articles July 1, 2004 at 8:07 am

The great big Panther SSL article.

Everything you’ll need to know about setting up SSL on Panther Server

—by Joel Rennich, [email protected]
12 December 2003—Revised 16 December 2003
The purpose of this article is to give you an idea of what you can do with SSL in Mac OS X Server and how you can use that to secure as many services as possible. So, first we’ll talk some about SSL in general and how to create the certificates, then we’ll discuss what to do with those certificates.
SSL certificate creation
Before we start I’d like to point out that we are going to be creating home-rolled SSL certificates here. As such you will run into problems when connecting to your Server using applications like a Web browser. Most applications will allow you to ignore the fact that your certificate hasn’t been validated by one of the internationally recognized certificate authorities, but it’s still a pain.
I’ll show you how to get around that by importing in your own certificate authority onto your client machines. This presumes that you have control over all of your clients, so for internal use where you control both the server and the client setups being your own certificate authority is great. If you plan on doing business with the general public, such as using it for credit card processing on a Web page, I would strongly recommend that you invest the money in a “real” certificate.
If you do buy one at least take a look at They have really good prices and are Mac-based, so you can keep it in the family.
Also all of the openSSL work, such as generating and signing certificates, can be done on Mac OS X client.
1. Make a certificate authority (CA).
This should be done in a secure place, since if your CA gets compromised then all of your security goes out the window. A decent place for this would be on your most secure server or on your own machine.
It doesn’t matter where on the filesystem you do this; however, I personally prefer to create a CA directory in /etc:

sudo mkdir -p /etc/certs
cd /etc/certs

Right now this folder has fairly relaxed permissions on it. As soon as were done we’ll change that to greatly limit access to the folder. Now that we have our place we need to begin generatng the CA. We do this by making a certificate signing request (CSR). This example will generate a 3DES encrypted 2048 bit key. This is a rather high security key which means it takes longer to process. So if you feel the need you can scale it down to 1024 bits if you like. Although I haven’t had any problems using this with Mac OS X 10.2 and Windows 2000.
openssl genrsa -des3 -out ca.key 2048
You will be asked for a passphrase for this key. You need to both remember this phrase and keep it secure. Your entire SSL system will depend on this passphrase being secure.
Now that you have the request you can sign it into a CA.
openssl req -new -x509 -days 4096 -key ca.key -out ca.crt
You’ll be asked for the passphrase that you just set up. After that your certificate authority will be valid for 4096 days.
You now have a full blown certificate authority for your machine. From this we will base all of your other certificates from it.
2. Generate a certificate for your server.
You will need one for each domain that you have; i.e., and will each need one if you want to secure both sites.
So first we will generate a new private key.
openssl genrsa -des3 -out server.key 1024
You will be prompted for a password here also. This should be different from the password for the CA. Just remember it because you will need to enter it into Server Admin to get SSL running.
Now you need to generate a request with the private key.
openssl req -new -key server.key -out server.csr
Again you will be asked for a password. This is the one you entered in the step above. Then you will get a bunch of questions. They all really don’t matter except for common name. This needs to be the fully qualified name of your Web server, like If this is wrong you will get errors in the browser. Also: leave the challenge password blank.
Now we need to set up a few folders so that we can actually sign the certificate.

mkdir -p demoCA/private
cp ca.key demoCA/private/cakey.pem
cp ca.crt demoCA/cacert.pem
mkdir demoCA/newcerts
touch demoCA/index.txt
echo “01” > demoCA/serial

You can now actually sign the server certificate with your newly minted CA.
openssl ca -policy policy_anything -in server.csr -out server.crt
The password you are prompted for is the password you assigned to the CA, the first one, not to the certificate itself. If you need to create more certificates you will only need to do the last three steps for each.
Finally to keep things secret and to keep things safe, change the permissions on this folder.
sudo chmod 700 /etc/certs
Now you can take all of your pieces and make the sites secure.
3. Securing your web site.
Go into Server Admin and make sure that the SSL module is enabled in the modules pane under settings.
Then go to the site that you want to secure. Change the port to 443, click on the security button, and enable SSL by checking the box at the top. Then you need to open up some of the files that you have created in TextEdit, or any text editor, and copy and paste them into the three appropriate spots. Copy server.crt into “Certificate File.” Copy server.key into “Key File,” and copy ca.crt into “CA File.”
Finally, you’ll want to enter the passphrase for the server certificate into the “Pass Phrase” field or else you’ll have to be at the server everytime it starts up.
A few parting thoughts about securing Web connections. You will need a separate IP address for every SSL site that you have. There’s a complicated reason for this, but it involves how SSL connections begin and I don’t know of any way around this. In 10.2 you had to edit the httpd_macosxserver.conf file to get higher level encryption. This requirement seems to be gone in 10.3 as it defaults to using all ciphers.
When you are done your certificates will be stored in /etc/httpd/ssl.crt and /etc/httpd/ssl.key. Your site’s specific config is stashed in /etc/httpd/sites/your site’s name. So look in there for any specific info. Also the passphrase that you used is stashed in /etc/httpd/servermgr_web_httpd_config.plist, which is root-readable only.
If you aren’t going to be using your ssl certs for anything else but securing web sites, you can probably remove anything that doesn’t start with “ca” from /etc/certs just so you don’t have multiple copies of the same thing lying around. Keep the CA pieces, though, in case you need to make a new cert.
4. Securing LDAP
We run into a bit of a problem here. OpenLDAP doesn’t like a server key that has a passpharse associated with it. Postfix and Cyrus are going to be the same way. So remove the passphrase.
openssl rsa -in server.key -out serverno.key
Now go back into Server Admin. Select the Open Directory settings and go to the “Protocols” tab. Check the “Use SSL” box and then put the path to your certificates in the three fields.

Certificate: /etc/certs/server.crt
SSL Key: /etc/certs/serverno.key
CA Certificate: /etc/certs/ca.crt

OpenSSL runs as root, so it will be able to get into /etc/certs without any issues. As soon as you save this config Server Admin will restart OpenLDAP with SSL support.
The SSL configuration for OpenLDAP is stored in /etc/openldap/slapd_macosxserver.conf.
5. Securing SMTP
Postfix can be setup to use the same certificate as the one you established for openLDAP. However, it wants to have both the key and the certificate in the same file. This is easily done.
sudo cat /etc/certs/serverno.key /etc/certs/server.crt > /etc/certs/server.pem
Now link that file to what Postfix is looking for.
ln -s /etc/certs/server.pem /etc/postfix/
Now reload Postfix through the GUI or by doing this from the command line.
sudo postfix reload
And start using encrypted SMTP services.
The SSL configuration for Postfix is kept in /etc/postfix/
6. Securing POP/IMAP
Cyrus can use the same certificate as Postfix, but it needs to be accessible by the cyrus user. That requires relaxing the permission a bit on the certificate store.

sudo chown :mail /etc/certs
sudo chmod 750 /etc/certs
sudo chmod -R 700 /etc/certs/demoCA

Now you can link the server.pem file into where Cyrus POP and IMAP want to find it.
ln -s /etc/certs/server.pem /var/imap/server.pem
Now go into Server Admin and set up POP/IMAP to use SSL in the Advanced button of the Mail Server settings.
Set your mail client accordingly and securely read your mail.
The SSL configuration for Cyrus is stored in /etc/imap.conf.
7. Enable your clients
Since your CA is self-signed all of your Mac OS X applications and services will yell at you for using it. You can get around this by adding the cert to the client’s x509 Anchors keychain. Essentially this is the root CA file for your machine.
Do this by copying over to the client machine the ca.crt file that you created in the first step. Then install it by doing
sudo certtool i ca.crt v k=/System/Library/Keychains/x509Anchors
Your client will now trust certificates that you have signed into being with this CA. If you do this right, you’ll use the same CA for all of your servers and their services. That way you’ll only have to import one file into the clients x509Anchors.
8. E-mail certs
This bit is for bonus points, but all the cool kids are doing it and so should you. in 10.3 allows the use of s/mime certificates. These are PKI certificates that act similar to SSL certificates and can sign and or encrypt e-mail.
The easiest way for a personal user to get a certificate is to head over to and sign up for their free community mail certificate. Really good instructions for this can be found here:
However, if for some reason you feel like making your own, read on. Note that this is mostly an exercise in what you can do with OpenSSL. Since the Thawte certificates are free and easily available you’re probably better off using them. However, if you want to outfit your entire organization with home rolled certificates, well here you go. Just be careful to only use this between users that have imported your root CA that you created.
To do this you need to first generate a certificate for your e-mail user. This is pretty much the same thing as generating one for a server.
openssl genrsa -des3 -out mail.key 1024
Give it a pass phrase to lock it up.
openssl req -new -key mail.key -out mail.csr
Here, you’ll want to use your real name for the Common Name. Joel Rennich is what I would use. Then make sure that you fill out the e-mail field with what you have set up in as your e-mail address. Capitalization is important here. I would use “[email protected]”.
Now sign this cert with your CA.
openssl ca -policy policy_anything -in mail.csr -out mail.crt
You’ll enter in your CA password and then commit the signature.
Finally you can convert the signed certificate into the format that is used for s/mime. When you do this it will first ask you for your mail certificate password that you set up a few commands before. Then it will ask you for an export password. This can be the same of different, it doesn’t matter, but you will need to use the export password when importing this certificate into your Keychain so you can use it with mail.
openssl pkcs12 -export -inkey mail.key -certfile mail.crt -in mail.crt -out mactroll.p12
This is your “official” e-mail certificate. Copy this over to your client machine and double-click. Keychain Access should launch and ask you for your export password. The certificate will then be imported into your keychain and immediately usable by for the account that you specified in the e-mail field when you generated it.
9. Other odds and ends
When you sign your certificates with your CA openssl uses a default config file which can be found at /System/Library/OpenSSL/openssl.cnf. If you want to change any of the defaults go here. For example, certificates that you sign will only be valid for 1 year, unless you edit this file to change that.

No Comments

  • Excellent article Mr. Troll. Just 2 questions:

    1. If I set my SMTP to "Require SSL", mail is bouncing that originates from the
    server because it does an SSL connection. Any suggestions?

    2. Some of the people in my office use Windows. Is there anyway that you
    know of to import the ca.crt file into a Windows box?

  • 1. Sure, most servers on the internet do not have an SSL certificate set up, so
    by setting this to require you’ll be missing out on most of your incoming

    2. I believe if you make a web connection to your server using your self-
    signed CA, IE on the PC will give you the option to install the CA cert. I’m sure
    google will be able to tell you more.

    Changing the world, one server at a time.

    Joel Rennich

    • In a somewhat related topic, in dealing with a server that has yet to be updated to 10.4, I am running into an issue where the server is convinced that SSL is required for all traffic if SSL is specified in any way in the "Advanced" settings.

      I had attempted to verify that and had references to tls commented out, but only futzing with Server Admin and re-setting the SSL settings to "Don’t Use" (even though that was the specified setting) and then saving was able to end the dreaded list of "tls_engine not running" errors that would fill the mail.log when the SSL cert symlink was not in place. With the link in place, the "SSL_accept errors" would fill the log, seeming to indicate that the server was requiring SSL.

      For now, I have no SSL, and I’m back to dealing with users that can’t send from specific locations when port 25 is blocked, even though all users are using Password or MD5 authentication.



  • In step 4 "Securing LDAP" the article states, " LDAP doesn’t like a server key
    that has a passpharse associated with it. Postfix and Cyrus are going to be
    the same way. So remove the passphrase."

    The command given to do so is:

    openssl rsa -in server.key -out serverno.key

    But when I enter this, I am prompted for a passphrase, and when I try
    entering null (just returning the blank line) I am told:

    "You must type in 4 to 1023 characters"

    So how do I remove the passphrase? I want to add SSL to my POP and SMTP
    services. I have already successfully add SSL for two different web sites
    running on the same server. I am using a purchased cert from

    • You need to enter the passphrase to decode the version of the key that
      has a passphrase, so that you can then write out a copy of it that does
      not have a passphrase.

  • This works so great for us in that we decided to do it for a client.
    Unfortunately they’re running Entourage, which apparently doesn’t respect
    the x509Anchors like does. Office 2004 has its own certificate
    manager (, but not so Office X, which is
    what the client has. Any ideas on how to get Entourage X to acknowledge the
    newly imported ca.crt?


  • I followed your article exactly and had no issues with the setup, until I got to
    restarting my web service. I get the following error:

    [Thu Mar 10 23:13:48 2005] [error] mod_ssl: Init: Pass phrase incorrect
    (OpenSSL library error follows)
    [Thu Mar 10 23:13:48 2005] [error] OpenSSL: error:0D07207B:asn1 encoding
    routines:ASN1_get_object:header too long

    I’m not sure what I’ve done wrong. Any ideas anyone?

    Also, there is only one section of the web site I want to be secured. How
    would I go about setting up Server Admin to reflect this one section?


    • Elliot,

      I got the same error Steve reported: mod_ssl: Init: Pass phrase
      . I was unable to resolve it by disabling performance caching,
      I think because it’s not the same error you saw (not intermittent.)

      I resolved my issue by reducing the length of my passphrase.

Leave a reply

You must be logged in to post a comment.