Flying Racoons: Host to host, coast to coast 
9 September 2002
Part Two: Creating encrypted communications between two hosts
Armed with our budding knowledge of IPSec, we can now create a secure connection across a network between two Mac OS X 10.2 machines. In this part of the series we will only be creating a transport setup. We will not be tunneling packets from one network to another, but just from one machine to another.
IPSec on Mac OS X uses two facilities to create a connection like this. The first is setkey. This utility keeps track of what communications should be encrypted. setkey doesn’t create the connections, though; that is the job of racoon, the key exchange agent. We could spend all day and thousands of pages on the exchange of IPSec keys, but we’ll gloss over a lot of the particulars in the interests of keeping you awake. Essentially: racoon conducts the IKE, or Internet Key Exchange, part of IPSec. IKE allows for the two parties to securely agree upon an IPSec encryption configuration and then to alter the passwords, or keys, that keep that configuration secure.
IPSec, by its very nature, tends to error on the side of security and as such many default installs will change keys every few minutes (if not seconds). All of this is done behind the scenes by racoon, so you don’t need to worry about it. The end result is that an attacker has to start from scratch every time the keys are changed when trying to decrypt your conversation.
Now that you know where we got the name for this series, forget about racoon for a bit. The default configuration works great between two Mac OS X machines, so we won’t worry about it for now.
For our example here we are going to be securing the communications between two machines on my wireless network. Since even 128-bit encyption with WEP is not very secure, we are going to wrap everything in IPSec. Overkill, sure, but Mac OS X 10.2 makes it so easy.
My laptop is on a fixed IP of 10.0.1.3 and my server is at 10.0.1.27, so swap them with your IPs where appropriate. We’re using NAT’d numbers, but you can use static IPs if you have them. Just make sure that your routers support IPSec passthrough.
Before we do anything we want to make sure that we don’t have any IPSec policies in place already. Drop down to the command line and execute these commands:
sudo setkey -DP sudo setkey -P
If both came back with nothing, then all’s good. Otherwise, you already have some IPSec policies set up on your machine. It’s perfectly fine to add new ones in addition to the ones you already have as long as they don’t conflict.
Now we can actually create our policies. The setkey command works a little differently than most others: instead of taking arguments on the command line, it wants to take its input from a text file. We’ll have some fun with input redirection to get around this.
We’ll start with creating the policies on the laptop. So back to the command line and type the following:
sudo setkey -c << EOF ? spdadd 10.0.1.3/32 10.0.1.27/32 any -P out ipsec esp/transport/10.0.1.3-10.0.1.27/require; ? spdadd 10.0.1.27/32 10.0.1.27/32 any -P in ipsec esp/transport/10.0.1.27-10.0.1.3/require; ? EOF
The first command allows you to feed subsequent commands to the setkey command as if it was a single text file. The next line sets up a policy to encrypt all communications from the laptop to the server. The require at the end of the line means that non-encrypted communication will not be allowed. The line after this is essentially the reverse. It forces all incoming communication from the server to be encrypted.
If you messed this up you can flush out these polices by using:
sudo setkey -DF sudo setkey -F
So far, so good. Now we need to create a shared secret between the two machines. This is a password that sets up the initial exchange of encryption keys between the two machines. After the initial communication, racoon will take over and create random keys that will make the conversation more secure.
You set the initial shared secret in a text file: /etc/racoon/psk.txt. You’ll need to be super-user to change this file, so use your favorite command line editor or chmod the file and use TextEdit. You want to enter an entry for the server:
10.0.1.27 supersecretpassword
Now on to the server. We’ll need to do essentially the same procedure to get this to work. First use setkey to create the policy:
sudo setkey -c << EOF ? spdadd 10.0.1.27/32 10.0.1.3/32 any -P out ipsec esp/transport/10.0.1.27-10.0.1.3/require; ? spdadd 10.0.1.3/32 10.0.1.27/32 any -P in ipsec esp/transport/10.0.1.3-10.0.1.27/require; ? EOF
And then an entry in /etc/racoon/psk.txt. The shared secret password is the same between the two machines.
10.0.1.3 supersecretpassword
Now start up racoon. You’ll need to do this on both machines.
sudo racoon -f /etc/racoon/racoon.conf
That should be it. The only difference you should notice when using the system is that the initial connection will be a bit slower as the racoons talk. After that you will be able to connect between the two machines exactly the same way as you did before IPSec was thrown into the mix. It is so seamless that you may even doubt that your connections are being encrypted. To set your mind at ease you can either use tcpdump to check the packets or use netstat -s (which will show you a list of packet types through your interface). If things are working, you will see how many IPSec packets that you have transmitted and received.