Migrating from UW IMAP to CYRUS
6 November 2003
With Mac OS X Server 10.3, Apple finally jettisoned its limited “Apple Mail Service” and adopted two excellent Open Source alternatives: Postfix for the mail transport, and CYRUS IMAP for the mail storage/database. In this article, I will demonstrate how you can migrate from a UNIX-style 'mbox' mail storage/database setup to the Apple-supplied CYRUS solution. Unfortunately, Apple chose not to include any utility for migrating from any source other than AMS (and I don’t particularly blame them), so we have to roll our own.
CAVEAT: As with any mission-critical operation, you should backup your mail prior to following the steps in this article. We explicitly disclaim any responsibility for the loss of e-mail—the steps outlined in this article have only been tested on our existing test and production servers, and your setup may be wildly different.
Our setup is currently configured using Postfix 2.0.9 and UW IMAP 2002e. About two dozen mail accounts currently exist on the server; all but one or two accounts access their e-mail through POP or simple IMAP, with no locally stored messages beyond the INBOX. This will make the transition somewhat simpler when it comes time to tell the users what to do.
Mac OS X Server 10.3 ships with Postfix 2.0.10 and CYRUS 2.1.13—in both cases behind the current releases, but not dangerously so. CYRUS IMAP stores mail with each message in a separate file, organized and indexed using a special algorithm. Because CYRUS IMAP doesn’t assume that each mail user is tied to a specific system user, it means that you could literally have joe@domain1 and joe@domain2 and not require them to have different—or any—entry in Workgroup Manager. Unfortunately, this format is quite unlike the one used by UW IMAP, so converting between servers will be somewhate tricky.
The ideal solution would be to have two Servers, one of which will run Mac OS X Server 10.3 while the other can still run 10.2...but that sort of setup is unlikely to be common, even in large enterprises. Instead, what we will do is temporarily run two IMAP servers on the same machine.
At this point, I assume you have already upgraded your Mac OS X Server from10.2 to 10.3, and your users are diligently “on hold” for their e-mail.
Part One: Building a New IMAP
The first thing we’re going to do is set up a new UW IMAP server, and run it off an unused port. To do this, grab a copy of the current UW IMAP software—you can use the IMAP-2002e distribution from ftp.cac.washington.edu, or just grab the Pine 4.58 package and use that version. Whichever you choose, download it to a local directory in a path without spaces and decompress it. Then change directories into the 'imap' directory.
Mac OS X Server 10.3 provides plugin authentication module (PAM) support. This is great, because it makes authentication much easier to perform. Unfortunately, Apple doesn’t include the header files in /usr/include/security, but rather in /usr/include/pam. To get around this unfortunate situation, we can either create a symlink or edit a single file in the source. To create the symlink, change to /usr/include and execute the command
ln -s /usr/include/pam security
Executing a long list (ls -alFG) will show:
lrwxr-xr-x 1 root wheel 16 5 Nov 16:08 security@ -> /usr/include/pam
(obviously, the date and time will be different for your system). Alternately, we can edit the source of imap/src/osdep/unix/ckp_pam.c and change the line
#include <security/pam_appl.h>
to
#include </usr/include/pam/pam_appl.h>
Either method will work, so do whichever makes you most comfortable.
Next, we need to tell PAM about IMAP. This is a change that you will probably want to undo once your conversion process is complete, just for security’s sake. Change to the /etc/pam.d directory and assume root permissions with sudo bash. PAM is similar to xinetd in that services have individual control files that manage how that service interacts with authentication. In this case, we need to add a control file for IMAP. Luckily for us, IMAP’s needs are the same as FTP's, so we can just duplicate the existing ftp file and call it imap.
Now we need to compile the software. Switch back to your source directory. Although we’re not planning to replace CYRUS with UW IMAP, we’re still going to exploit Mac OS X Server 10.3’s support for SSL. Execute the following command:
make osx PASSWDTYPE=pam SSLTYPE=unix \ SSLDIR=/usr SSLCERTS=/System/Library/OpenSSL/certs
You will be presented with a warning message and prompt:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + Building in PARTIAL compliance with IESG security requirements: + Compliant: ++ TLS/SSL encryption is supported + Non-compliant: ++ Unencrypted plaintext passwords are permitted + + In order to rectify this problem, you MUST build with: ++ SSLTYPE=unix.nopwd +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Do you want to continue this build anyway? Type y or n please:
Assuming you were previously using 'plain-jane' UW IMAP, go ahead and respond 'y' to this prompt. Remember, we’re not planning to keep UW IMAP around. If you do want to replace CYRUS with UW IMAP and lose access to the Mail GUI in Server Admin, support for Mac OS X Server 10.3 is being added to the latest build of UW IMAP. See this thread on the Apple Support Discussion board, and the UW IMAP Web site, for more information.
Take the binary that was built and put it someplace 'natural,' like /usr/local/bin:
sudo mkdir -p /usr/local/bin sudo mv imapd/imapd /usr/local/bin
Once the build process is complete, you will have a functioning UW IMAP server in the imapd directory. Go ahead and test it by typing 'imapd/imapd'—you should see a respons similar to below:
* OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS AUTH=LOGIN] testserve.afp548.com IMAP4rev1 2003.339 at Thu, 6 Nov 2003 12:45:03 -0500 (EST)
Respond with '001 logout' to quit the server.
Now we want to tell the Server to accept requests on an unused port and route them to the UW IMAP server, instead of CYRUS (which lives at port 143). We need to run the servers in parallel, because it’s the only way to transfer your mail from mbox to CYRUS format. To do this, we need to edit two files in /etc: /etc/inetd.conf and /etc/services. You will need to assume root access to edit these files.
In /etc/services, we will create a bogus service called “imap4b” and assign it to port 32767. Add the following lines to the file to do so (you don’t have to add them in numeric port order, but it won’t hurt to):
imap4b 32767/tcp imap4b 32767/upd
Remember, we’ll be removing these lines once the conversion is over—there’s no need to keep unused services lying around waiting for some unscrupulous hacker to try to compromise your system.
In /etc/inetd.conf, we’re going to uncomment the existing imap4 entry. Mac OS X Server 10.3 would really prefer that we use /etc/xinetd.d and create a service file, but why bother? This takes less time :-)
Change the existing line for imap4, and change it to:
imap4b stream tcp nowait root /usr/libexec/tcpd /usr/local/bin/imapd
Brilliant! This part of the process is complete. Restart the server to kick off the new inetd and pam hooks, and you should be able to telnet localhost 32767 and see your UW IMAP login.
Part Two: Migrating Mail
Pulling off the rest of your migration is dependent on using a good IMAP e-mail client. I am particularly fond of Mulberry, from Cyrusoft, a powerful, cross-platform, standards-compliant e-mail application. The process will also work with Apple’s Mail program, but you will have to trick Mail into having two IMAP accounts on the same server: use the hostname in one entry, and the IP address for the other.
Unfortunately, this process is going to have to be repeated for every user on your system. If your users aren’t particularly capable with e-mail, I recommend doing this for them. This would be an excellent opportunity for you to remind your users about regularly changing their passwords, since you will need to become them to complete the process.
Launch your e-mail client, and create a new IMAP account (the terminology used by your mail program may be different). Point it at your mail Server, but tell it to point to port 32767. In Mulberry, for example, this is accomplished by entering the following in the Server field:
yourserver.domain.tld:32767
In Apple Mail, you set the port used in the Advanced tab of the Accounts Preference.
Once you have created the account, log it and your 'regular' account in. You will now have a connection to the old UW IMAP mail store and new CYRUS mail store. At this point, migrating mail is as simple as replicating your folder structure and dragging messages from the UW IMAP to CYRUS servers. Apple Mail has the advantage here in that you can drag the entire folders, and Mail will migrate both messages and folder hierarchies. Rinse and repeat for every user, and you’re done.
Part Three: Begone, Old Server!
When you have migrated all of your users to CYRUS IMAP, you will want to disable UW IMAP access. Do this by un-doing the steps from Part One. Remove the imap file in /etc/pam.d. Re-comment the imap4b line in /etc/inetd.conf and remove the two port lines for imap4b in /etc/services. Delete /usr/local/bin/imap (or wherever you put it), reboot your Server to ensure everything is properly cleaned up, and you’re good to go.