Forensic Analysis of a Compromised Mac OS X (Client) Machine

—Joel Rennich,

7 May 2002

While not really “Server” related. I recently had the opportunity to cleanse a Mac OS X Client machine of a previous system administrator that had been fired. The admin knew what he was doing and had been running a number of services on this machine: ftp, ssh, www, and a mailing list, among others. The client needed the machine and was not willing to do a complete reformat and install of the system, which is the only real way that you can ensure that there isn’t anything left on the machine. As you go through these steps, be sure to take careful notes about what you see going on. Print out or save your Terminal sessions if you need evidence. Plus, these can come in handy down the road if you find yourself in this situation again.

Based on that experience, which was really quite fascinating to work through, I have come up with list of suggested best practices for this type of situation.

1. First, remove the machine from the network if possible. This will limit the amount of damage that it might be able to cause and make sure that no one else is in the box working against you.
2. Back up all important data and potential evidence from the machine. Use a FireWire hard drive or some other storage device to back up the data. If you really need evidence, duplicate the drive using psync or some other utility. Otherwise I would suggest copying everything in /var/log/, /var/db/ and the /etc/ directories. If this is a Mac OS X Server also grab /Library/Log/. Plus, take whatever is left of users' home folders, or at least your prime suspects' home folders. Grab more if you have the space, but this should hit the important bits. The logs are going to be invaluable if you need to recreate the crime. The wtmp log, for example, can tell when the machine was turned off and on and when each user last logged in and from where. You can also get a large amount of information from the system log (/var/log/system.log). Additionally, don’t forget such simple things as the malicious user’s Internet cache. Bits and pieces like this can really help you figure out what you are up against.
3. If you can, format the drive and reinstall the system. This is the only way that you can be sure that everything has been expunged. Sometimes you won’t be able to do this, so follow through with the rest of the steps.
4. Run the Process Viewer, or ps aux (as root!) from the CLI. Take note of all of the processes that are running. Make sure that only the ones that you need are there. Compare this against a known good machine under the same load to see if there are any discrepancies. Any process that you are not familiar with try looking them up on the web. Once you find the ones that you don’t need, or are certainly malicious, go ahead and kill these off. Make sure that they stay dead. On Mac OS X Server they may be respawned by the watchdog process.
5. Now check your basic config files. Check /etc/inetd.conf for any strange services that might be uncommented. Check /etc/httpd/httpd.conf against a known good or the default that is supplied with the system. Make sure that you check the modification dates on these files. While this isn’t foolproof, it is a good way to check if they have been tampered with. Essentially you want to take a look at most any file in the /etc/ directory that ends in .conf. Certainly check the modification date, but you should also take a glance at each one to make sure it is good.
6. Now port scan your machine, using nmap or another utility, or do a netstat -a from the CLI. These will both tell you what ports the machine is listening on. Connect to all of these ports using telnet, or better yet netcat. Just because a service is running on port 110 doesn’t mean that it is a POP3 server. Be very skeptical. Make sure that you need every port that you have open. Each open port is a potential way for the black hats to get back in.
7. Check all, and I mean all, of your crontabs. Either through Terminal or a GUI app like CronniX, an essential admin tool by the way. Make sure that there are no hidden scripts to reinstall a user sometime next week. Make sure you have a reason for every event in the crontabs. Mac OS X only ships with three cron scripts, a daily, a weekly and a monthly shell script. Any others have been added. For any shell scripts that are in the tab make sure they are doing what you want them to do. Inspect each one against a known good to spot any changes.
8. Check /Library/StartupItems/ for anything strange. Just like the crontab, go through each of these files to make sure it is starting what you think it is starting. Also check /System/Library/StartupItems/ for the same things. Reboot the machine in verbose mode to look for anomalies and to check for any startup items you may have missed.
9. Check the process list again to make sure that nothing evil is running.
10. Check your open ports again to make sure nothing evil is listening.
11. Install a software firewall. Use Brickhouse or any of the other great ipfw GUIs, or heck, just edit the ipfw list yourself. Block everything but what you need. This way even if another service starts up the firewall will still block it. You should also put this machine behind a hardware firewall, if possible to further secure it.
12. Use the NetInfo Manager to ensure that only the users that you need are on the machine. Any system-installed user should have “*” for a password. This will prevent it from being used to login to the machine. Change every user password, each and every one. Also disable root for the time being. You can use sudo for the next week or two.
13. Change the IP address of the machine. While not a huge hurdle to overcome, this will at least make life a little harder for the bad guys that might want to get back into the machine.
14. Do a reinstall of the system if possible. You don’t necessarily need to format the drive, but at least reinstall the system. It is common in UNIX attacks to replace such system binaries as ls and top with compromised versions that enable or hide backdoors into your system. The only way you can be absolutely sure that these don’t exist is to reinstall the system.
15. Cross your fingers and keep checking the logs. You need to keep your shields up for at least a week or two.

Now having said this, if I wanted to leave a backdoor in a system I would write a small shell script that would create a new admin user on the system. Or to be real sneaky I would reset the password on an existing system user like “daemon” or “www” being sure to give them an appropriate home folder like /etc/ and a usable shell. This script would be called from a single line within the monthly script at /etc/monthly. The script itself would be called something innocuous like accounting_conf and be stashed away in the /etc/ folder. In addition to creating a user, the script would flush the ipfw rules and enable ssh, but on a different port, like 53. Now at the end of the month I know I will have a new user with admin rights waiting for me at the other end of a non-standard ssh connection. Not too bad for 10 minutes of scripting.

The above guidelines would allow you to catch something like this, but you would have to be looking very closely for it.

Remember, we aren’t dealing with a rogue extension that makes the cursor go all funny, anymore. This is a whole new ballgame.