Serious performance issues serving one billion images/month on OS X Server

Hi folks,

Thanks for reading this long-winded post! We’re big fans of Macs and a full 50% of our company uses them as their workstations. We wanted to try using Mac OS for serving, but we’ve hit some pretty big speed bumps.

We’re using Lighttpd 1.on Mac OS 10.5.4 Server as one of our image servers and we’ve hit several issues that just never came up when we were running Centos (Linux) and Lighttpd 1.4.18

This Mac serves up around 30 million image hits per day; the old machine was on a slower quad core machine running Centos 5.1 and had no issues; this new server is an 8-core XServe with 8 gigs of RAM.

In addition, it handles image uploads (around 150,000 per day) and some basic ImageMagick processing via minimagick and ruby on rails.

[We’re not considering Apache anymore, but just FYI, we first tried the built-in Apache installation using default settings and it crashed and burned in about 1 hour, hard locking the machine (using default settings with our traffic would result in too many apache processes spawning and the system being unable to start new processes). Turning the settings down resulted in very slow performance when servicing our volume of live users; we ended up going to Lighttpd, which we use on our Linux boxes; performance has been phenomenal at 1/10th the memory footprint of Apache]

After a day or so of running, we got this error in the lighttpd logs:
(network_writev.c.167) Too many open files

Using the same codebase, we’ve never seen this error in Linux.

And this error (the file definitely does exist and appears on refresh):
(response.c.544) file not found … or so: Too many open files /images/(some image path)

During times of heavy traffic, we’d get several of these errors per second (usually the file not found error). Users would see broken images, and specific requests to the actual image would result in a HTTP 500 error. Refreshing the page would sometimes result in the image being served depending on their luck.

Sometimes, we get batches of this error (which is not as unusual):
NOTE: a request for /images/(some image) timed out after writing 335630 bytes. We waited 360 seconds.

What we’ve tried so far:
————————
The first thing we tried was to set in our lighttpd.conf:
server.max-fds = 16384

but this raises an error when lighttpd starts on Mac OS stating that max filedescriptors is an invalid argument (it works in Centos Linux and FreeBSD). This happens even if Lighttpd is started by the root user (both sudo and sudo su).

Does anyone know of a workaround to this max-fds problem?

We’ve even gone in and mucked with the kernel settings (though we haven’t rebooted the machine yet):
sh-3.2# sysctl kern.maxfiles kern.maxfilesperproc
kern.maxfiles: 12288
kern.maxfilesperproc: 10240
sh-3.2# sysctl -w kern.maxfilesperproc=20480 kern.maxfiles=22528
kern.maxfilesperproc: 10240 -> 20480
kern.maxfiles: 12288 -> 22528
sh-3.2# sysctl kern.maxfiles kern.maxfilesperproc
kern.maxfiles: 22528
kern.maxfilesperproc: 20480

Here’s some more info:
$ ulimit -a -H
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) unlimited
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 65532
cpu time (seconds, -t) unlimited
max user processes (-u) 532
virtual memory (kbytes, -v) unlimited

Time machine: we’ve tried using Time Machine, but it stayed in the “Preparing…” state for over a week. Perhaps it can’t handle tens of millions of files (in several directories)? It’s not running right now.

Spotlight and MDS: The entire drive RAID has been excluded from Spotlight and yet the mds process still takes 900 megs of RAM and sometimes takes 100% of one core. I know we can completely rip it out, but there were concerns over Time Machine functioning without the supporting Spotlight processes.

We really want to make this Mac server work, but if it keeps up, we’ll need to throw in the towel and install Linux on the XServe 🙁 Thanks for any help or advice.