On Mac OS X 10.5, if you connect two different ports on a host to two different networks and assign them with two different IP schemes then only one can act as the default gateway. This means that any incoming packets will all, by default, get responded to over the default gateway; despite which interface the packets came in on. You can customize your routing table using the route command but this can be tedious and doesn't always necessarily net the desired results.
Read on for a bit more…
To show this, run the command netstat -rn on a Mac OS X 10.5 host with two NICs installed, each with its own gateway, and you'll noticethat only one entry is listed as the default.
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 192.168.210.1 UGSc 11 039 en0
In Snow Leopard we now have Source Based Routing, or Source Routing, according to how you feel like referencing it and the type of traffic that you are referencing. Connect two NICs, again, each with its own gateway, and you'll notice that both can have an entry as the default when you run that same netstat -rn command:
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 192.168.210.1 UGSc 58 0 en0
default 192.168.55.1 UGScI 0 0 en1
This is because the source of a packet has specified the route that the packet will use, basically meaning that they're going to dance with the one that brought them and the response will be on the same network that the packet came in on. The long and short of it means that if you take an Xserve, connect one port the the outside of your network, or a second network and the second port directly to your production network and it will work fairly easily, without a lot of command line jockeying. Best of all, it's the default behavior, so you won't have to do a thing to use it.
by 
Setups like this typically won’t do any load balancing, but they will respond to incoming connections on the interface that they came in on, although I’ve not had enough time to play on Snow to really say what’s going on here.
If it is truly source-based routing then having two DSL lines would work, but similar to 802.3ad, you’d only be using one pipe for any one connection. And unlike 802.3ad you would have absolutely no fail-over functionality.
—
Changing the world, one server at a time.
Joel Rennich
Although it would be trivial to put a polling process on one of the network interfaces and in the event that the main interface goes down it would activate the secondary interface. I’d still use an appliance for that but possible nonetheless.
—
Charles Edge
Author :: Mac Tiger Server Little Black Book
Author :: Foundations of Mac OS X Leopard Security
Partner :: 318 Consulting
http://www.318.com
c
One reason I might use it would be to put a mailserver on two different links- that way if the primary fails the backup MX record can just point to the second connection and the server will continue to receive email properly (and if you had a third port to connect to the LAN network in some fashion the office wouldn’t really know-only remote user would notice). Nothing huge, but it would be convenient.