Port Trunking, Trunked Ethernet and Bonded Interfaces are some of the terms that get thrown around for what is more commonly known as 802.3ad. In both Apple’s documentation and more generally accepted is the term Link Aggregation, or Link Aggregate Group (LAG) which is perhaps a more visual description of what is really going on here – an aggregate of network interfaces which allow you to beef up the total bandwidth to your server with little to no further cable or networking hardware changes.
Read on for more…
First question is, why should you use link aggregation? When you combine multiple physical links from your server to a switch you’re creating a fault-tolerant link, with a bandwidth equal to the the sum of the bandwidths of the physical links. In other words, if you have Link Aggregation on your server and someone disconnects one of the ports on your switch, your server is still accessible, although with only half the bandwidth. It’s also a nice way of extending the life of some older hardware; if you happen to be running a tower as your server, and have some 100Mb PCI ethernet cards lying around, you can combine these into a link aggregate to get greater bandwidth than a single 100Mb link.
With that said, if this is your first time getting into Link Aggregation you’re probably going to need to get a new switch that understands how to pass the resulting traffic. Several of the higher end switches have supported this function for a while (i.e. the Ciscos and Foundrys of the world) and now we’re starting to see some more affordable options out there for smaller businesses (i.e. NetGear). What you’re going to be looking for in a switch is the designation that it’s 802.3ad or LACP (Link Aggregate Control Protocol) compatible. Do be aware that there are some proprietary formats out there which to accomplish the same result, but typically only work or interconnecting equipment from the same manufacturer; Cisco’s EtherChannel is an example of this, even though Cisco does also support 802.3ad in their product line.
A fundamental concept to remember here is that the Link Aggregation you’re creating in OS X Server is software based. What this means is your two ethernet ports appear to the rest of the network as a single IP. This is particularly important to remember when implementing the switch immediately connected to your server – with the OS X Server appearing as a single IP any traffic destined for it can travel over either port on the switch. More likely than not, your 802.3ad compatible switch has a setup screen for something called “Trunking” do not be seduced by this setup screen – it’s not required here, in fact no manual configuration of your switch is required for OS X Server’s implementation of Link Aggregation. This trunking panel would be used if you were bonding your interfaces at the switch level, rather than software based, and currently OS X Server does not support this.
Not all ports are created equal either, here are screen shots from a switch that has had an OS X Server with Link Aggregation running for a while:
port1 & port2
You’ll notice that there’s a difference of nearly 45 million packets between the two ports. The number of requests to and from the server don’t fill one pipe completely before sending traffic to the second pipe, rather they are dynamically distributed across the ports, so administration of what data actually flows across a given port is taken care of automatically within the aggregated link. Also, the correct order of your packets, or frames, is only guaranteed if a conversation between your server and client is maintained across a single link in the trunk. Although this is not as efficient link utilization as simply shipping each frame over any available connection, it avoids the extra logic required to monitor frame ordering and to reassemble them before delivery to the recipient – simply put this is a unicast, not multicast protocol. At the same time, additional transactions by other clients benefit from the availability of the aggregated link, and some bottlenecks can be avoided.
Setting up your link for the first time is relatively straightforward. Your starting point will be from a server that is already up and running, with an existing ethernet interface that you’ve tested the communication of on your 802.3ad compatible switch. For this example we’re going to be replacing that interface with a link aggregated interface.
Connect your second ethernet cable to the switch first, then within your Network System Preference, you’re going to select “Network Port Configurations” and create a new configuration. From the “Port” drop-down menu, “Link Aggregate” will be available to you – you’ll be required to name this new port, and select the ports to aggregate. You’ll notice at this time you are only provided with Ethernet interfaces to chose from, as the Link Aggregation protocol does only support IEEE 802.3.
Once selected, and you proceed you will be prompted with a warning that your existing configurations will be lost. That includes the valid link you had already setup. Next step is configuring your new Link Aggregate, which should be exactly what your single link ethernet settings were. If you are currently using VNC or Apple Remote Desktop to control your server over that ethernet link you will lose communication with your server momentarily once you hit apply, but it should return. If you’re at all concerned, IP over firewire is very handy at times like these.
You’ve probably noticed an additional tab has been enabled called Status. As your link is software based, the only place to check if your link is up and configured correctly is going to be on your server, and this status pane is going to give you those details. Similar to the lights in Server Admin, green indicates everything is running smoothly, amber for warning, and red for incorrect configurations. A healthy Link Aggregate is shown here:
When there is an issue with the link, some explanation is given as to what could be causing the issue when you select the affected row. This invalid link screen shot shows a server this is still available to it’s clients as the first port is still up, however, the second port has an amber indicator that could have been caused by the ethernet cable being unplugged, or a configuration has been made on the switch that has affected the link.
In troubleshooting this particular scenario, here are a couple things to keep in mind:
– Are you trying to use dissimilar data rates in your link? All of your ports must be using the same data rate, i.e. all 100Mb/s or 1000Mb/s. Check your ethernet cables also if you know you have similar ports, but are still getting different data rates – an older or degraded cable may not be capable of passing the same data rate.
– Are you forcing Half duplex at the switch level? Link Aggregation is only supported in Full Duplex mode.
One of the most important things to take away from this is that you are not creating one huge pipe here as there is the limitation that a conversation between a client and server must be maintained across a single link. Overall though, Ethernet Link Aggregation is quite a useful ability for your server to have, and has many viable uses such as having multiple client machines who require simultaneous high bandwidth from your server, or if you are using your OS X server as a backup server thus enabling several machines to push their data simultaneously, or simply for redundancy in case one of your cables or ethernet ports should fail.