Articles July 25, 2006 at 8:11 am

Ethernet Herding

Port Trunking, Trunked Ethernet and Bonded Interfaces are some of the terms that get thrown around for what is more commonly known as 802.3ad. In both Apple’s documentation and more generally accepted is the term Link Aggregation, or Link Aggregate Group (LAG) which is perhaps a more visual description of what is really going on here – an aggregate of network interfaces which allow you to beef up the total bandwidth to your server with little to no further cable or networking hardware changes.

Read on for more…
First question is, why should you use link aggregation? When you combine multiple physical links from your server to a switch you’re creating a fault-tolerant link, with a bandwidth equal to the the sum of the bandwidths of the physical links. In other words, if you have Link Aggregation on your server and someone disconnects one of the ports on your switch, your server is still accessible, although with only half the bandwidth. It’s also a nice way of extending the life of some older hardware; if you happen to be running a tower as your server, and have some 100Mb PCI ethernet cards lying around, you can combine these into a link aggregate to get greater bandwidth than a single 100Mb link.

With that said, if this is your first time getting into Link Aggregation you’re probably going to need to get a new switch that understands how to pass the resulting traffic. Several of the higher end switches have supported this function for a while (i.e. the Ciscos and Foundrys of the world) and now we’re starting to see some more affordable options out there for smaller businesses (i.e. NetGear). What you’re going to be looking for in a switch is the designation that it’s 802.3ad or LACP (Link Aggregate Control Protocol) compatible. Do be aware that there are some proprietary formats out there which to accomplish the same result, but typically only work or interconnecting equipment from the same manufacturer; Cisco’s EtherChannel is an example of this, even though Cisco does also support 802.3ad in their product line.

A fundamental concept to remember here is that the Link Aggregation you’re creating in OS X Server is software based. What this means is your two ethernet ports appear to the rest of the network as a single IP. This is particularly important to remember when implementing the switch immediately connected to your server – with the OS X Server appearing as a single IP any traffic destined for it can travel over either port on the switch. More likely than not, your 802.3ad compatible switch has a setup screen for something called “Trunking” do not be seduced by this setup screen – it’s not required here, in fact no manual configuration of your switch is required for OS X Server’s implementation of Link Aggregation. This trunking panel would be used if you were bonding your interfaces at the switch level, rather than software based, and currently OS X Server does not support this.

Not all ports are created equal either, here are screen shots from a switch that has had an OS X Server with Link Aggregation running for a while:
port1 & port2
You’ll notice that there’s a difference of nearly 45 million packets between the two ports. The number of requests to and from the server don’t fill one pipe completely before sending traffic to the second pipe, rather they are dynamically distributed across the ports, so administration of what data actually flows across a given port is taken care of automatically within the aggregated link. Also, the correct order of your packets, or frames, is only guaranteed if a conversation between your server and client is maintained across a single link in the trunk. Although this is not as efficient link utilization as simply shipping each frame over any available connection, it avoids the extra logic required to monitor frame ordering and to reassemble them before delivery to the recipient – simply put this is a unicast, not multicast protocol. At the same time, additional transactions by other clients benefit from the availability of the aggregated link, and some bottlenecks can be avoided.

Setting up your link for the first time is relatively straightforward. Your starting point will be from a server that is already up and running, with an existing ethernet interface that you’ve tested the communication of on your 802.3ad compatible switch. For this example we’re going to be replacing that interface with a link aggregated interface.

Connect your second ethernet cable to the switch first, then within your Network System Preference, you’re going to select “Network Port Configurations” and create a new configuration. From the “Port” drop-down menu, “Link Aggregate” will be available to you – you’ll be required to name this new port, and select the ports to aggregate. You’ll notice at this time you are only provided with Ethernet interfaces to chose from, as the Link Aggregation protocol does only support IEEE 802.3.

Once selected, and you proceed you will be prompted with a warning that your existing configurations will be lost. That includes the valid link you had already setup. Next step is configuring your new Link Aggregate, which should be exactly what your single link ethernet settings were. If you are currently using VNC or Apple Remote Desktop to control your server over that ethernet link you will lose communication with your server momentarily once you hit apply, but it should return. If you’re at all concerned, IP over firewire is very handy at times like these.

You’ve probably noticed an additional tab has been enabled called Status. As your link is software based, the only place to check if your link is up and configured correctly is going to be on your server, and this status pane is going to give you those details. Similar to the lights in Server Admin, green indicates everything is running smoothly, amber for warning, and red for incorrect configurations. A healthy Link Aggregate is shown here:

When there is an issue with the link, some explanation is given as to what could be causing the issue when you select the affected row. This invalid link screen shot shows a server this is still available to it’s clients as the first port is still up, however, the second port has an amber indicator that could have been caused by the ethernet cable being unplugged, or a configuration has been made on the switch that has affected the link.

In troubleshooting this particular scenario, here are a couple things to keep in mind:
– Are you trying to use dissimilar data rates in your link? All of your ports must be using the same data rate, i.e. all 100Mb/s or 1000Mb/s. Check your ethernet cables also if you know you have similar ports, but are still getting different data rates – an older or degraded cable may not be capable of passing the same data rate.
– Are you forcing Half duplex at the switch level? Link Aggregation is only supported in Full Duplex mode.

One of the most important things to take away from this is that you are not creating one huge pipe here as there is the limitation that a conversation between a client and server must be maintained across a single link. Overall though, Ethernet Link Aggregation is quite a useful ability for your server to have, and has many viable uses such as having multiple client machines who require simultaneous high bandwidth from your server, or if you are using your OS X server as a backup server thus enabling several machines to push their data simultaneously, or simply for redundancy in case one of your cables or ethernet ports should fail.

About

Andrina Kelly is responsible for anything and everything touched by, or connected to, a Mac at Bell Media, Canada's premiere multimedia company. You may recognize her name from the end credits of Canada's evening news broadcast. She has previously spoken at MacSysAdmin, JAMF National Users Conference, Apple's WWDC, Macworld IT conferences, Mac Networkers Retreat, and Canada MacExpo.

No Comments

  • Informative! Thanks for posting this.

    • Works nicely on my G4 XServe’s, although not ’til somewhere around the 10.4.4 update. I actually configured LACP on my Cisco’s…not sure if it’s overkill though.

  • We’ve been using this with a G5 Xserve 2.0dp and an HP ProCurve Switch
    since February with zero problems.


    Matthew Kosterman • Chief Executive Geek • DeltaQuest Imaging, Inc.

  • Can anyone outline the pros & cons of doing it this way, versus configuring at
    the switch or both?

    Cheers
    Dan

  • Any switch that has LACP enabled.

    The whole point of LACP is that you don’t need to manually set it up. I believe
    that some switches will ship with this enabled by default, so in that case you
    wouldn’t have to touch anything. Of course some network admins will have
    turned it off, even if it does ship enabled by default, under the idea that if you
    don’t need it don’t enable it.


    Changing the world, one server at a time.

    Joel Rennich
    http://www.afp548.com

  • Nope my network boys had to manually enter the pair’s into our cisco switches.
    I don’t begin to know/care how they have their switches configured as long as it
    works when I ask for something.

    I just had to specifically tell them that I was not interested in EtherChannel, but
    802.3ad and voila 5 minutes later I was golden.

    peet

  • This is the sequence I used to get LACP working with a Cisco 2950. I imagine
    that commands for different Cisco boxes will vary slightly.
    1. Log into switch and get to enable mode
    conf t
    #puts you in global configuration command#
    interface Gigabit 0/1
    #puts you in interface config mode for port 25#
    channel-group 1 mode active
    #You are doing two things here. One creating a channel group to put ports in
    and
    giving it a number. The mode(active) relates to which of the 5 modes you
    want the channel
    to be in. "active" mode puts the interface into LACP( not PaGP since I doubt
    we
    support that) only. You could also use "passive" mode but then the channel
    will
    only enable LACP if it senses another LACP device on the other end of the
    wire. I
    went the "active" route just to be on the safe side. #
    interface Gigabit 0/2
    # in the previous steps you created a channel and add a single Gig port. Now
    we
    are going to go back and add the second Gig port to the same logical
    channel#
    interface Gigabit 0/2
    #puts you interface config mode for port 26#channel-group 1 mode active
    #This puts the other Gig port in the same channel
    SHORTCUT!!
    You could also substitute this command for the two individual interface
    commands#
    interface range Gigabitethernet 0/1 – 2
    (config-if-range)
    # now when you execute the channel-group commands you get both ports at
    the same time
    END OF SHORTCUT!!#
    end
    #ends the config session#
    sh run
    #shows the running config. If you are successful you will "interface Port-
    channel
    1" at the top of the interface section of the config file. Also you will see the
    Gigabit ports have the entry " channel- group mode 1 mode active". Kind of a
    double check.
    You can also use this command to check the status of all your EtherChannel
    groups#
    sh etherchannel
    Channel Group listing:
    Group: 1
    Group state – L2
    Ports:2 MaxPorts=16
    Port-channels: 1 Max Port-channels=16
    Protocol: LACP
    #Once this is complete the trunk will be up and operating in a default "src-
    mac"
    forwarding mode. This means that packets from different hosts will actually
    forward along different links in the trunk while packets from the same host will
    use the same port in the channel. The other mode is "dst-mac" forwarding
    which
    will send packets to the same destination on the same port but packets to
    different destinations are sent on different ports in the channel.
    Optional config – You can change this behavior by executing the following:#
    conf t
    #back to config mode#
    port-channel load-balance dst-mac
    # or vice versa#

  • Are the two interfaces you’re trying to use the same speed – i.e. are they both
    running at either 100MB/s or GigE? The other thing to check would be if your
    switch is set up with the ports not both set to autonegotiate, or two similar
    settings – i.e. is one set to full duplex, while the other is set to half?

  • Yep – I get the same on my Xserve’s logs:

    kernel[0]: bond0 duplicate IP address 192.168.1.2 sent from address 00:0d:
    93:9b:f8:8c

    However, the link aggregate appears to be otherwise functioning perfectly
    (status OK and full duplex on both ethernet cards) so I’ve been ignoring it.
    Anyone see a reason for concern?

    Cheers

  • No manual trunking configuration. It’s all done with LACP automatically.


    Changing the world, one server at a time.
    Joel Rennich

  • The same happened to me. Curious, but when I disabled port trunking in the switch, the link activated and everything works as it should.

Leave a reply

You must be logged in to post a comment.