Basic Traffic Shaping with dummynet
A common question we see is how to manage the bandwidth on Mac OS X Server. Often people want this as a way to be able to offer a service, say FTP, but not have it suck their server --or worse, internet connection-- dry. For a long time now the easy answer on BSD based systems has been to use the dummynet facilities of ipfw but it was never part of Mac OS X. Starting with Tiger however, dummynet is in there and you can use it to implement rate limiting bandwidth management on your servers.Read on for more...
dummynet is a tool that has been a BSD staple for a long time now. Originally it was developed to allow for simulation of various network configurations and scenarios. With dummynet you can do fun things like simulate a ASDL line that was suffering 70% packet loss, or take a trip back in time to a 300bps modem connection. (I used a 300 baud modem from 1984 to 1992!) dummynet is still often used for tasks like this, but sysadmins are crafty people and had the idea to commandeer these network simulation tools into a way to control the packets flowing through a server. You can do this for all packets or you can be selective, guaranteeing one protocol or service receives a greater --or lesser-- share of the managed bandwidth than the others. This is called traffic shaping.
It's probably important to point out here that the traffic shaping done by dummynet is not QoS. All traffic in a dummynet flow will get its turn in the pipe even if a higher weighted traffic is backed up. As we go on you will see why this is and it makes sense in this implementation of traffic shaping.
Basic Ideas
In its operation, dummynet is a kernel level operation. It is fed packets by it's close friend ipfw, so you will need to know how to write an ipfw rule to be able to setup dummynet. When a packet enters the system it is evaluated by your ipfw ruleset. Typically on Mac OS X these rules either allow or deny the packets, but for our purposes we are going to send them into dummynet.
When we do this there are two dummynet constructs that we can send the packets to, a pipe or a queue.
How all this ties together is that you use pipes to define the hard bandwidth limits and then you can use queues to further define how traffic will share the bandwidth of that hard pipe. Any traffic that is not caught by a queue or pipe simply runs down the rest of the ipfw ruleset an is not managed.
The queues have a weight defined for them that ranges from 1 to 100 and these roughly correspond to the total bandwidth of the pipe. Remember this isn't QoS though, so a flow with a weight of 1 will still make it into the pipe if there is a 100 weight flow using it. Keep in mind that these weights are proportional as well. A pipe with three queues of 3, 2, and 1 will shape traffic in the same way as weights of 6, 4, and 2. Queues that have no traffic are not considered when shaping traffic. If you have queues with the weights of 1 and 99 then they will get 1% and 99% of the pipe respectively when they both have traffic. If the 99% queue has no traffic flowing then the 1% queue will be able to get the total bandwidth of the pipe. It is in this way that dummynet is an implementation of rate limiting.
In its most basic operation ipfw feeds packets into a dummynet pipe or queue. These packets are buffered into a bucket and then fed into the pipe depending on the settings of the pipe or queue. It is a simple and effective means of controlling traffic on your server and can be used in combination with NAT to control how your users access the world outside your network.
Let's jump right in.
Pipefitting
The most basic element of dummynet is the pipe, in fact that's all we need to start shaping the traffic on our server. First you configure the pipe with ipfw, then you direct traffic to it with an ipfw rule. When configuring the pipe there are two main options for traffic shaping, bandwidth and mask. We'll take a look at the bandwidth first and come back to the mask later.
Bandwidth is easy to set, you just create the pipe and tell it the amount of traffic you want it to carry in Kbit/s, Mbit/s, KByte/s, or MByte/s. The syntax is simple.
sudo ipfw pipe 1 config bw 30KByte/s
Now we can use this pipe to limit the total outbound FTP traffic on our server.
sudo ipfw add 1 pipe 1 src-port 20-21
Note here that I can use port ranges or anything defined in /etc/services as it's just an ipfw rule. Also take special notice of where this rule it, it's at the top. By default, Mac OS X Server does not re-inject packets into the firewall ruleset. This means that if you have another rule that catches FTP traffic before the pipe then those packets will never reach the pipe. Any traffic shaping rules that you add must be integrated into the rest of your ipfw rulesets. This doesn't mean that they need to be at the very top of the list however. They just need to come before any other rule that would snag the packet flow first.
If I wanted to completely restrict all FTP traffic to 30KB/sec I just need a second pipe and another rule.
sudo ipfw pipe 2 config bw 30KByte/s
sudo ipfw add 2 pipe 2 dst-port 20-21
We need the second pipe because dummynet pipes are half-duplex. We can use a single pipe to simulate half-duplex connections, but that's not really what we are after here. Since these are just ipfw rules we can specify bandwidth based on the source or destination of the traffic. One common use for this on a server acting as a router is to restrict bandwidth from local clients to the web. If we had a network of 192.168.100.0/24 we could restrict traffic from that network thusly.
sudo ipfw pipe 3 config bw 500Kbit/s
sudo ipfw add pipe 3 ip from 192.168.100.0/24 to any out
Now users can't hog all the outgoing bandwidth and stop more critical services, like mail, from getting traffic out to the internet. You could further refine this by defining a pipe for each department's network and have each one allotted a different amount of bandwidth. That, however, is a task best handled by a queue.
Wait your turn
You can see that pipe are easy to setup but that you could quickly go nuts trying to maintain a pipe for each flow of traffic you want to regulate. Queues are a mechanism that allows us to assign multiple packet flows to a single pipe, weighting the traffic from each flow to determine how much of the total pipe it will consume. So instead of divvying the total bandwidth up into hard pipes, we can make one or two pips that cover 95% of our total and then assign flows into those pipes. When doing traffic shaping with queues you need to make sure that the pipes are a little bit smaller than the actual link. Without this no packets will get queued up and dummynet will never get a chance to pick them based on weight.
For this example let's pretend that we are using a Mac OS X Server as a router. We've already cut of p2p clients with the firewall, but web access is needed and our beloved users have figured out how to use download accelerators to suck the web pipe dry. We need to limit http traffic and put a less severe limit on FTP so that clients can continue to quickly send jobs via FTP. Added into all of this is our internal Software Update server which needs to be able to quickly download updates from Apple, but still not saturate the pipe and the need for incoming SMTP to get to the head of the line. Here we go...
First we need our incoming pipe. As we only have a T1 we are limited to 1.5Mb of traffic so we should set the incoming pipe to something below that.
sudo ipfw pipe 1 config bw 1.4Mbit/s
Now we can start to shake out the queues to fill that pipe. In our case the external interface is en0 and the internal Software Update server is at 192.168.10.100.
sudo ipfw queue 1 config pipe 1 weight 20
sudo ipfw queue 2 config pipe 1 weight 70
sudo ipfw queue 3 config pipe 1 weight 100
sudo ipfw queue 4 config pipe 1 weight 90
sudo ipfw queue 5 config pipe 1 weight 1
sudo ipfw add 10 queue 4 from any to 192.168.10.100 dst-port http in via en0
sudo ipfw add 20 queue 1 from any to any dst-port http in via en0
sudo ipfw add 30 queue 2 from any to any dst-port 20-21 in via en0
sudo ipfw add 40 queue 3 from any to any dst-port smtp in via en0
sudo ipfw add 500 queue 5 from any to any in via en0
Now our traffic should be nice and shapely. Note that I had to place the queue for the SUS above the general http queue. Since queues without traffic are not evaluated uses won't notice a change most of the time, but when traffic, FTP, SMTP, new software updates that really need to get in the door arrives it will be able to.
Advance pipe fitting
So far all of our examples have been network or interface based. What happens when you want to shape traffic on a per host basis? Luckily dummynet has this covered too through the use of address masking.
Let's say for example that you don't have a total limit on outgoing FTP bandwidth to enforce on your server, but you do want to limit each individual connection to 25Kbit/s. This is easy to do by defining a mask for the network that you want to restrict.
sudo ipfw pipe 50 config mask dst-ip 0x000000ff bw 25Kbit/s
sudo ipfw add 100 pipe 50 src-port 20-21
Now each outbound FTP connection will find itself in a dynamically generated 25Kbit/s pipe. Keep in mind that masks can be applied to queues as well, so it is entirely possible to have LOTS of queues and pipes in your ipfw ruleset. This is a good place to test and apply common sense.
Mac OS X Server gotchas...
We now know how to create pipes and queues, great. The problem is how do we apply these on Mac OS X Server. If you just use the ipfw commands above the rules will not survive a reboot or ipfw flush. Server Admin doesn't seem to help out here either, so we will need to resort to editing configuration files, but there is some good news here. Unlike other services on Mac OS X Server, Apple has gone out of it's way to make it easy for you to maintain a custom ruleset from the standard config files. They've even commented the files to explain how servermgrd composites and loads the rules. Just take a look at /etc/ipfilter/ipfw.conf and you will find all the info you need to make these rules stick. Many thanks to Apple on this one.
Wrapping up
With the addition of dummynet, Mac OS X 10.4 has finally caught up to regular FreeBSD in the arena of traffic shaping. With these tools you can make sure that your critical services always have the bandwidth they need to keep everything running smoothly. Remember that like all ipfw rules the dummynet rules are dynamic in their application. You can take advantage of this for real-time traffic shaping. If the CEO calls and says that his FTP transfer is going much too slowly you can change the overall FTP queue, or just create a new queue for him. The changes take effect immediately and are easily rolled back once the crisis has past.
The biggest challenge most sysadmins will face is the task of integrating your dummynet rules into your existing ipfw rulesets. Just take the time to plan it out carefully and you should be OK.
As always, have fun and read the man pages!