A practical HOWTO on basic traffic-shaping

This article covers the basic shaping of network traffic, based on an example of four users sharing an ADSL connection in an equal manner.

The target audience are junior network-administrators and above, so some knowledge of iptables and bash scripting is assumed (that is to say I didn't comment the scripts as well as I probably should have.) ;)

Introduction

The concept of sharing anything with other people usually creates problems when it is put in practice. These usually arise when the act of sharing actively interferes with ones own personal use of the thing in question.

Sharing an Internet connection is a perfect example of this. Take one connection, share it between several geeks (to whom this connection is as important as oxygen is to everyone else) and wait. If you are reading this then chances are you know all about the outcome already. If you do not, well, let me just say that it is not a pretty sight.

Eventually some manner of traffic control will have to be introduced. This will fall into one of the two broad categories: personal negotiation and rules, or technological measures to ensure fair use of the resource.

Personally, I prefer the technological option (implemented though agreement, if possible,) as it frees everyone to just get on with it, without too much thinking. This is where traffic shaping comes in.

The scene

Picture a 512Kbit ADSL connection (although the speed itself is very much irrelevant).

Picture four geeks eager to use this connection.

Picture a Linux firewall, whose mission is to stop the above mentioned geeks from killing each other, as each of them tries to download their share of warez, pr0n and software updates. Some theory

When implementing such a simple traffic-shaping solution, there are only a few basic requirements.

Ensure that upload traffic does not interfere with download traffic (which is does by default). We will do this by limiting the overall upload traffic to a rate that is lower than the maximum upload rate of our line.
Ensure that bandwidth is shared equally between users and that unused bandwidth is available to everyone, rather than simply wasted. To achieve this we will divide the upload and download bandwidth into five classes. Each of these will be reserved for one of the users and one will be spare, in case a "guest" machine is added onto the network. If any of the classes have unused bandwidth, then this will be shared equally between the active users.

The firewall computer will, in this example, be running the 'stable' branch of Debian GNU/Linux (version 3.1, or "Sarge", at the time of writing). This is not a requirement, any recent version of Linux (kernels 2.4 and above) should also work.

The only real requirement is that the firewall must have a minimum of two network interfaces. The reason for this is that traffic-shaping is all about control. However, the only traffic that we can definitely control is the traffic that we are sending out of an interface, as we can't really control people are sending into the interface. Therefore, shaping of upload traffic will be done on the external network interface of the firewall and the shaping of the download traffic will be done on the internal interface of the firewall.

Traffic-shaping in practice

For this example we well need iptables (with the appropriate CLASSIFY patch, if necessary) and iproute2. Debian packages are iptables (installed by default) and iproute.

We will be using the HTB shaper (or "queueing discipline").

We will classify each packet into one of the defined classes. We will look at two slightly different methods of classifying packets:

Marking the packets with iptables and then using the tc filter mechanism to classify them.
Classifying the packets using iptables directly.

Personally I prefer the second method, because it is more direct (and because I have had problems with the first method on a machine with a 2.6 kernel and several interfaces, i.e. it did not work) but I include the first method as well, because it will work with older versions of iptables (without support for CLASSIFY). The scripts provided at the bottom of this page have an option to use either.

Firewall

Each of our four users has a static IP address Therefore, we can use iptables to classify, or mark packets based on source and destination IP address (for uploads and downloads respectively.) All other traffic (for example a guest PC connected to the network) will be assigned to the default class.

Here is an example of using iptables to classify packets directly:

/sbin/iptables -t mangle -A POSTROUTING -s 192.168.1.11 -j CLASSIFY --set-class 1:11
/sbin/iptables -t mangle -A POSTROUTING -d 192.168.1.11 -j CLASSIFY --set-class 2:11
/sbin/iptables -t mangle -A POSTROUTING -s 192.168.1.12 -j CLASSIFY --set-class 1:12
/sbin/iptables -t mangle -A POSTROUTING -d 192.168.1.12 -j CLASSIFY --set-class 2:12
/sbin/iptables -t mangle -A POSTROUTING -s 192.168.1.13 -j CLASSIFY --set-class 1:13
/sbin/iptables -t mangle -A POSTROUTING -d 192.168.1.13 -j CLASSIFY --set-class 2:13
/sbin/iptables -t mangle -A POSTROUTING -s 192.168.1.14 -j CLASSIFY --set-class 1:14
/sbin/iptables -t mangle -A POSTROUTING -d 192.168.1.14 -j CLASSIFY --set-class 2:14

Alternatively, we can mark the packets:

/sbin/iptables -t mangle -A FORWARD -s 192.168.1.11 -j MARK --set-mark 111
/sbin/iptables -t mangle -A FORWARD -d 192.168.1.11 -j MARK --set-mark 211
/sbin/iptables -t mangle -A FORWARD -s 192.168.1.12 -j MARK --set-mark 112
/sbin/iptables -t mangle -A FORWARD -d 192.168.1.12 -j MARK --set-mark 212
/sbin/iptables -t mangle -A FORWARD -s 192.168.1.13 -j MARK --set-mark 113
/sbin/iptables -t mangle -A FORWARD -d 192.168.1.13 -j MARK --set-mark 213
/sbin/iptables -t mangle -A FORWARD -s 192.168.1.14 -j MARK --set-mark 114
/sbin/iptables -t mangle -A FORWARD -d 192.168.1.14 -j MARK --set-mark 214

The classes and marks used in the above examples can be anything, as long as they match the values in the traffic-shaping script.

Traffic Control

This can be much simplified by using more variables and "while... do... done" loops. I decided not to do this in the examples, or the scripts at the bottom of the page. as it would probably make them (even) harder to read. The scripts have a debug option that, when enabled, will show exactly what is happening.

The traffic control script consists of several parts. First, we define the root queueing discipline (attached directly to the interface) and also state the default class (in this example this would be for traffic not belonging to any of our four users.) Our external interface is eth0 and the internal interface is eth1.

The r2q value is an integer, calculated using the formula: Lowest bandwidth chunk (Bps) / MTU (Bps) > r2q. In our example the maximum upload speed is limited to 220Kbps (less than the line maximum of 256Kbps) so the minimum chunk would be 220/5 = 44Kbps. This means the r2q value will be 3 (5632Bps / 1500B > 3). The download r2q is calculated in the same way.

/sbin/tc qdisc add dev eth0 root handle 1: htb r2q 3 default 15
/sbin/tc qdisc add dev eth1 root handle 2: htb r2q 8 default 15

Next we attach to them a default class for each interface, which will define the maximum bandwidth available:

/sbin/tc class add dev eth0 parent 1: classid 1:10 htb rate 220Kbit ceil 220Kbit
/sbin/tc class add dev eth1 parent 2: classid 2:10 htb rate 510Kbit ceil 510Kbit

Then we define the classes for our users and the default class, attaching these to the root classes:

# User 1
/sbin/tc class add dev eth0 parent 1:10 classid 1:11 htb rate 44Kbit ceil 220Kbit
/sbin/tc class add dev eth1 parent 2:10 classid 2:11 htb rate 102Kbit ceil 510Kbit
# User 2
/sbin/tc class add dev eth0 parent 1:10 classid 1:12 htb rate 44Kbit ceil 220Kbit
/sbin/tc class add dev eth1 parent 2:10 classid 2:12 htb rate 102Kbit ceil 510Kbit
# User 3
/sbin/tc class add dev eth0 parent 1:10 classid 1:13 htb rate 44Kbit ceil 220Kbit
/sbin/tc class add dev eth1 parent 2:10 classid 2:13 htb rate 102Kbit ceil 510Kbit
# User 4
/sbin/tc class add dev eth0 parent 1:10 classid 1:14 htb rate 44Kbit ceil 220Kbit
/sbin/tc class add dev eth1 parent 2:10 classid 2:14 htb rate 102Kbit ceil 510Kbit
# Default, for everyting else
/sbin/tc class add dev eth0 parent 1:10 classid 1:15 htb rate 44Kbit ceil 220Kbit
/sbin/tc class add dev eth1 parent 2:10 classid 2:15 htb rate 102Kbit ceil 510Kbit

Then, because we are only configuring basic shaping, we add a simple SFQ queueing discipline to each of the classes. This will balance the bandwidth between any number of network 'conversations' within the class.

# User 1
/sbin/tc qdisc add dev eth0 parent 1:11 handle 111: sfq perturb 10
/sbin/tc qdisc add dev eth1 parent 2:11 handle 211: sfq perturb 10
# User 2
/sbin/tc qdisc add dev eth0 parent 1:12 handle 112: sfq perturb 10
/sbin/tc qdisc add dev eth1 parent 2:12 handle 212: sfq perturb 10
# User 3
/sbin/tc qdisc add dev eth0 parent 1:13 handle 113: sfq perturb 10
/sbin/tc qdisc add dev eth1 parent 2:13 handle 213: sfq perturb 10
# User 4
/sbin/tc qdisc add dev eth0 parent 1:14 handle 114: sfq perturb 10
/sbin/tc qdisc add dev eth1 parent 2:14 handle 214: sfq perturb 10
# Default, for everyone else
/sbin/tc qdisc add dev eth0 parent 1:15 handle 115: sfq perturb 10
/sbin/tc qdisc add dev eth1 parent 2:15 handle 215: sfq perturb 10

If we are using iptables to mark packets, then we will also need to add some filters to classify the marked packets. (If, on the other hand we use iptables to classify packets directly, then we can skip this step.) The filters are attached directly to the root queueing disciplines:

# User 1
/sbin/tc filter add dev eth0 parent 1: protocol ip prio 1 handle 111 fw classid 1:11
/sbin/tc filter add dev eth1 parent 2: protocol ip prio 1 handle 211 fw classid 2:11
# User 2
/sbin/tc filter add dev eth0 parent 1: protocol ip prio 1 handle 112 fw classid 1:12
/sbin/tc filter add dev eth1 parent 2: protocol ip prio 1 handle 212 fw classid 2:12
# User 3
/sbin/tc filter add dev eth0 parent 1: protocol ip prio 1 handle 113 fw classid 1:13
/sbin/tc filter add dev eth1 parent 2: protocol ip prio 1 handle 213 fw classid 2:13
# User 4
/sbin/tc filter add dev eth0 parent 1: protocol ip prio 1 handle 114 fw classid 1:14
/sbin/tc filter add dev eth1 parent 2: protocol ip prio 1 handle 214 fw classid 2:14

Conclusion

That's it. Hopefully this has helped you avoid unnecessary conflict over your shared Internet connection. The potential for a more complex and powerful setup is there. Have a read of the documentation in the reference section below for an idea of just how powerful Linux traffic-shaping can be. Even the simple example above is probably equivalent to a few thousand pounds in dedicated hardware. :)

Files

The two scripts in a format suitable for /etc/init.d/ use:

References

My article is, for the most part, little more than a fusion of the excellent documents listed below. My thanks go their authors for helping me to begin to understand this rather complex subject.

Disclaimer

The information above is accurate to my knowledge, however I provide no guarantees to this effect and consequently accept no liability whatsoever for any bad things that may happen as a result of the reader using this information in practice. Use at your own risk. Oh, and backup your data while you're at it.