QoS & Traffic Shaping

Deciding Who Goes First#

A network link is a shared resource. When more traffic arrives than a link can send, packets must wait in a queue or be dropped. Quality of Service (QoS) is the set of techniques for deciding which traffic waits, which is dropped, and which goes first. It is how a video call stays smooth while someone else on the same connection uploads a large backup.

Why Not Just Add Bandwidth?#

Extra capacity helps, but congestion happens in bursts and at bottlenecks you do not always control: a home uplink, a busy WAN circuit, a Wi-Fi channel, a peering link at peak hour. Even a mostly idle link is congested for the milliseconds during which a burst arrives faster than it can drain. QoS matters wherever a fast link feeds a slower one, because that is where the queue forms. Its goal is not to eliminate congestion but to control who experiences it.

Where the Delay Actually Comes From: Bufferbloat#

When a link is saturated, packets wait in the buffer of the device in front of the bottleneck. If that buffer is large and managed as a simple first-in-first-out queue, a single bulk upload fills it, and every other packet, including your video call's audio and your game's input, waits behind it. This is bufferbloat, and on consumer connections it commonly turns 20 ms of idle latency into hundreds of milliseconds under load.

You can measure it yourself:

# Terminal 1: baseline latency
ping -c 30 1.1.1.1

# Terminal 2: saturate the uplink (against an iperf3 server you control)
iperf3 -c iperf.example.net -t 30

# Terminal 1 again, while the upload runs
ping -c 30 1.1.1.1

Compare the average and maximum RTT between the two runs. If loaded latency is many times the idle latency, the bottleneck buffer is bloated, and fixing the queue will do more for interactive traffic than a faster plan would.

The Traffic Management Toolkit#

• Classification identifies traffic by port, protocol, address, existing markings, or deep inspection.
• Marking records the classification in the packet's DSCP field so later devices do not have to reclassify.
• Queuing places classes into separate queues with different scheduling (strict priority, weighted fair, class-based).
• Shaping holds excess packets in a buffer and releases them at a target rate: it delays.
• Policing enforces a rate by dropping or re-marking whatever exceeds it: it never delays.
• Active queue management (AQM), such as CoDel, drops or marks packets early when a queue's delay grows, signalling TCP senders to slow down before the buffer fills.

Shaping vs. Policing, Simulated#

Both use a token bucket: tokens arrive at the configured rate, the bucket holds up to a burst size, and sending a packet costs tokens equal to its size. The difference is what happens when the bucket is empty.

RATE = 125_000       # token refill: 125,000 bytes/s = 1 Mbit/s
BURST = 4_500        # bucket depth: three full-size packets
PKT = 1_500

# A burst of 8 full-size packets arriving 1 ms apart
arrivals = [(i * 0.001, PKT) for i in range(8)]

def police(packets):
    tokens, last, verdicts = BURST, 0.0, []
    for t, size in packets:
        tokens = min(BURST, tokens + (t - last) * RATE)
        last = t
        if tokens >= size:
            tokens -= size
            verdicts.append("pass")
        else:
            verdicts.append("DROP")          # a policer never waits
    return verdicts

def shape(packets):
    tokens, clock, departures = BURST, 0.0, []
    for t, size in packets:
        start = max(t, clock)                 # FIFO: wait behind earlier packets
        tokens = min(BURST, tokens + (start - clock) * RATE)
        if tokens < size:                     # not enough tokens: hold the packet
            start += (size - tokens) / RATE
            tokens = size
        tokens -= size
        clock = start
        departures.append(start)
    return departures

for (t, _), verdict, sent in zip(arrivals, police(arrivals), shape(arrivals)):
    print(f"arrive {t*1000:4.0f} ms | policer: {verdict:<4} | shaper sends at {sent*1000:5.1f} ms")

arrive    0 ms | policer: pass | shaper sends at   0.0 ms
arrive    1 ms | policer: pass | shaper sends at   1.0 ms
arrive    2 ms | policer: pass | shaper sends at   2.0 ms
arrive    3 ms | policer: DROP | shaper sends at  12.0 ms
arrive    4 ms | policer: DROP | shaper sends at  24.0 ms
arrive    5 ms | policer: DROP | shaper sends at  36.0 ms
arrive    6 ms | policer: DROP | shaper sends at  48.0 ms
arrive    7 ms | policer: DROP | shaper sends at  60.0 ms

The first three packets fit in the burst allowance. After that, the policer drops five of eight packets, and TCP will have to detect and retransmit them. The shaper drops nothing but spaces packets 12 ms apart (1,500 bytes at 1 Mbit/s), adding up to 53 ms of delay. Hence the rules of thumb: shape your own outbound traffic so the queue forms where you control it, and police at boundaries where you enforce a contract on someone else's traffic, as ISPs do at a customer's subscribed rate. API rate limiting is policing at the application layer; a well-behaved client that paces its own requests is shaping, the approach recommended in rate limiting and politeness.

Priority vs. Fairness#

• Strict priority queuing always sends the high-priority queue first. It gives voice the lowest possible delay, but a flood of "high-priority" traffic starves everything else, so the priority class must be rate-limited (policed) to a small share of the link.
• Fair queuing divides the link between flows so no single flow can monopolise it. Combined with AQM, as in Linux's fq_codel and CAKE, it keeps sparse, interactive flows (DNS, game packets, voice) near the front of the line automatically, without anyone marking anything. This is the most effective single fix for bufferbloat.

DSCP: Marking Traffic#

The DSCP field is the top six bits of the IPv4 ToS byte (or the IPv6 Traffic Class). Commonly used values:

An application can mark its own packets. On Linux:

import socket

EF = 46
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.IPPROTO_IP, socket.IP_TOS, EF << 2)   # DSCP sits above 2 ECN bits: 0xB8
sock.sendto(b"voice frame", ("192.0.2.50", 5004))

Verify with sudo tcpdump -vni eth0 udp port 5004, which should show tos 0xb8 in each packet header. The shift by two is the non-obvious part: the lower two bits of the byte belong to ECN, so DSCP 46 becomes 184. Windows ignores IP_TOS from applications by default and applies markings through Group Policy QoS rules instead.

Markings are only a request. Your own network should re-mark at the trust boundary, meaning it should not trust DSCP values arriving from user devices or the internet, or everyone will mark their downloads as EF. Most ISPs reset or ignore customer markings.

DiffServ vs. IntServ#

Differentiated Services (DiffServ), the model behind DSCP, classifies and marks at the edge, and each router in the core applies a per-hop behaviour for each class without keeping any per-flow state. It scales to any size. Integrated Services (IntServ) reserved bandwidth per flow with RSVP signalling along the path; it offers stronger guarantees but requires state for every flow on every router, which does not scale to internet traffic. DiffServ is what enterprise and provider networks run; RSVP survives mainly in traffic-engineering roles inside provider networks.

Doing It on Linux#

Linux's tc (traffic control) configures queuing disciplines ("qdiscs") per interface:

# Fair queuing with AQM on an interface (often already the default)
sudo tc qdisc replace dev eth0 root fq_codel

# CAKE shaping the uplink to just below the real bottleneck rate,
# with four priority tiers based on DSCP
sudo tc qdisc replace dev eth0 root cake bandwidth 18mbit diffserv4

# Inspect: sent bytes, drops, backlog, and per-tier delay statistics
tc -s qdisc show dev eth0

The key detail is shaping slightly below the true bottleneck, typically 85–95% of the measured line rate. The queue then builds in your router, where CAKE manages it intelligently, instead of in the modem's or ISP's bloated buffer, where you have no control. Shape at or above the line rate and nothing changes. Shaping the download direction is harder, because packets have already crossed the bottleneck by the time they reach you; on Linux it requires redirecting ingress traffic through an IFB device, and it works because TCP senders slow down in response to the drops and delays.

How QoS Fails in Practice#

• Marking without matching queues. Packets are marked at the edge, but downstream devices have no policy for the classes, so every class is treated identically. Verify the policy on every hop you control.
• Shaping above the bottleneck. The shaper never engages, and the real queue still forms upstream.
• An oversized priority class. When half the traffic is marked "priority", nothing is. Keep strict priority for a small, policed slice.
• Expecting guarantees across the internet. Other operators' queues ignore your markings. Real-time applications compensate end to end with jitter buffers, loss-tolerant codecs, forward error correction, and adaptive bitrate; see TCP vs UDP for why such media usually runs over UDP.
• Measuring idle latency. What matters is latency, jitter, and loss under load. Rerun the bufferbloat test after every change and watch the per-class drop counters in tc -s qdisc.

QoS cannot create capacity. It decides who experiences the shortage when there is one, which on a well-tuned link means bulk transfers slow down slightly and interactive traffic does not notice.

Practice#

Run the bufferbloat test on your own connection and note the loaded latency. On a Linux router or a VM in the path, apply CAKE at about 90% of the measured upload rate, repeat the test, and compare both the latency figures and the drop counters in tc -s qdisc show.