Networking11 min read

iproute2 & Traffic Control

Modern Linux networking tools: ip, tc, and sysctl knobs

iproute2: The Modern Networking Toolbox

iproute2 is the modern replacement for the legacy net-tools package. It provides a unified interface (ip) for configuring every aspect of Linux networking, replacing a patchwork of older commands that had inconsistent syntax and limited functionality.

Legacy Command	iproute2 Replacement	Purpose
`ifconfig`	`ip addr` / `ip link`	Interface addresses and state
`route`	`ip route`	Routing table
`arp`	`ip neigh`	ARP / NDP neighbor table
`netstat`	`ss`	Socket statistics
`brctl`	`ip link` + `bridge`	Bridge management
`vconfig`	`ip link add ... type vlan`	VLAN interfaces
`iptunnel`	`ip tunnel`	Tunnel interfaces

Warning

Stop using net-tools. ifconfig and route are deprecated, unmaintained, and don't support modern features (namespaces, policy routing, VRFs, multiple addresses per interface displayed correctly). Many minimal container images don't even include them. Always use ip.

`ip addr` — Interface Addresses

console


    # List all interfaces and addresses

    $ ip addr show

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

        inet 127.0.0.1/8 scope host lo

        inet6 ::1/128 scope host

    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP

        link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff

        inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0

        inet6 fe80::5054:ff:fe12:3456/64 scope link


    # Show only a specific interface

    $ ip addr show dev eth0


    # Add an address

    $ ip addr add 192.168.1.100/24 dev eth0


    # Delete an address

    $ ip addr del 192.168.1.100/24 dev eth0


    # Show only IPv4 addresses (brief format)

    $ ip -4 -br addr show

    lo       UNKNOWN  127.0.0.1/8

    eth0     UP       10.0.2.15/24

`ip route` — Routing Table

console


    # Show routing table

    $ ip route show

    default via 10.0.2.1 dev eth0 proto dhcp metric 100

    10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100

    172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1


    # Which route would be used for a given destination?

    $ ip route get 8.8.8.8

    8.8.8.8 via 10.0.2.1 dev eth0 src 10.0.2.15 uid 0


    # Add a static route

    $ ip route add 10.244.0.0/16 via 10.0.2.100 dev eth0


    # Add a route via a specific interface (no gateway)

    $ ip route add 192.168.100.0/24 dev vlan100


    # Delete a route

    $ ip route del 10.244.0.0/16


    # Replace a route (add or update atomically)

    $ ip route replace default via 10.0.2.1 dev eth0

`ip link` — Network Interfaces

console


    # List all interfaces (brief format)

    $ ip -br link show

    lo       UNKNOWN  00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>

    eth0     UP       52:54:00:12:34:56 <BROADCAST,MULTICAST,UP,LOWER_UP>

    docker0  UP       02:42:ac:11:00:01 <BROADCAST,MULTICAST,UP,LOWER_UP>


    # Bring interface up/down

    $ ip link set eth0 up

    $ ip link set eth0 down


    # Set MTU

    $ ip link set eth0 mtu 9000


    # Create virtual interfaces

    $ ip link add br0 type bridge

    $ ip link add veth0 type veth peer name veth1

    $ ip link add vlan100 link eth0 type vlan id 100


    # Delete an interface

    $ ip link del br0

`ip neigh` — ARP / NDP Table

console


    # Show ARP table

    $ ip neigh show

    10.0.2.1 dev eth0 lladdr 52:54:00:12:35:02 REACHABLE

    10.0.2.3 dev eth0 lladdr 52:54:00:12:35:03 STALE

    172.17.0.2 dev docker0 lladdr 02:42:ac:11:00:02 REACHABLE


    # Add a static ARP entry

    $ ip neigh add 10.0.2.50 lladdr aa:bb:cc:dd:ee:ff dev eth0 nud permanent


    # Flush ARP cache for a device

    $ ip neigh flush dev eth0

`ip netns` & `ip rule`

console


    # Network namespaces (covered in detail in namespaces-veth.html)

    $ ip netns list

    $ ip netns add testing

    $ ip netns exec testing ip addr show

    $ ip netns del testing


    # Policy routing rules (covered in next section)

    $ ip rule list

    0:    from all lookup local

    32766:  from all lookup main

    32767:  from all lookup default

Policy-Based Routing

Standard routing makes decisions based solely on destination IP. Policy-based routing (PBR) extends this by allowing routing decisions based on source IP, incoming interface, fwmark, TOS, UID, and more. It works through multiple routing tables and rules that select which table to consult.

How It Works

Packet arrives
has src, dst, mark, etc.
ip rule match
walk rules by priority
Select routing table
e.g., "lookup table1"
Route lookup
in that specific table
Forward packet
via matched route

Default Routing Tables

console


    $ ip rule list

    0:      from all lookup local      # Highest priority: local addresses (127.0.0.1, etc.)

    32766:  from all lookup main       # Normal routing table (what "ip route show" displays)

    32767:  from all lookup default    # Fallback (usually empty)

Note

Rule priority: Lower number = higher priority. Rules are evaluated top-to-bottom until a match is found. You insert custom rules between the defaults.

Use Case: Dual ISP Uplinks

Route traffic from different subnets through different ISPs:

Subnet A

10.0.1.0/24

→

Linux Router

policy routing

→

ISP 1

via 203.0.113.1

Subnet B

10.0.2.0/24

→

Linux Router

policy routing

→

ISP 2

via 198.51.100.1

console


    # Create custom routing tables (names in /etc/iproute2/rt_tables)

    $ echo "100 isp1" >> /etc/iproute2/rt_tables

    $ echo "200 isp2" >> /etc/iproute2/rt_tables


    # Add default routes to each table

    $ ip route add default via 203.0.113.1 dev eth1 table isp1

    $ ip route add default via 198.51.100.1 dev eth2 table isp2


    # Add rules: source subnet determines which table

    $ ip rule add from 10.0.1.0/24 lookup isp1 priority 100

    $ ip rule add from 10.0.2.0/24 lookup isp2 priority 200


    # Verify

    $ ip rule list

    0:      from all lookup local

    100:    from 10.0.1.0/24 lookup isp1

    200:    from 10.0.2.0/24 lookup isp2

    32766:  from all lookup main

    32767:  from all lookup default

Use Case: Kubernetes Pod Routing

How CNIs Use Policy Routing

Kubernetes CNIs (e.g., AWS VPC CNI, Calico) use policy routing extensively
Pod traffic is routed through a separate table using ip rule add from <pod-cidr> lookup <pod-table>
This keeps pod routing independent from the node's default routing table
AWS VPC CNI creates per-ENI routing tables, with rules selecting the table based on the source pod IP
fwmark-based rules are used for traffic that has been marked by iptables (e.g., traffic needing SNAT)

Traffic Control (`tc`)

tc is the Linux traffic control subsystem. It shapes, schedules, polices, and drops network traffic at the egress (and sometimes ingress) of network interfaces. It operates below iptables/netfilter in the stack.

Architecture: qdiscs, classes, filters

Packet ready to send
from kernel stack
Filters (classifiers)
match → assign to class
Classes
bandwidth allocation
Qdisc (queue)
scheduling algorithm
NIC driver
transmit to wire

Component	Role	Analogy
qdisc (queueing discipline)	Scheduling algorithm that decides when packets are sent	The queue itself (FIFO, priority, fair, etc.)
class	Subdivision within a classful qdisc, each with its own rate/ceiling	Lanes at a toll booth with different speed limits
filter	Classifier that assigns packets to specific classes	Traffic cop directing cars into lanes

Common Queueing Disciplines

Qdisc	Type	Description	Use Case
`pfifo_fast`	Classless	3 priority bands based on TOS bits. Default on older kernels.	General purpose (legacy default)
`fq_codel`	Classless	Fair Queuing + Controlled Delay. Per-flow queuing with AQM to fight bufferbloat.	Modern default (since kernel 3.12+). Best general-purpose qdisc.
`htb`	Classful	Hierarchical Token Bucket. Rate limiting with borrowing between classes.	Bandwidth sharing with guarantees. ISP traffic shaping.
`tbf`	Classless	Token Bucket Filter. Simple rate limiter.	Hard rate limit on an interface.
`netem`	Classless	Network Emulator. Adds delay, packet loss, duplication, reordering.	Chaos engineering, testing. Simulating WAN conditions.
`ingress`	Special	Allows policing (drop/rate-limit) on incoming traffic.	Ingress policing. Used by Cilium bandwidth manager.

Note

Classful vs classless: Classless qdiscs handle all traffic the same way (or with built-in internal logic). Classful qdiscs (HTB, CBQ) let you define a hierarchy of classes, each with its own rate, and attach filters to direct traffic into classes.

Examples

Rate Limiting with TBF

console


      # Limit eth0 to 10 Mbit/s

      $ tc qdisc add dev eth0 root tbf rate 10mbit burst 32kbit latency 400ms


      # View current qdisc

      $ tc qdisc show dev eth0

      qdisc tbf 8001: root refcnt 2 rate 10Mbit burst 4Kb lat 400ms


      # Remove the qdisc (restore default)

      $ tc qdisc del dev eth0 root

Network Emulation with netem (Chaos Engineering)

console


      # Add 100ms delay with 10ms jitter (normal distribution)

      $ tc qdisc add dev eth0 root netem delay 100ms 10ms distribution normal


      # Add 1% packet loss

      $ tc qdisc change dev eth0 root netem delay 100ms loss 1%


      # Add packet duplication (0.5%) and reordering (25%)

      $ tc qdisc change dev eth0 root netem delay 100ms duplicate 0.5% reorder 25%


      # Simulate complete network partition

      $ tc qdisc add dev eth0 root netem loss 100%


      # Clean up

      $ tc qdisc del dev eth0 root

HTB Bandwidth Sharing with Classes

console


      # Create HTB root qdisc

      $ tc qdisc add dev eth0 root handle 1: htb default 30


      # Root class: 100 Mbit total

      $ tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit


      # Child class: guaranteed 50 Mbit, can burst to 100 Mbit

      $ tc class add dev eth0 parent 1:1 classid 1:10 htb rate 50mbit ceil 100mbit


      # Child class: guaranteed 30 Mbit, can burst to 100 Mbit

      $ tc class add dev eth0 parent 1:1 classid 1:20 htb rate 30mbit ceil 100mbit


      # Default class: 20 Mbit

      $ tc class add dev eth0 parent 1:1 classid 1:30 htb rate 20mbit ceil 50mbit


      # Filter: port 80/443 traffic → class 1:10

      $ tc filter add dev eth0 parent 1: protocol ip prio 1 u32 \

          match ip dport 80 0xffff flowid 1:10

      $ tc filter add dev eth0 parent 1: protocol ip prio 1 u32 \

          match ip dport 443 0xffff flowid 1:10

Tip

Kubernetes bandwidth limiting: Cilium's bandwidth manager uses tc with EDT (Earliest Departure Time) scheduling on the fq qdisc, which is more efficient than HTB policing. The kubernetes.io/egress-bandwidth annotation on pods is implemented via tc under the hood.

Key sysctl Networking Knobs

The Linux kernel exposes hundreds of tunable parameters via /proc/sys/net/. These are the ones you'll encounter most often in production systems, K8s nodes, and load balancers.

sysctl Parameter	Default	What It Does	When to Tune
`net.ipv4.ip_forward`	0	Enable IP forwarding between interfaces (turns host into a router)	Always enable on K8s nodes, Docker hosts, any router/NAT gateway. Required for container networking.
`net.ipv4.conf.all.rp_filter`	1 (strict)	Reverse path filtering: validates source IP is reachable via the interface it arrived on. Anti-spoofing.	Set to `0` (off) or `2` (loose) for asymmetric routing, multi-homed hosts, overlay networks (Calico, Flannel).
`net.ipv4.tcp_keepalive_time`	7200 (2h)	Seconds before first keepalive probe on idle TCP connection	Lower to 60-300s for LB health checks, cloud environments where idle connections get killed by NAT gateways (AWS ALB: 350s timeout).
`net.ipv4.tcp_keepalive_intvl`	75	Seconds between keepalive probes after first one	Lower alongside keepalive_time. Common: 10-30s.
`net.ipv4.tcp_keepalive_probes`	9	Number of failed probes before connection is considered dead	Lower to 3-5 for faster dead connection detection.
`net.core.somaxconn`	4096	Maximum length of the listen() backlog queue for a socket	Increase on high-traffic servers (nginx, envoy). Application must also set a high backlog in listen().
`net.ipv4.tcp_tw_reuse`	2	Allow reuse of TIME_WAIT sockets for new outgoing connections (to same dst)	Enable (`1`) on servers making many short-lived outbound connections (proxies, app servers calling APIs). Safe with timestamps enabled.
`net.netfilter.nf_conntrack_max`	262144	Maximum entries in the conntrack table	Increase on NAT gateways, K8s nodes with many services, load balancers. Monitor `nf_conntrack_count` vs this value.
`net.ipv4.tcp_max_syn_backlog`	1024	Maximum number of queued connection requests not yet ACKed	Increase on high-connection-rate servers. Works with `somaxconn`.
`net.ipv4.tcp_fin_timeout`	60	Time (seconds) to hold socket in FIN-WAIT-2 state	Lower to 15-30s on busy servers to free sockets faster.
`net.ipv4.ip_local_port_range`	32768-60999	Range of ephemeral ports for outgoing connections	Widen to `1024-65535` on high-connection-rate systems (proxies, NAT gateways).
`net.core.netdev_max_backlog`	1000	Maximum packets queued on INPUT side when interface receives faster than kernel can process	Increase to 5000-10000 on high-throughput servers (10G+).

Applying sysctl Settings

console


    # Temporarily set (lost on reboot)

    $ sysctl -w net.ipv4.ip_forward=1


    # Read current value

    $ sysctl net.ipv4.ip_forward

    net.ipv4.ip_forward = 1


    # Persist across reboots: add to /etc/sysctl.d/99-k8s.conf

    $ cat /etc/sysctl.d/99-k8s.conf

    net.ipv4.ip_forward = 1

    net.bridge.bridge-nf-call-iptables = 1

    net.bridge.bridge-nf-call-ip6tables = 1

    net.netfilter.nf_conntrack_max = 1048576

    net.ipv4.tcp_keepalive_time = 120

    net.ipv4.tcp_keepalive_intvl = 30

    net.ipv4.tcp_keepalive_probes = 3

    net.core.somaxconn = 65535


    # Apply persisted settings

    $ sysctl --system

Warning

bridge-nf-call-iptables: This sysctl (net.bridge.bridge-nf-call-iptables = 1) makes bridged traffic pass through iptables rules. Kubernetes requires this so that kube-proxy's iptables rules can intercept pod-to-service traffic that traverses a bridge. Without it, pod-to-ClusterIP traffic bypasses iptables entirely.

Recommended Production Baseline (K8s Nodes)

console

# /etc/sysctl.d/99-kubernetes-node.conf

# Required for K8s networking
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

# Conntrack (increase for large clusters)
net.netfilter.nf_conntrack_max = 1048576

# TCP keepalive (detect dead connections faster)
net.ipv4.tcp_keepalive_time = 120
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3

# Connection handling
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1

# Disable rp_filter for overlay networks
net.ipv4.conf.all.rp_filter = 0

# High-throughput servers
net.core.netdev_max_backlog = 5000

Summary: Quick Reference

iproute2 Cheat Sheet

`ip -br addr`	Brief interface/address list
`ip route get <ip>`	Which route is used?
`ip route replace`	Atomic add-or-update route
`ip neigh show`	ARP table
`ip rule list`	Policy routing rules
`ip -s link show`	Interface statistics (RX/TX bytes, errors, drops)
`ip monitor`	Real-time event stream (link, route, addr changes)

tc Cheat Sheet

`tc qdisc show dev eth0`	Show current qdisc
`tc -s qdisc show dev eth0`	Show with statistics
`tc qdisc add ... root tbf`	Rate limit
`tc qdisc add ... root netem`	Network emulation
`tc qdisc add ... root htb`	Hierarchical bandwidth
`tc qdisc del dev eth0 root`	Remove all tc config
`tc class show dev eth0`	Show classes

iproute2: The Modern Networking Toolbox

ip addr — Interface Addresses

ip route — Routing Table

ip link — Network Interfaces

ip neigh — ARP / NDP Table

ip netns & ip rule

Policy-Based Routing

How It Works

Packet arrives

ip rule match

Select routing table

Route lookup

Forward packet

Default Routing Tables

Use Case: Dual ISP Uplinks

Use Case: Kubernetes Pod Routing

Traffic Control (tc)

Architecture: qdiscs, classes, filters

Packet ready to send

Filters (classifiers)

Classes

Qdisc (queue)

NIC driver

Common Queueing Disciplines

Examples

Key sysctl Networking Knobs

Applying sysctl Settings

Summary: Quick Reference

`ip addr` — Interface Addresses

`ip route` — Routing Table

`ip link` — Network Interfaces

`ip neigh` — ARP / NDP Table

`ip netns` & `ip rule`

Traffic Control (`tc`)