Virtualization7 min read

Controllers (CPU, Memory, I/O)

CPU throttling, memory limits, I/O control, and fork bomb protection

CPU Controller

The CPU controller manages how much processor time a cgroup gets. It has two distinct mechanisms: proportional sharing (how to divide CPU when contended) and bandwidth limiting (hard cap regardless of contention).

cpu.weight — Proportional CPU Sharing

cpu.weight (v2) / cpu.shares (v1)

Default: 100 (range: 1 - 10000)
Only matters when CPU is contended — if only one cgroup is busy, it gets 100% regardless of its weight
Weights are relative: a cgroup with weight 200 gets 2x the CPU time of a cgroup with weight 100
Think of it as "priority shares" — the kernel divides available CPU proportionally

CPU Contention Example: 1 CPU, 3 cgroups all busy

App A

weight: 200

50%

200/(200+100+100)

App B

weight: 100

25%

100/(200+100+100)

App C

weight: 100

25%

100/(200+100+100)

If App B and App C become idle, App A gets 100% — weights only matter under contention

cpu.max — Hard CPU Bandwidth Limit

cpu.max (v2) / cpu.cfs_quota_us + cpu.cfs_period_us (v1)

Format: $MAX $PERIOD (both in microseconds)
The cgroup can use at most $MAX microseconds of CPU per $PERIOD microsecond window
Default: max 100000 (unlimited quota, 100ms period)
CFS bandwidth throttling: when quota is exhausted, all threads in the cgroup are paused until the next period

cpu.max Value	Effective Limit	Docker Equivalent
`200000 100000`	2 CPUs (200ms per 100ms period)	`--cpus 2`
`100000 100000`	1 CPU (100ms per 100ms period)	`--cpus 1`
`50000 100000`	0.5 CPUs (50ms per 100ms period)	`--cpus 0.5`
`25000 100000`	0.25 CPUs (25ms per 100ms period)	`--cpus 0.25`
`max 100000`	Unlimited	(default — no `--cpus` flag)

Note

The period is typically 100ms (100000us). With 200000 100000, the cgroup can burn 200ms of CPU time per 100ms wall-clock period. Since this exceeds 100ms, it must use multiple CPU cores to fill the quota — that is how you express "2 CPUs" in cgroup terms.

cpuset Controller — CPU Pinning

cpuset: Pin Processes to Specific Cores

cpuset.cpus = 0-3 means only use cores 0, 1, 2, 3
cpuset.cpus = 0,2,4 means only use cores 0, 2, and 4
cpuset.mems = NUMA node pinning (which memory banks to use)

Use Cases for CPU Pinning

Latency-sensitive workloads (avoid cache thrashing from core migration)
NUMA-aware placement (keep processes close to their memory)
Isolating noisy neighbors (dedicate cores to critical services)
Real-time workloads that need consistent performance

cpuset vs cpu.max

cpuset: controls which cores — spatial partitioning
cpu.max: controls how much time — temporal throttling
They can be combined: pin to cores 0-3, AND limit to 2 CPUs of bandwidth
cpuset alone does not limit CPU usage within the allowed cores

Memory Controller

The memory controller manages RAM and swap consumption. It provides a layered set of limits — from hard kills to soft pressure — giving fine-grained control over how the kernel handles memory scarcity.

Memory Limit Hierarchy

memory.max

Hard limit. OOM killer fires when exceeded.

512M

memory.high

Soft throttle. Triggers aggressive reclaim, slows allocation.

400M

memory.low

Best-effort protection. Memory below this is protected from reclaim.

128M

memory.min

Hard protection. Memory below this is NEVER reclaimed.

64M

Limits enforce a ceiling. Protections enforce a floor.

OOM Killer Behavior

What Happens When memory.max Is Hit

The kernel attempts memory reclaim (swap, page cache eviction) first
If reclaim cannot free enough memory, the OOM killer activates
The OOM killer selects and kills a process within the cgroup — not system-wide
memory.oom.group = 1 — when set, the OOM killer kills all processes in the cgroup instead of picking one
This is what happens when a Kubernetes pod gets OOMKilled

Without memory.oom.group

OOM killer picks one process in the cgroup (based on oom_score). Other processes in the cgroup continue running.

May leave the application in an inconsistent state if only one worker dies.

With memory.oom.group = 1

OOM killer terminates all processes in the cgroup at once.

Clean kill. This is what containers typically want — kill the whole container, let the orchestrator restart it.

Swap Limits and Accounting

File	Purpose
`memory.swap.max`	Maximum swap usage (independent of RAM limit). Set to `0` to disable swap for this cgroup.
`memory.current`	Current total memory usage (RSS + page cache)
`memory.swap.current`	Current swap usage
`memory.stat`	Detailed breakdown: anon (RSS), file (page cache), kernel stacks, slab, sock, shmem, swap
`memory.peak`	High-water mark — maximum memory.current ever reached

console


    $ cat /sys/fs/cgroup/my-app/worker/memory.stat

    anon 25165824

file 16777216

kernel 1048576

sock 0

shmem 0

zswap 0

zswapped 0

file_mapped 4194304

file_dirty 0

file_writeback 0

swapcached 0

anon_thp 0

inactive_anon 12582912

active_anon 12582912

inactive_file 8388608

active_file 8388608

pgfault 15234

pgmajfault 3

...

I/O Controller

The I/O controller limits disk read/write bandwidth and IOPS per block device. Like the CPU controller, it supports both proportional sharing and hard limits.

I/O Control Files

File	Purpose	Format
`io.weight`	Proportional I/O bandwidth (like cpu.weight)	`default 100` or `MAJ:MIN 200`
`io.max`	Hard I/O limits per device	`MAJ:MIN rbps=X wbps=Y riops=Z wiops=W`
`io.latency`	Latency-based I/O control	`MAJ:MIN target=Xus`
`io.stat`	Per-device I/O statistics	(read-only)

io.max Examples

console


    # # Find the major:minor numbers for /dev/sda

    $ lsblk -o NAME,MAJ:MIN

    NAME    MAJ:MIN

sda       8:0

|-sda1    8:1

|-sda2    8:2

    

    # # Limit /dev/sda reads to 10 MB/s

    # echo "8:0 rbps=10485760" > /sys/fs/cgroup/my-app/worker/io.max

    

    # # Limit both reads and writes, plus IOPS

    # echo "8:0 rbps=10485760 wbps=5242880 riops=1000 wiops=500" > io.max

    

    # # Limit format explanation:

    # #   rbps  = read bytes per second   (10 MB/s)

    # #   wbps  = write bytes per second  (5 MB/s)

    # #   riops = read I/O ops per second  (1000)

    # #   wiops = write I/O ops per second (500)

Warning

Caveat: I/O cgroup throttling only works reliably with direct I/O or when the filesystem supports cgroup-aware writeback. With buffered I/O, writes go to the page cache first and are written back by kernel threads that may be in a different cgroup — the limits may not apply as expected. ext4 and btrfs support cgroup writeback; XFS gained support in kernel 5.18+.

PIDs Controller

The simplest controller — it limits the maximum number of processes (and threads) in a cgroup.

Fork Bomb Protection

pids.max — maximum number of processes/threads allowed in the cgroup
pids.current — current count of processes/threads
When the limit is hit, fork() and clone() return EAGAIN
Without this, a runaway container running :(){ :|:& };: can exhaust the host's entire PID space and freeze the system

Without pids.max

console

# Fork bomb in a container
:(){ :|:& };:

# Result: exponential process creation
# Consumes all PIDs on the host
# Host becomes unresponsive
# Even SSH may stop working

With pids.max = 100

console

# Fork bomb in a container
:(){ :|:& };:

# Result: stops at 100 processes
# fork() returns EAGAIN
# Container is broken, but host is fine
# Other containers unaffected

Devices Controller

Controls which block and character devices a cgroup can access. This prevents containers from accessing host hardware directly.

Device Access Model

v1: Explicit device controller with devices.allow and devices.deny
v2: Device access is managed via eBPF programs attached to the cgroup (the devices controller was replaced by BPF_PROG_TYPE_CGROUP_DEVICE)
Allowlist model: deny all by default, then allow specific devices

Typical Container Device Allowlist

Device	Type	Allowed	Purpose
`/dev/null`	char 1:3	Yes	Discard output
`/dev/zero`	char 1:5	Yes	Zero bytes source
`/dev/urandom`	char 1:9	Yes	Random number generator
`/dev/tty`	char 5:0	Yes	Terminal access
`/dev/sda`	block 8:0	No	Host disk — denied
`/dev/kmsg`	char 1:11	No	Kernel message buffer — denied

Tip

Docker --device flag: When you run docker run --device /dev/fuse, Docker adds that device to the cgroup's allowlist and bind-mounts it into the container's filesystem. Without --device, the container cannot access it even if it has the right capabilities.

Controller Summary

Controller	Key Files (v2)	What It Controls	Default Behavior
cpu	`cpu.weight`, `cpu.max`	CPU time (proportional sharing + bandwidth)	Unlimited, fair share
memory	`memory.max`, `memory.high`, `memory.low`	RAM + swap consumption	Unlimited
io	`io.weight`, `io.max`	Disk bandwidth and IOPS	Unlimited
pids	`pids.max`	Process/thread count	Unlimited
cpuset	`cpuset.cpus`, `cpuset.mems`	CPU core and NUMA pinning	All CPUs, all NUMA nodes
devices	(eBPF-based in v2)	Block/char device access	All denied (allowlist model)

Note

Remember: All of these have "unlimited" as the default. Containers get their limits because the container runtime (Docker, containerd, CRI-O) writes values to these files when creating the container's cgroup. Without a runtime setting limits, a container could consume all host resources.