Virtualization6 min read

cgroup Fundamentals

Resource control at the kernel level — not isolation, but limits and accounting

What cgroups Are

Control Groups (cgroups) are a Linux kernel feature for limiting, accounting, and isolating the resource usage of process groups. They answer the question: "how much of the system can this group of processes consume?"

The Key Distinction: Namespaces vs cgroups

Namespaces

Isolate WHAT you can see

PID namespace: can only see your own processes
Net namespace: can only see your own interfaces
Mount namespace: can only see your own filesystems

Visibility and identity isolation

cgroups

Limit HOW MUCH you can use

CPU controller: limit CPU time
Memory controller: limit RAM usage
I/O controller: limit disk bandwidth

Resource metering and enforcement

What cgroups Control

CPU

Proportional sharing, bandwidth throttling, core pinning

Memory

Hard limits, soft throttles, OOM kill policy, swap control

I/O

Disk read/write bandwidth and IOPS limits per device

PIDs

Maximum process count — fork bomb protection

Devices

Allowlist access to block/character devices

cpuset

Pin processes to specific CPU cores and NUMA nodes

Note

Key rule: Every process belongs to exactly one cgroup in each hierarchy. When you run cat /proc/self/cgroup, you see which cgroup the current process belongs to. A new process inherits its parent's cgroup membership.

The cgroup Virtual Filesystem

cgroups are managed through a virtual filesystem mounted at /sys/fs/cgroup/. You create cgroups by making directories, configure them by writing to files, and move processes by writing PIDs to cgroup.procs. No special system calls needed — it is all file I/O.

console


    $ ls /sys/fs/cgroup/

    cgroup.controllers    cgroup.procs          cpu.stat

cgroup.max.depth      cgroup.subtree_control  cpuset.cpus.effective

cgroup.max.descendants  cgroup.threads        io.stat

cgroup.stat           cpu.pressure           memory.stat

...

cgroups v1 vs v2

The cgroup subsystem has gone through two major design iterations. Understanding the difference matters because you will encounter both in production systems, and the mental model is fundamentally different.

cgroups v1 (2007)

Multiple independent hierarchies, one per controller

Each controller (cpu, memory, blkio, etc.) has its own directory tree
A process can be in different cgroups in different hierarchies
Complex to manage — race conditions between hierarchies
Inconsistent interfaces across controllers
Hard to coordinate limits (CPU + memory together)

console

# v1: separate trees per controller
/sys/fs/cgroup/cpu/
    docker/
        container-abc/
            cpu.shares
            cpu.cfs_quota_us

/sys/fs/cgroup/memory/
    docker/
        container-abc/
            memory.limit_in_bytes

/sys/fs/cgroup/blkio/
    docker/
        container-abc/
            blkio.throttle.read_bps_device

cgroups v2 (2016, default since kernel 5.x)

Single unified hierarchy for all controllers

One tree, all controllers attached to the same hierarchy
A process is in exactly one cgroup — consistent across all resources
Simpler, predictable, no cross-hierarchy coordination
Consistent interface naming (e.g., cpu.max not cpu.cfs_quota_us)
Better resource distribution model

console

# v2: unified tree, all controllers together
/sys/fs/cgroup/
    docker/
        container-abc/
            cpu.max
            cpu.weight
            memory.max
            memory.high
            io.max
            pids.max
            cgroup.procs

Tip

How to check which version you are running: If /sys/fs/cgroup/cgroup.controllers exists, you are on v2. If you see directories like /sys/fs/cgroup/cpu/, /sys/fs/cgroup/memory/, you are on v1 (or hybrid mode).

v2 Mechanics

cgroups v2 has a clean, consistent interface. All configuration happens through files in the cgroup directory. Here are the essential control files.

Core Control Files

File	Purpose	Read/Write
`cgroup.controllers`	Lists available controllers in this cgroup (inherited from parent)	Read-only
`cgroup.subtree_control`	Enables controllers for children of this cgroup	Read-write
`cgroup.procs`	Lists PIDs in this cgroup; write a PID here to move a process	Read-write
`cgroup.threads`	Like `cgroup.procs` but for individual threads (threaded mode)	Read-write
`cgroup.type`	Set to `threaded` for per-thread control	Read-write
`cgroup.events`	Reports state changes (populated, frozen)	Read-only

Enabling Controllers for Children

Controllers are not automatically available to child cgroups. You must explicitly enable them by writing to cgroup.subtree_control in the parent.

Check available
cat cgroup.controllers
Enable for children
echo "+cpu +memory" > cgroup.subtree_control
Create child
mkdir child-cgroup/
Configure child
echo "200000 100000" > child-cgroup/cpu.max

The "No Internal Processes" Rule

Warning

Critical v2 rule: A cgroup that has children cannot have its own processes directly. Processes must live in leaf cgroups only. This prevents ambiguous resource competition between a parent cgroup's own processes and its children. If you try to write a PID to cgroup.procs on a cgroup that has children with controllers enabled, the kernel will return EBUSY.

Invalid: processes in non-leaf cgroup

parent cgroup

PID 100, PID 200 (NOT ALLOWED)

child-a

PID 300

child-b

PID 400

Valid: all processes in leaf cgroups

parent cgroup

(no direct processes)

child-a

PID 100, PID 200, PID 300

child-b

PID 400

Delegation for Rootless Containers

By default, only root can create and manage cgroups. But unprivileged users can manage cgroups if they own the cgroup directory. This is the mechanism that enables rootless containers (Podman, rootless Docker).

How systemd Enables Delegation

systemd creates per-user cgroup slices automatically. Each user gets their own subtree in the cgroup hierarchy:

/sys/fs/cgroup/ (root cgroup)

system.slice

System services (sshd, docker, etc.)

user.slice

user-1000.slice

Owned by UID 1000

Podman rootless containers live here

user-1001.slice

Owned by UID 1001

Key Delegation Concepts

Ownership: When a user owns the cgroup directory, they can create child cgroups, move processes, and configure resource limits within their subtree
Scope: systemd-run --user --scope creates a delegated scope inside the user's slice — this is how Podman gets a cgroup it can manage
Limitation: Delegated users cannot raise limits above what the parent cgroup allows — they can only subdivide their own allocation
No privilege escalation: cgroup delegation does not grant any extra capabilities or namespace access

Hands-On: Working with cgroups v2

Check Available Controllers

console


    $ cat /sys/fs/cgroup/cgroup.controllers

    cpuset cpu io memory hugetlb pids rdma misc

Check What Is Enabled for Children

console


    $ cat /sys/fs/cgroup/cgroup.subtree_control

    cpuset cpu io memory hugetlb pids

Create a Child cgroup and Enable Controllers

console


    # mkdir /sys/fs/cgroup/my-app

    

    # ls /sys/fs/cgroup/my-app/

    cgroup.controllers  cgroup.events  cgroup.procs  cgroup.stat

cgroup.subtree_control  cgroup.threads  cgroup.type

cpu.max  cpu.stat  cpu.weight  memory.current  memory.high

memory.low  memory.max  memory.stat  io.max  pids.max  ...

    

    # # Enable cpu and memory controllers for children of my-app

    # echo "+cpu +memory" > /sys/fs/cgroup/my-app/cgroup.subtree_control

    

    # # Create a leaf cgroup for the actual workload

    # mkdir /sys/fs/cgroup/my-app/worker

Configure Limits and Move a Process

console


    # # Set CPU limit: max 50% of one CPU (50ms per 100ms period)

    # echo "50000 100000" > /sys/fs/cgroup/my-app/worker/cpu.max

    

    # # Set memory limit: 256 MB hard cap

    # echo 268435456 > /sys/fs/cgroup/my-app/worker/memory.max

    

    # # Move process PID 12345 into this cgroup

    # echo 12345 > /sys/fs/cgroup/my-app/worker/cgroup.procs

    

    # # Verify the process is in the cgroup

    # cat /proc/12345/cgroup

    0::/my-app/worker

Monitor Resource Usage

console


    # cat /sys/fs/cgroup/my-app/worker/memory.current

    41943040

    

    # # That is 40 MB (41943040 / 1024 / 1024)

    

    # cat /sys/fs/cgroup/my-app/worker/cpu.stat

    usage_usec 1523456

user_usec 1200000

system_usec 323456

nr_periods 1542

nr_throttled 23

throttled_usec 450000

Note

Note: nr_throttled tells you how many times the cgroup was throttled due to hitting its cpu.max limit, and throttled_usec tells you the total time spent throttled. These are the first metrics to check when debugging container CPU performance.

Clean Up

console


    # # Move processes out first (to root cgroup or another cgroup)

    # echo 12345 > /sys/fs/cgroup/cgroup.procs

    

    # # Remove cgroups (must be empty — no processes, no children)

    # rmdir /sys/fs/cgroup/my-app/worker

    # rmdir /sys/fs/cgroup/my-app

Warning

You cannot rm -rf a cgroup directory. The cgroup virtual filesystem only supports rmdir, and the cgroup must be empty (no processes and no children) before removal. If a process is still running inside, you must move it out first.

What cgroups Are

What cgroups Control

The cgroup Virtual Filesystem

cgroups v1 vs v2

v2 Mechanics

Core Control Files

Enabling Controllers for Children

Check available

Enable for children

Create child

Configure child

The "No Internal Processes" Rule

Delegation for Rootless Containers

How systemd Enables Delegation

Hands-On: Working with cgroups v2

Check Available Controllers

Check What Is Enabled for Children

Create a Child cgroup and Enable Controllers

Configure Limits and Move a Process

Monitor Resource Usage

Clean Up