cgroups in Practice (Docker & K8s)

How container platforms translate user-facing flags to cgroup files

Docker Flag to cgroup File Mapping

Every Docker resource flag is a thin wrapper around writing a value to a cgroup file. There is no magic — Docker computes the value, writes it to the correct cgroup file, and the kernel enforces it.

CPU Mappings

--cpus 2 --> cpu.max = 200000 100000

--cpus 0.5 --> cpu.max = 50000 100000

--cpu-shares 512 --> cpu.weight ~= 50 (converted: shares/1024*100)

--cpuset-cpus "0,1" --> cpuset.cpus = 0,1

Memory Mappings

--memory 512m --> memory.max = 536870912 (512 * 1024 * 1024)

--memory-reservation 256m --> memory.low = 268435456

--memory-swap 1g --> memory.swap.max = 536870912 (1G total - 512M RAM = 512M swap)

I/O and PIDs Mappings

--pids-limit 100 --> pids.max = 100

--device-read-bps /dev/sda:10mb --> io.max = 8:0 rbps=10485760

--device-write-iops /dev/sda:1000 --> io.max = 8:0 wiops=1000

cgroup Driver: systemd vs cgroupfs

Container runtimes need to create cgroup directories and write to files. There are two approaches to managing the cgroup hierarchy:

# cgroupfs path
/sys/fs/cgroup/docker/
    <container-id>/
        cpu.max
        memory.max

# systemd path
/sys/fs/cgroup/system.slice/
    docker-<container-id>.scope/
        cpu.max
        memory.max

Kubernetes Resource Model

Kubernetes has its own abstraction layer over cgroups: requests and limits. These map directly to cgroup files, but serve different purposes.

Requests vs Limits

docker run \
  --cpus 2 \
  --memory 512m \
  my-app

resources:
  requests:
    cpu: "500m"
    memory: "256Mi"
  limits:
    cpu: "2"
    memory: "512Mi"

K8s Resource to cgroup File Mapping

QoS Classes

Kubernetes classifies every pod into one of three Quality of Service classes based on how requests and limits are configured. This classification determines eviction priority when the node runs out of resources.

resources: {}  # nothing specified

resources:
  requests: { cpu: "250m", memory: "128Mi" }
  limits:   { cpu: "1",    memory: "512Mi" }

resources:
  requests: { cpu: "1", memory: "512Mi" }
  limits:   { cpu: "1", memory: "512Mi" }

Pod cgroup Hierarchy

Kubernetes organizes cgroups in a two-level hierarchy: pod-level cgroups enforce total pod limits, and container-level cgroups enforce per-container limits within the pod.

ResourceQuota & LimitRange

These are admission-time controls — they are enforced by the Kubernetes API server when pods are created, not at runtime by the kernel. They determine which cgroup values kubelet will configure.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"
    pods: "50"

apiVersion: v1
kind: LimitRange
metadata:
  name: container-limits
spec:
  limits:
  - type: Container
    default:
      cpu: "500m"
      memory: "256Mi"
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
    max:
      cpu: "2"
      memory: "1Gi"

1. kubectl apply

   User submits pod spec

2. API Server

   LimitRange injects defaults, ResourceQuota checks totals

3. Scheduler

   Places pod based on requests

4. kubelet

   Creates cgroups with the computed values

HPA and VPA

Autoscalers use cgroup accounting data as their input signal. The entire metrics pipeline is built on reading cgroup files.

The Metrics Pipeline

1. cgroup Files

   cpu.stat, memory.current, etc.

2. kubelet

   Reads cgroup files, exposes /stats/summary

3. metrics-server

   Scrapes kubelet, aggregates into metrics API

4. HPA / VPA

   Reads metrics, adjusts replicas or resources

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
spec:
  targetRef:
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - minAllowed:
        cpu: "100m"
      maxAllowed:
        cpu: "4"

The Full Picture: From User Config to Kernel Enforcement

Here is the complete chain from a Kubernetes manifest to kernel-level enforcement:

1. User writes a pod spec with resources.requests and resources.limits
2. API server admission: LimitRange injects defaults if missing. ResourceQuota checks namespace totals. Pod is rejected or accepted.
3. Scheduler places the pod on a node with enough allocatable resources (based on requests, not limits).
4. kubelet on the target node asks the container runtime (containerd/CRI-O) to create the container.
5. Container runtime creates cgroups (via systemd or direct cgroupfs) and writes the computed values to cpu.max, cpu.weight, memory.max, memory.low, pids.max, etc.
6. Linux kernel enforces the limits at the CFS scheduler (CPU), memory allocator (memory), and block I/O layer (I/O). This is all kernel code — no userspace overhead.
7. kubelet reads cgroup accounting files (cpu.stat, memory.current) and exposes them via /stats/summary.
8. metrics-server and HPA/VPA consume these metrics to make scaling decisions, which loop back to step 1 (new pod specs with adjusted resources).