Kubelet & Container Runtime

The node agent — from scheduled pod to running containers

Kubelet Overview

The kubelet is the node-level agent that runs on every node in the cluster. It takes pod specs from the API server and ensures the described containers are running and healthy.

Pod Lifecycle: End-to-End

Follow a Deployment from kubectl apply all the way to a running container on a node. This shows how all control plane components collaborate.

1. kubectl apply

   User submits Deployment
   to API server

2. API Server

   Validates, stores
   Deployment in etcd

3. Deployment Ctrl

   Creates ReplicaSet
   stored in etcd

4. ReplicaSet Ctrl

   Creates Pod (unscheduled)
   stored in etcd

1. Scheduler

   Assigns node to Pod
   sets .spec.nodeName

2. Kubelet

   Picks up Pod via watch
   calls CRI

3. Container Runtime

   Pulls image, creates
   and starts container

4. Running

   Kubelet reports status
   back to API server

CRI in Action

When the kubelet needs to start a pod, it makes a sequence of CRI (Container Runtime Interface) gRPC calls to the container runtime (containerd, CRI-O). Here is the exact sequence:

1. RunPodSandbox — Creates the pod sandbox, which is the "pause" container. This tiny container (runs /pause, an infinite sleep) exists solely to hold the pod's Linux namespaces (network, IPC, PID). All other containers in the pod join these namespaces.
   The CNI plugin is invoked here to set up pod networking — assigns IP, creates veth pair, configures routes.
2. PullImage — Downloads the container image from the registry (if not already cached locally). Image pull policy: Always, IfNotPresent, or Never.
3. CreateContainer — Creates the container within the sandbox. The container shares the network namespace (and optionally PID namespace) with the pause container. OCI runtime spec is generated here.
4. StartContainer — Starts the container process. The OCI runtime (runc, crun, kata) is invoked to actually launch the process with the appropriate namespaces, cgroups, and seccomp profiles.

Init Containers

Init containers run before application containers, one at a time, in order. Each must complete successfully before the next starts. If an init container fails, the kubelet restarts it (subject to the pod's restart policy).

1. Init 1

   e.g., wait for DB
   to be reachable

2. Init 2

   e.g., clone config
   from git repo

3. Init 3

   e.g., run DB
   migrations

4. App Containers

   All start together
   after inits complete

Common use cases:

• Wait for dependencies — poll until a database or service is ready
• Set up configuration — clone config from a repo, generate certs
• Run migrations — apply schema changes before the app starts
• Download artifacts — fetch data files, ML models

# Init container example in pod spec
spec:
  initContainers:
    - name: wait-for-db
      image: busybox:1.36
      command: ['sh', '-c', 'until nc -z postgres 5432; do sleep 2; done']
    - name: run-migrations
      image: myapp:latest
      command: ['python', 'manage.py', 'migrate']
  containers:
    - name: web
      image: myapp:latest

Sidecar Containers (K8s 1.28+)

Before K8s 1.28, sidecars were just regular containers in the pod — no guaranteed startup order, no graceful shutdown order. The native sidecar feature fixes this by using restartPolicy: Always on init containers.

# Native sidecar container (K8s 1.28+)
spec:
  initContainers:
    - name: envoy-proxy
      image: envoyproxy/envoy:v1.28
      restartPolicy: Always    # ← this makes it a sidecar
      ports:
        - containerPort: 15001
  containers:
    - name: app
      image: myapp:latest
      # envoy-proxy is guaranteed running before app starts

Probes

Probes let the kubelet monitor container health and readiness. They determine whether a container should be restarted, receive traffic, or keep starting up.

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 10
  # 30 × 10 = 300s max startup

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 20
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 3

Termination Sequence

Graceful shutdown is critical for zero-downtime deployments. Understanding this exact sequence prevents dropped connections and data loss.

1. Pod deletion requested — API server sets deletionTimestamp on the pod. The pod enters Terminating state.
2. Pod removed from Service endpoints — the Endpoint controller removes the pod's IP from all Services. No new traffic is routed to it. (This happens in parallel with step 3 — race condition risk!)
3. preStop hook runs (if defined) — e.g., drain connections, deregister from service discovery, flush buffers. This is your chance for graceful cleanup.
4. SIGTERM sent — the container's main process (PID 1) receives SIGTERM. Your application should handle this signal: stop accepting new requests, finish in-flight work, close connections.
5. Grace period — default 30 seconds (configurable via terminationGracePeriodSeconds). The kubelet waits for the process to exit cleanly.
6. SIGKILL sent — if the process is still running after the grace period, the kubelet sends SIGKILL (forceful, uncatchable kill).
7. Cleanup — container is removed, pod status is updated to Succeeded or Failed, API server is notified.

# Graceful shutdown configuration
spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: web
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5 && /app/graceful-shutdown.sh"]

Device Plugins

Device plugins extend the kubelet to advertise and manage custom hardware resources — GPUs, FPGAs, InfiniBand NICs, and other accelerators.

# Requesting a GPU in a pod spec
spec:
  containers:
    - name: training
      image: pytorch/pytorch:2.1
      resources:
        limits:
          nvidia.com/gpu: "1"   # request 1 NVIDIA GPU

Cross-References