Skip to content
Book a call
Menu

Virtualization7 min read

Control Plane

The brain of the cluster — API server, etcd, scheduler, and controllers

Architecture Overview

A Kubernetes cluster is split into the control plane (the brain) and worker nodes (the muscle). The control plane makes global decisions — scheduling, detecting failures, responding to events — while worker nodes run your actual workloads.

Control Plane Node
⚙ kube-apiserver REST API gateway
🗃 etcd Key-value store
📋 kube-scheduler Pod placement
🔄 controller-manager Reconciliation loops
☁ cloud-controller-mgr Cloud integrations
↕ gRPC / HTTPS ↕
Worker Nodes
Worker Node 1
kubelet Node agent
kube-proxy Network rules
Container Runtime containerd / CRI-O
Worker Node 2
kubelet Node agent
kube-proxy Network rules
Container Runtime containerd / CRI-O

Note

Key insight: All communication between components goes through the API server. The scheduler, controllers, and kubelets never talk to etcd directly — only the API server does.

API Server (kube-apiserver)

The API server is the front door to the cluster. It's the only component that communicates with etcd directly. Every operation — whether from kubectl, a kubelet heartbeat, or a controller reconciliation loop — goes through the API server's RESTful interface.

  • Exposes a RESTful API over HTTPS (port 6443 by default)
  • Stateless — can be horizontally scaled behind a load balancer for HA
  • Every kubectl command, every kubelet heartbeat, every controller loop goes through it
  • Acts as the single gateway to etcd — no other component reads/writes etcd directly

Admission Pipeline

Every API request passes through a strict pipeline before anything is written to etcd. Understanding this pipeline is essential for debugging "why was my resource rejected?" or "how did my pod get a sidecar injected?"

  1. Authentication

    Who are you?
    certs, tokens, OIDC

  2. Authorization

    Can you do this?
    RBAC policies

  3. Mutating Webhooks

    Modify the request
    inject sidecars, add labels

  4. Schema Validation

    Is the object valid?
    field types, required fields

  5. Validating Webhooks

    Accept or reject?
    policy checks, OPA

  6. Write to etcd

    Persisted
    resource stored

Tip

Mutating vs. validating webhooks: Mutating webhooks run before validation — they can change the object (e.g., Istio injecting a sidecar container). Validating webhooks run after validation — they can only accept or reject (e.g., a policy engine blocking privileged containers). Order matters: mutate first, then validate the final result.

Watch Mechanism

The API server supports watch — clients open long-lived HTTP connections and receive chunked responses whenever a watched resource changes. This is the fundamental mechanism behind Kubernetes' reactive architecture.

  • Controllers, kubelets, and other components use watches to react to changes in near-real-time
  • Backed by etcd's own watch feature
  • Each watch has a resourceVersion — enables resumable watches after disconnection
  • API server maintains a watch cache to avoid hammering etcd with redundant watches
  • Informers (client-go) combine watch + in-memory cache + index for efficient access

How a Controller Uses Watch

  1. Controller starts an informer that lists all resources of interest (initial sync)
  2. Informer opens a watch connection to the API server
  3. API server pushes events (ADDED, MODIFIED, DELETED) as they happen
  4. Informer updates its local cache and enqueues the object key onto a work queue
  5. Worker goroutines dequeue keys and run the reconcile function

etcd

etcd is a distributed, consistent key-value store that serves as the single source of truth for all cluster state. If etcd is lost and unrecoverable, the cluster is gone.

Core Properties

  • Strongly consistent (linearizable reads/writes)
  • Uses Raft consensus — requires a quorum (majority) for writes
  • All data is protobuf-encoded for efficiency
  • Supports watches natively — the API server leverages this
  • Typical cluster sizes: 3 nodes (tolerates 1 failure) or 5 nodes (tolerates 2 failures)

Operational Concerns

  • etcd is the cluster's bottleneck — keep write latency under 10ms
  • Use fast SSDs for etcd storage (avoid network-attached storage)
  • Regular compaction to reclaim space from old revisions
  • Regular defragmentation to reclaim freed disk space
  • Back up etcd regularly — etcdctl snapshot save
  • Default DB size limit: 2 GB (configurable up to 8 GB)

Key Organization

etcd stores Kubernetes objects under a hierarchical key structure:

bash
// Key format
/registry/<resource-type>/<namespace>/<name>

// Examples
/registry/pods/default/nginx-deployment-abc123
/registry/services/kube-system/kube-dns
/registry/deployments/production/web-app
/registry/nodes/worker-node-01         // cluster-scoped (no namespace)
/registry/secrets/default/db-credentials

Raft Consensus

★ etcd Leader Handles all writes
↔ replicate
○ etcd Follower Receives log entries
↔ replicate
○ etcd Follower Receives log entries

Note

Quorum: With 3 nodes, quorum = 2. The leader must replicate a write to at least 1 follower before acknowledging. If the leader dies, the remaining 2 nodes elect a new leader. With 5 nodes, quorum = 3 — tolerates 2 simultaneous failures.

Scheduler (kube-scheduler)

The scheduler watches for newly created pods that have no node assigned (.spec.nodeName is empty). It selects the optimal node for each pod through a two-phase process.

  1. Watch

    Detect unscheduled pods
    (.spec.nodeName is empty)

  2. Filter

    Eliminate unsuitable nodes
    resources, taints, affinity

  3. Score

    Rank remaining nodes
    0-100 per plugin

  4. Bind

    Set .spec.nodeName
    kubelet picks it up

Filtering Phase (Predicates)

Nodes that cannot run the pod are eliminated:

  • PodFitsResources — enough CPU/memory available?
  • PodFitsHostPorts — requested host ports free?
  • NodeSelector — node labels match?
  • NodeAffinity — required affinity rules satisfied?
  • TaintToleration — pod tolerates node taints?
  • PodTopologySpread — would placing here violate spread constraints?
  • VolumeZone — PV in same zone as node?

Scoring Phase (Priorities)

Remaining nodes are ranked by weighted scores:

  • LeastRequestedPriority — prefer nodes with most free resources
  • BalancedResourceAllocation — prefer balanced CPU/memory usage
  • NodeAffinityPriority — prefer nodes matching preferred affinity
  • PodTopologySpread — prefer nodes that improve distribution
  • ImageLocality — prefer nodes that already have the container image
  • InterPodAffinity — prefer nodes near preferred co-located pods

Tip

Extensibility: The scheduler is plugin-based. You can write custom scheduler plugins, create multiple scheduler profiles, or run multiple schedulers. Pods can specify .spec.schedulerName to target a specific scheduler.

Controller Manager (kube-controller-manager)

The controller manager runs approximately 30 controllers as goroutines within a single process. Each controller implements the same fundamental pattern: watch resources, compare desired state vs. actual state, and reconcile the difference.

The Reconciliation Loop — Kubernetes' Core Pattern

  1. Observe

    Watch API server for
    resource changes

  2. Diff

    Compare desired state
    vs. actual state

  3. Act

    Take action to converge
    actual towards desired

  4. Repeat

    Continuously loop
    forever

This is the declarative model: you declare what you want (desired state), and controllers work to make reality match. You never say "start 3 pods" — you say "I want 3 replicas" and the controller makes it happen.

Key Controllers

Controller Watches Reconciles
Deployment Deployment objects Creates/updates ReplicaSets to match the deployment spec. Manages rolling updates by scaling new RS up and old RS down.
ReplicaSet ReplicaSet objects, Pods Ensures exactly N pod replicas exist. Creates or deletes pods as needed.
Node Node objects Monitors node health via heartbeats. Taints unresponsive nodes, evicts pods from unreachable nodes after a grace period.
Service Account Namespace objects Creates a default service account in every new namespace.
Endpoint / EndpointSlice Services, Pods Populates Endpoints/EndpointSlice objects with the IPs of pods matching a service's selector.
Job Job objects, Pods Creates pods for jobs, tracks completions, handles failures and retries.
Namespace Namespace objects When a namespace is deleted, cleans up all resources within it.
PersistentVolume PVs, PVCs Binds PVCs to matching PVs. Handles reclaim policies on release.

Warning

Leader election: In HA setups, multiple controller-manager instances may run, but only the leader actively reconciles. Others are standby. Leader election uses a Lease object in the API server. Same applies to the scheduler.

Cloud Controller Manager

Separated from the main controller manager since Kubernetes 1.6, the cloud controller manager runs controllers that integrate with cloud provider APIs. Each cloud provider (AWS, GCP, Azure) ships their own implementation.

Cloud-Specific Controllers

  • Node controller — detects when a cloud VM is terminated and removes the corresponding K8s node object
  • Route controller — configures cloud network routes so pods on different nodes can communicate
  • Service controller — creates cloud load balancers for services of type: LoadBalancer (e.g., AWS ELB, GCP LB)

Why Separate?

  • Kubernetes core remains cloud-agnostic
  • Cloud providers can release on their own schedule, independent of K8s releases
  • Cleaner separation of concerns — cloud APIs evolve independently
  • Different clouds have different capabilities (zones, LB features, network models)

Note

Managed Kubernetes: In managed services (EKS, GKE, AKS), the control plane is fully managed by the cloud provider. You never see the API server, etcd, or controllers — the provider runs and maintains them. You only manage worker nodes (or use managed node pools / Fargate / Autopilot).

Solidnines — solidnines.com