Skip to content
Book a call
Menu

Networking6 min read

Pod Networking

How every Pod gets a unique, routable IP

Kubernetes Networking Requirements (The 3 Rules)

The Kubernetes networking model is built on three non-negotiable constraints. Every CNI plugin must satisfy all three — there is no wiggle room.

The Three Fundamental Rules

  1. Every Pod gets a unique, cluster-wide IP address
    No two Pods share an IP. Containers within a Pod share the same network namespace (and IP), but each Pod is distinct. This is the flat networking model — no port-mapping hacks.
  2. Pods on any node can communicate with all Pods on any other node without NAT
    Pod A on Node 1 can reach Pod B on Node 3 using Pod B's real IP. No SNAT, no DNAT, no port translation. The source IP seen by the destination is the real Pod IP.
  3. Agents on a node (kubelet, system daemons) can reach all Pods on that node
    The kubelet must be able to health-check pods, pull metrics, and stream logs. System-level daemons (node exporter, log collectors) need direct Pod access.

Note

Note: These rules intentionally exclude Services. The Service abstraction (ClusterIP, kube-proxy) is a separate layer built on top of the pod networking model. The CNI only needs to deliver the three rules above.

The Pause Container: Network Namespace Anchor

Every Pod contains a hidden infrastructure container that you rarely see but is critical to how networking works.

What is the pause container?

  • Image: registry.k8s.io/pause:3.9 (previously gcr.io/google_containers/pause)
  • It runs a single instruction: pause() — literally does nothing, sleeps forever
  • Its sole purpose is to hold the network namespace for the Pod
  • All app containers in the Pod join the pause container's network namespace via --net=container:pause

  1. kubelet creates Pod

    Calls CRI (containerd)

  2. Pause container starts

    Creates network namespace

  3. CNI plugin called

    Assigns IP, creates veth pair

  4. App containers start

    Join pause's netns

Tip

Why this matters: If an app container crashes and kubelet restarts it, the network namespace persists because the pause container is still running. The Pod keeps its IP, its veth pair, its network identity. This is why you can restart containers without breaking connections.

Containers within a Pod share:

Shared Not Shared
IP address (same eth0) Filesystem (separate rootfs)
Port space (container A on :8080 blocks container B from :8080) Process namespace (by default)
Loopback (localhost — containers talk via 127.0.0.1) CPU/memory cgroups (separate limits)
Network interfaces Environment variables (per container)

Pod-to-Pod Communication: Same Node

When two Pods are on the same node, traffic stays local and flows through a Linux bridge via veth pairs.

How veth Pairs Work

A veth (virtual ethernet) pair is a tunnel with two ends. One end sits in the Pod's network namespace (as eth0), the other end sits in the host namespace and is plugged into a bridge.

Node 1
Pod A
10.244.1.2
eth0
Pod B
10.244.1.3
eth0
vethA
vethB
cbr0 (bridge)
10.244.1.1/24
eth0 (host NIC)
192.168.1.10

Traffic Flow: Pod A -> Pod B (Same Node)

  1. Pod A eth0

    10.244.1.2

  2. vethA

    Host namespace

  3. cbr0 bridge

    L2 MAC lookup

  4. vethB

    Host namespace

  5. Pod B eth0

    10.244.1.3

Note

Key detail: The bridge learns MAC addresses just like a physical switch. Pod A ARPs for Pod B's MAC, the bridge forwards the frame to the correct veth. No routing needed — it's pure L2 switching.

Pod-to-Pod Communication: Across Nodes

When pods live on different nodes, the packet must leave one node and reach the other. There are two fundamentally different approaches.

Approach 1: Overlay Networks (VXLAN Encapsulation)

Used by Flannel, Weave, and Calico in VXLAN mode. The original pod packet is wrapped inside a new UDP packet for transport across the underlay network.

Original Payload (application data)
Inner TCP/UDP Header (src: Pod A port, dst: Pod B port)
Inner IP Header (src: 10.244.1.2, dst: 10.244.2.5)
VXLAN Header (VNI: 1, 8 bytes)
Outer UDP Header (src: ephemeral, dst: 4789)
Outer IP Header (src: Node 1 IP, dst: Node 2 IP)
Outer Ethernet Frame (src: Node 1 MAC, dst: Node 2 MAC / Gateway MAC)

Warning

Overhead: VXLAN adds ~50 bytes per packet (outer Ethernet 14B + outer IP 20B + outer UDP 8B + VXLAN 8B). This reduces your effective MTU from 1500 to ~1450. If jumbo frames aren't configured, you'll see fragmentation and performance degradation. Set --mtu=1450 in your CNI config or enable jumbo frames on your underlay.

Overlay Network (VXLAN)
Node 1 (192.168.1.10)
Pod A
10.244.1.2
cbr0 bridge
flannel.1
VTEP (VXLAN endpoint)
Underlay
UDP:4789
Encapsulated
Node 2 (192.168.1.11)
Pod B
10.244.2.5
cbr0 bridge
flannel.1
VTEP (VXLAN endpoint)

Approach 2: Native Routing (BGP)

Used by Calico in BGP mode. No encapsulation — nodes announce their Pod CIDRs via BGP, and the infrastructure routes Pod traffic directly.

Native Routing (BGP)
Node 1 (192.168.1.10)
Pod A
10.244.1.2
BIRD BGP agent
Announces: 10.244.1.0/24
ToR / Fabric
BGP peer
Native L3 routing
Node 2 (192.168.1.11)
Pod B
10.244.2.5
BIRD BGP agent
Announces: 10.244.2.0/24

Overlay vs Native Routing

Aspect Overlay (VXLAN) Native Routing (BGP)
Encapsulation Yes — VXLAN (UDP:4789) None
Per-packet overhead ~50 bytes 0 bytes
MTU impact Reduces effective MTU to ~1450 No impact
Infrastructure requirements Just IP connectivity between nodes BGP-capable routers or cloud VPC routes
Performance Good — slight CPU overhead for encap/decap Best — native kernel routing
Debugging Harder — tcpdump shows encapsulated packets Easier — pod IPs visible on wire
CNI examples Flannel, Weave, Calico (VXLAN mode) Calico (BGP mode), Cilium (BGP)
Cloud-native option Works anywhere AWS VPC CNI, GKE native routing

Tip

Cloud shortcut: AWS VPC CNI and GKE's native routing assign Pod IPs directly from the VPC subnet — no overlay, no BGP. Pods get VPC-routable IPs. Best performance, but ties you to the cloud provider and consumes VPC IP space.

IP Address Allocation

Kubernetes uses a hierarchical IP allocation scheme to ensure every Pod gets a unique IP without centralized coordination.

CIDR Hierarchy

  1. Cluster CIDR — The overall range for all pod IPs
    --cluster-cidr=10.244.0.0/16 (set on kube-controller-manager)
    Provides 65,536 addresses across the entire cluster
  2. Node CIDR — Each node gets a subnet carved from the cluster CIDR
    --node-cidr-mask-size=24 (default)
    Node 1: 10.244.1.0/24 (256 IPs) | Node 2: 10.244.2.0/24 | Node 3: 10.244.3.0/24
  3. IPAM (IP Address Management) — Allocates individual IPs from the node's subnet
    CNI plugin's IPAM module (host-local, calico-ipam, etc.) hands out IPs to Pods as they're created

Example: Full Allocation

bash
Cluster CIDR:   10.244.0.0/16
                        |
        +---------------+---------------+
        |               |               |
   Node 1            Node 2           Node 3
  10.244.1.0/24    10.244.2.0/24    10.244.3.0/24
        |               |               |
  Pod: .2, .3, .4  Pod: .2, .3      Pod: .2, .3, .4, .5

Warning

Capacity planning: With /16 cluster CIDR and /24 per node, you get 256 nodes max (2^(24-16) = 256). Each node supports ~253 pods (256 minus network, gateway, broadcast). For larger clusters, use /12 for the cluster CIDR or /25 for node masks. Plan your CIDRs before cluster creation — changing them later requires cluster rebuild.

Key Takeaways

The Model

  • Every Pod gets a real, unique IP — no NAT between pods
  • The pause container anchors the network namespace
  • Same-node traffic: veth pairs + bridge (L2)
  • Cross-node: overlay (VXLAN) or native routing (BGP)

Operational Notes

  • Always configure MTU correctly when using overlays
  • Plan your cluster CIDR and node CIDR sizes early
  • In cloud: consider native VPC CNIs for best performance
  • On-prem: Calico with BGP if your fabric supports it, VXLAN otherwise
Solidnines — solidnines.com