Networking6 min read

Services

Service discovery, load balancing, and the kube-proxy implementation

Why Services Exist

Pod IPs are ephemeral. Every time a Pod restarts, gets rescheduled, or scales, it gets a new IP. You cannot hardcode Pod IPs in your application config. Services solve this by providing:

Stable IP

A virtual IP (ClusterIP) that never changes for the lifetime of the Service, regardless of how many Pods come and go behind it.

Stable DNS

A predictable DNS name: my-svc.my-ns.svc.cluster.local. Applications connect by name, DNS resolves to ClusterIP.

Load Balancing

Traffic to the Service is distributed across all healthy Pod endpoints — no client-side balancing logic needed.

Service Types

ClusterIP (default)

ClusterIP: Internal-only virtual IP

Allocates a VIP from the service CIDR (e.g., 10.96.0.0/12, set via --service-cluster-ip-range)
This IP is virtual — it is not assigned to any network interface on any node
It exists only as iptables/IPVS rules programmed by kube-proxy on every node
When a Pod sends traffic to ClusterIP:port, kube-proxy's rules DNAT the packet to a real Pod IP:port

yaml

apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: production
spec:
  type: ClusterIP          # default — can be omitted
  selector:
    app: my-app             # selects pods with this label
  ports:
    - port: 80              # port exposed on the ClusterIP
      targetPort: 8080      # port the Pod is listening on
      protocol: TCP

Note

DNS: CoreDNS creates an A record: my-service.production.svc.cluster.local -> 10.96.x.x. Within the same namespace, Pods can use just my-service. Cross-namespace: my-service.production.

NodePort

NodePort: Expose on every node's IP

Allocates a port from the range 30000-32767 on every node in the cluster
External clients connect to <any-node-IP>:<NodePort>
kube-proxy on the receiving node DNATs to a backend Pod (may be on a different node)
Also creates a ClusterIP — so it's accessible internally too

yaml

spec:
  type: NodePort
  selector:
    app: my-app
  ports:
    - port: 80              # ClusterIP port
      targetPort: 8080      # Pod port
      nodePort: 31234       # optional — auto-assigned if omitted

External Client
192.168.1.10:31234
kube-proxy (any node)
DNAT to Pod IP:8080
Backend Pod
10.244.2.5:8080

Warning

Production warning: NodePort is rarely used directly in production. The high port range (30000+) is awkward. Use LoadBalancer or Ingress instead. NodePort is the building block underneath them.

LoadBalancer

LoadBalancer: Cloud-provisioned external LB

The cloud controller manager (CCM) provisions an external load balancer (AWS ELB/NLB, GCP LB, Azure LB)
The external LB receives traffic and forwards it to the NodePort on cluster nodes
Includes a NodePort + ClusterIP (it's a superset)

yaml

spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
    - port: 443
      targetPort: 8443

Internet Client
a]b-1234.elb.amazonaws.com
Cloud LB
External IP
NodePort
Any node:3xxxx
Pod
10.244.x.x:8443

externalTrafficPolicy

Policy	Behavior	Trade-off
`Cluster` (default)	Any node can receive, kube-proxy may forward to a pod on a different node	Even distribution but extra hop; source IP is SNAT'd (lost)
`Local`	LB only sends to nodes that have a local Pod; no second hop	Preserves source IP; uneven distribution if pods aren't evenly spread

ExternalName

ExternalName: DNS-only alias

No proxy, no ClusterIP, no endpoints — purely a DNS CNAME record
CoreDNS returns: my-service.ns.svc.cluster.local CNAME external-db.example.com
Use case: abstract external dependencies behind a K8s Service name so your app config doesn't change if you migrate the external DB to in-cluster

console

spec:
  type: ExternalName
  externalName: db.prod.example.com
  # No selector, no ports, no ClusterIP

Service Types Summary

Type	ClusterIP	NodePort	External LB	DNS	Use Case
ClusterIP	Yes	No	No	A record	Internal service-to-service
NodePort	Yes	Yes (30000-32767)	No	A record	Dev/testing, bare-metal expose
LoadBalancer	Yes	Yes	Yes	A record	Production external access
ExternalName	No	No	No	CNAME	DNS alias for external services

How kube-proxy Works

kube-proxy is the component that makes Services actually work. It watches the API server for Service and Endpoint changes, then programs the data plane on each node.

iptables Mode (Default)

kube-proxy writes iptables rules in the nat table. Here's the chain structure for a Service with 3 endpoints:

console

# KUBE-SERVICES: entry point — match on ClusterIP + port
-A KUBE-SERVICES -d 10.96.0.100/32 -p tcp --dport 80 \
   -j KUBE-SVC-XXXX

# KUBE-SVC-XXXX: probability-based load balancing
-A KUBE-SVC-XXXX -m statistic --mode random --probability 0.33333 \
   -j KUBE-SEP-AAA
-A KUBE-SVC-XXXX -m statistic --mode random --probability 0.50000 \
   -j KUBE-SEP-BBB
-A KUBE-SVC-XXXX \
   -j KUBE-SEP-CCC

# KUBE-SEP-xxx: DNAT to actual Pod IP
-A KUBE-SEP-AAA -p tcp -j DNAT --to-destination 10.244.1.5:8080
-A KUBE-SEP-BBB -p tcp -j DNAT --to-destination 10.244.2.3:8080
-A KUBE-SEP-CCC -p tcp -j DNAT --to-destination 10.244.3.7:8080

Note

Probability math: The probabilities are 1/n, 1/(n-1), ..., 1/1. For 3 endpoints: first rule matches with 1/3, second with 1/2 of remaining (= 1/3 total), third with 1/1 of remaining (= 1/3 total). Uniform distribution.

Warning

Scale problem: iptables rules are evaluated linearly — O(n). With 10,000 services x 10 endpoints = 100,000+ rules. Every packet traverses this list. At scale, you see increased latency, CPU overhead, and slow rule updates (full rewrite on every change). This is why IPVS exists.

IPVS Mode

IPVS (IP Virtual Server) is a Linux kernel L4 load balancer. kube-proxy can use it instead of iptables for the data plane.

console

# Enable IPVS mode in kube-proxy config
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  scheduler: "rr"  # round-robin (default), lc, sh, etc.

iptables vs IPVS: Comparison

IPTABLES MODE

Lookup: O(n) — linear chain traversal
Algorithm: Random probability only
Rule updates: Full rewrite of all rules
Connection tracking: Via conntrack
Scale: Degrades beyond ~5,000 services
Kernel module: netfilter (always present)
Debugging: iptables-save | grep KUBE

IPVS MODE

Lookup: O(1) — hash table in kernel
Algorithms: rr, lc (least conn), sh (source hash), dh, sed, nq
Rule updates: Incremental (add/remove single entry)
Connection tracking: Built-in + conntrack
Scale: Handles 10,000+ services efficiently
Kernel module: Requires ip_vs, ip_vs_rr, etc.
Debugging: ipvsadm -Ln

Tip

Recommendation: For clusters with more than a few hundred services, switch to IPVS mode. It's been stable since Kubernetes 1.11+. The only caveat: ensure IPVS kernel modules are loaded on all nodes (ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, nf_conntrack).

Endpoint Slices

The original Endpoints resource stored all pod IPs for a service in a single object. For a service with 5,000 pods, that's one massive object that gets fully rewritten and broadcast to all nodes on every pod change.

ENDPOINTS (LEGACY)

Single object per Service
Contains ALL pod IPs
Full object rewrite on any change
Etcd object size limit: 1.5MB
Watch storm: every node gets full update

ENDPOINT SLICES

Multiple slices per Service (max 100 endpoints each)
Only the affected slice is updated
Smaller watch payloads
Supports dual-stack (IPv4 + IPv6)
Topology hints for zone-aware routing

Note

Note: EndpointSlices are the default since Kubernetes 1.21. The legacy Endpoints resource still exists for backward compatibility but EndpointSlice is what kube-proxy actually consumes.

Headless Services

A headless service has clusterIP: None. No virtual IP is allocated, and DNS behaves differently.

yaml

apiVersion: v1
kind: Service
metadata:
  name: my-headless-svc
spec:
  clusterIP: None           # This makes it headless
  selector:
    app: my-stateful-app
  ports:
    - port: 5432

Headless DNS Behavior

Aspect	Normal ClusterIP Service	Headless Service
DNS A record	Returns single ClusterIP (10.96.x.x)	Returns all Pod IPs (multiple A records)
Load balancing	kube-proxy (iptables/IPVS)	Client-side (DNS round-robin or app logic)
kube-proxy involvement	Yes — programs DNAT rules	None

StatefulSet + Headless Service

When combined with a StatefulSet, each Pod gets a stable, predictable DNS name:

console

# StatefulSet "postgres" with headless service "postgres-svc"
postgres-0.postgres-svc.default.svc.cluster.local  -> 10.244.1.5
postgres-1.postgres-svc.default.svc.cluster.local  -> 10.244.2.3
postgres-2.postgres-svc.default.svc.cluster.local  -> 10.244.3.8

This gives you stable network identity — even if a Pod restarts on a different node, its DNS name stays the same. Essential for databases, Kafka brokers, etcd, ZooKeeper — anything that needs peer discovery.

Key Takeaways

Services

ClusterIP: internal VIP, the building block
NodePort: ClusterIP + port on all nodes
LoadBalancer: NodePort + cloud LB
ExternalName: DNS CNAME only
Headless: DNS returns Pod IPs directly

Data Plane

kube-proxy programs iptables or IPVS on every node
iptables: O(n), probability-based — fine up to ~5K services
IPVS: O(1), multiple LB algorithms — use at scale
EndpointSlices replaced Endpoints for efficiency
externalTrafficPolicy: Local preserves source IP

Why Services Exist

Service Types

ClusterIP (default)

NodePort

External Client

kube-proxy (any node)

Backend Pod

LoadBalancer

Internet Client

Cloud LB

NodePort

Pod

ExternalName

Service Types Summary

How kube-proxy Works

iptables Mode (Default)

IPVS Mode

iptables vs IPVS: Comparison

Endpoint Slices

Headless Services

Key Takeaways