CPU Rings & Traps
How CPUs protect themselves — and why virtualizing x86 was a 20-year puzzle
1. Protection Rings (Ring 0–3)
x86 CPUs implement hardware-enforced privilege levels called protection rings.
They are numbered 0 through 3, where lower numbers mean higher privilege.
The CPU checks the Current Privilege Level (CPL) stored in the CS register
before executing any instruction that touches hardware or system state.
Ring Privileges at a Glance
| Ring | Name | Who Runs Here | Can Do |
|---|---|---|---|
| Ring 0 | Kernel mode | OS kernel, drivers | Everything — direct hardware I/O, modify page tables, disable interrupts |
| Ring 1 | OS services | Historically unused on modern OSes | Originally for device drivers (OS/2 used it) |
| Ring 2 | OS services | Historically unused on modern OSes | Originally for privileged utilities |
| Ring 3 | User mode | Applications, shells, browsers | Compute, read own memory — must syscall for anything else |
Note
In practice, modern OSes (Linux, Windows, macOS) use only Ring 0 and Ring 3. Rings 1 and 2 are architecturally present but ignored — the OS puts everything privileged in Ring 0 and everything else in Ring 3. This two-ring model is sometimes called "kernel mode vs. user mode."
Ring Nesting Diagram
Think of rings as concentric protection boundaries. Inner rings have strictly more power than outer ones.
Full hardware access: page tables, I/O ports, interrupt control, MSRs
Applications execute here. Attempting a privileged instruction triggers a #GP (General Protection) fault — the CPU traps to Ring 0.
nginx
python3
bash
chrome
Privileged vs. Sensitive Instructions
These two categories are the key to understanding the virtualization problem. They sound similar but are architecturally distinct:
Definition: Instructions that trap (cause a #GP fault) when executed outside Ring 0.
Examples:
HLT— halt the CPULGDT— load Global Descriptor Table registerMOV to CR3— change page table baseCLI / STI— disable / enable interruptsWRMSR— write Model-Specific Register
Tip
Good for virtualization: these trap automatically, so a VMM can intercept and emulate them.
Definition: Instructions that read or modify machine state (reveal privileged information or change system behavior).
Examples:
SGDT— store GDT register (reveals kernel state)SIDT— store IDT register (reveals kernel state)PUSHF— push FLAGS (includes IF bit — interrupt state)POPF— pop FLAGS (silently ignores IF in Ring 3)SMSW— store Machine Status Word
Warning
Bad for virtualization: some of these execute silently in Ring 3 without trapping — the "x86 hole."
2. The Fundamental Virtualization Problem (Popek-Goldberg)
The 1974 Theorem (Popek & Goldberg)
In their seminal 1974 paper, Popek and Goldberg defined the formal requirements for a Virtual Machine Monitor (VMM). The key theorem:
"For any conventional third-generation computer, a VMM may be constructed if the set of sensitive instructions is a subset of the set of privileged instructions."
In plain English: every instruction that can reveal or affect machine state must trap when executed by a guest OS. If it traps, the VMM catches it and emulates it. If it silently succeeds, the VMM is blind to it — and correctness breaks.
The x86 Virtualization Hole
x86 violated the Popek-Goldberg theorem. There are ~17 instructions that are sensitive but not privileged. They execute in Ring 3 without trapping, but they reveal or affect Ring 0 state:
The Problematic Instructions
| Instruction | What It Does | Why It Breaks Virtualization |
|---|---|---|
SGDT |
Stores the GDT register to memory | Guest reads the host's GDT address, not its own — breaks isolation |
SIDT |
Stores the IDT register to memory | Guest sees host's IDT, can fingerprint the VMM |
SLDT |
Stores the LDT register to memory | Leaks host LDT selector |
SMSW |
Stores Machine Status Word (CR0 low bits) | Reveals real protection/paging mode |
PUSHF / POPF |
Push/pop FLAGS register | PUSHF exposes the real IF (interrupt flag); POPF silently ignores IF changes in Ring 3 |
LAR, LSL, VERR, VERW |
Segment descriptor queries | Expose host segment configuration |
Warning
The practical consequence: you cannot simply run a guest OS in Ring 3 and rely on traps
to catch everything. A guest Linux kernel calling SGDT would silently get the host's GDT
address — it wouldn't trap, and the VMM would never know it happened. The guest kernel
would get confused or, worse, an attacker could use this to detect or escape the VM.
Visualizing the Problem
3. Software Solutions (Working Around the Hole)
Before hardware support arrived in 2005–2006, three major approaches were developed to work around the x86 virtualization hole. Each makes a different trade-off.
3a. Trap-and-Emulate (Classical Approach)
How It Works
- Run the guest OS in a lower privilege ring (e.g., Ring 1 or Ring 3).
- When the guest executes a privileged instruction, the CPU traps (#GP fault).
- The VMM (running in Ring 0) catches the trap and emulates the instruction on behalf of the guest.
- The VMM returns control to the guest, which continues unaware.
Warning
Why it fails on x86: The ~17 sensitive-but-unprivileged instructions don't trap. They execute silently in Ring 3, so the VMM never gets a chance to intercept them. Pure trap-and-emulate is not possible on x86.
3b. Binary Translation (VMware, 1999)
VMware's Breakthrough
VMware's insight: if the hardware won't trap on those instructions, rewrite the guest code before it runs. The VMM scans each basic block of guest kernel code and replaces problematic instructions with calls into the VMM.
- Scan: Before executing a block of guest kernel code, the VMM inspects it instruction by instruction.
- Translate: Replace sensitive instructions (
SGDT,PUSHF, etc.) with safe equivalents or VMM hypercalls. - Cache: Store translated blocks in a translation cache for reuse (avoids re-scanning hot paths).
- Execute: Run the translated code. Safe instructions run natively at full speed; only problematic ones are replaced.
- No guest OS modification required — runs unmodified Windows, Linux, etc.
- User-mode code (Ring 3) runs natively at full speed — only kernel code needs translation
- Translation cache amortizes cost over time
- ~5–20% overhead on kernel-heavy workloads (syscall-heavy apps, I/O)
- Complexity — the translator is essentially a JIT compiler
- Edge cases: self-modifying code, code in read-only pages, etc.
3c. Paravirtualization (Xen, 2003)
A Different Philosophy
Instead of hiding virtualization from the guest, tell the guest it's virtualized and give it explicit APIs to talk to the hypervisor. The guest OS is modified to replace sensitive instructions with hypercalls — explicit calls into the VMM, analogous to syscalls but guest-to-hypervisor instead of app-to-kernel.
// App in Ring 3 needs to write to a file
// Uses syscall to transition to Ring 0
write(fd, buf, len);
// Triggers: SYSCALL instruction
// CPU: Ring 3 → Ring 0
// Kernel handles the I/O
// Returns to Ring 3// Guest kernel needs to update page tables
// Instead of MOV to CR3 (would trap or fail),
// calls the hypervisor directly
HYPERVISOR_mmu_update(reqs, count);
// Triggers: special trap (e.g., VMCALL)
// CPU: Guest → Hypervisor
// Hypervisor handles page table update
// Returns to guest- Near-native performance — no binary translation overhead
- Fewer VM exits than trap-and-emulate (batch operations via hypercalls)
- Guest is aware of virtualization — can cooperate (e.g., balloon drivers)
- Requires guest OS source code modification — can't run unmodified Windows
- Porting effort for each OS
- Guest is tied to a specific hypervisor's hypercall ABI
Software Solutions Compared
Side-by-Side Summary
| Approach | Guest Modified? | Handles x86 Hole? | Overhead | Key Product |
|---|---|---|---|---|
| Trap-and-Emulate | No | No (fails on x86) | Low (on compliant CPUs) | IBM System/370 |
| Binary Translation | No | Yes | 5–20% (kernel code) | VMware Workstation (1999) |
| Paravirtualization | Yes | Yes (avoids them) | <5% | Xen (2003) |
4. Hardware-Assisted Virtualization (VT-x / AMD-V)
In 2005–2006, Intel and AMD finally fixed the x86 virtualization hole in silicon. Instead of patching around the problem in software, they added new CPU operating modes designed specifically for virtualization.
Intel VT-x (2005) / AMD-V (2006)
Both introduce two new CPU modes that exist orthogonally to the Ring 0–3 privilege levels:
Who runs here: The hypervisor / VMM
Privilege: Full control of hardware. Can configure which guest operations cause VM exits.
Ring 0 in root mode = traditional Ring 0 with VMM superpowers.
Who runs here: The guest OS and its applications
Privilege: Guest kernel runs in Ring 0 of non-root mode — it thinks it has full control, but sensitive operations trigger VM exits.
Ring 0 in non-root mode = looks like Ring 0, but the CPU silently intercepts configured operations.
Tip
Key insight: The guest kernel runs in Ring 0 — its own Ring 0. It doesn't need to be deprivileged to Ring 1 or Ring 3. The CPU's non-root mode automatically traps sensitive operations regardless of the ring level. This eliminates the Popek-Goldberg hole entirely.
Nesting Diagram: VMX Modes + Rings
Hypervisor (KVM, ESXi, Hyper-V)
Guest Linux / Windows kernel — thinks it has full control
Guest applications (nginx, python, etc.)
VMCS — Virtual Machine Control Structure
The Per-vCPU Data Structure
Each virtual CPU has its own VMCS — a 4 KB hardware-managed data structure in memory. The CPU reads from and writes to the VMCS during VM entries and exits. It contains three main areas:
Guest-State Area
Saved/restored on VM entry/exit:
- General-purpose registers
CR0,CR3,CR4CS,SS,DS(segments)GDTR,IDTRRIP,RSP,RFLAGS
Host-State Area
Loaded on VM exit (return to VMM):
- Host
CR0,CR3,CR4 - Host segment selectors
- Host
RIP(VMM entry point) - Host
RSP - Host
GDTR,IDTR
VM-Execution Controls
Configure what triggers VM exits:
- Pin-based (external interrupts, NMIs)
- Processor-based (HLT, I/O, CR access, MSR access)
- Exception bitmap (which exceptions cause exits)
- I/O bitmap
- MSR bitmap
Note
VMCS access: The hypervisor reads/writes VMCS fields using special instructions
VMREAD and VMWRITE — it's not a simple memory-mapped struct. The CPU
manages the internal format, and it can vary between CPU generations.
VM Entry / VM Exit Cycle
The VM Entry → Guest Execution → VM Exit cycle is the heartbeat of hardware-assisted virtualization. Every vCPU spends its life oscillating between these states:
Hypervisor (VMX Root)
VMM prepares VMCS, decides which guest to run
VMLAUNCH / VMRESUME
VM Entry: CPU loads guest state from VMCS, switches to non-root mode
Guest Runs (VMX Non-Root)
Guest kernel + apps execute natively at near-full speed
VM Exit (Trap)
Sensitive op, I/O, interrupt, or timer → CPU saves guest state to VMCS, returns to root mode
VMM Handles Exit
Emulates the operation, updates VMCS, then VMRESUME back to step 3
Common VM Exit Reasons
| Exit Reason | Trigger | VMM Action |
|---|---|---|
EXIT_REASON_CPUID |
Guest executes CPUID |
Return virtualized CPU feature flags |
EXIT_REASON_IO_INSTRUCTION |
Guest does port I/O (IN/OUT) |
Emulate the I/O device |
EXIT_REASON_CR_ACCESS |
Guest writes to CR0, CR3, CR4 |
Validate and apply (e.g., update shadow page tables) |
EXIT_REASON_MSR_WRITE |
Guest writes an MSR | Emulate or pass through |
EXIT_REASON_HLT |
Guest executes HLT |
Schedule another vCPU or idle |
EXIT_REASON_EPT_VIOLATION |
Guest accesses unmapped guest-physical memory | Handle memory mapping (lazy allocation, MMIO emulation) |
EXIT_REASON_EXTERNAL_INTERRUPT |
Hardware interrupt arrives for the host | Return to host to handle the interrupt |
Warning
VM exits are expensive: Each exit costs ~1,000–3,000 CPU cycles (context switch between root and non-root mode, VMCS state save/restore). Minimizing unnecessary exits is a major optimization target. This is why the VMCS has fine-grained control bitmaps — you only trap on the operations you actually need to intercept.
5. Extended Page Tables (EPT / NPT)
Memory virtualization is one of the most performance-critical aspects of running a VM. The core challenge: the guest OS manages its own page tables, but those tables map to guest-physical addresses — not real hardware addresses.
The Two-Level Address Translation Problem
Address Spaces in a VM
Guest Virtual Address (GVA) What guest applications use → Guest page table Guest Physical Address (GPA) What the guest kernel thinks is "physical memory"
Guest Physical Address (GPA) An abstraction managed by the hypervisor → EPT / NPT Host Physical Address (HPA) Actual DRAM address on the hardware
Full translation path: GVA → (guest page table) → GPA → (EPT/NPT) → HPA
Before EPT: Shadow Page Tables
The VMM maintains a shadow copy of the guest's page tables that maps directly from GVA → HPA.
- VMM intercepts every guest page table write (trap on
MOV to CR3) - VMM walks the guest page table, translates each GPA to HPA, writes shadow entries
- Hardware MMU uses the shadow tables (guest never knows)
- Every guest page fault = VM exit + VMM handling
- Every guest
CR3switch (context switch) = VM exit + full shadow rebuild
Warning
Cost: Extremely high VM exit rate for memory-intensive workloads. A guest context switch that takes ~1 μs natively can take 10–50 μs with shadow page tables.
The CPU hardware walks both levels of page tables automatically — no VMM intervention needed.
- Guest manages its own page tables (GVA → GPA) — no trapping
- VMM maintains EPT (GPA → HPA) — set up once, rarely changed
- Hardware MMU walks both tables in a single operation
- Guest page faults handled by guest kernel (no VM exit)
- Guest
CR3switches are free (no VM exit)
Tip
Result: 5–10x fewer VM exits for memory-intensive workloads. Guest context switches run at near-native speed.
The Two-Dimensional Page Walk
With EPT, a single guest memory access triggers a 2D page walk. The hardware walks the guest page table (4 levels on x86-64), but at each level it also needs to translate the guest-physical address of the next page table entry through the EPT (another 4 levels).
Worst-Case Walk Depth
For a 4-level guest page table with 4-level EPT, a single TLB miss can require up to:
4 × 4 + 4 = 20 memory accesses
(4 guest page table levels × 4 EPT levels each, plus 4 EPT levels for the final data page)
This is why the TLB (Translation Lookaside Buffer) is even more critical in VMs. Modern CPUs have:
- VPID (Virtual Processor ID) — tags TLB entries per-vCPU so they survive VM exits (no TLB flush on every exit)
- Large page support in EPT — 2 MB and 1 GB EPT pages reduce walk depth
- EPT-specific TLB entries — cached GPA → HPA translations
EPT Performance Impact
Real-World Numbers (approximate)
| Metric | Shadow Page Tables | EPT / NPT |
|---|---|---|
| VM exits per second (memory-heavy workload) | 50,000–200,000 | 1,000–5,000 |
| Guest context switch overhead | 10–50 μs | 1–3 μs |
| TLB miss penalty | ~4 memory accesses (uses shadow tables directly) | Up to 20 accesses (2D walk, mitigated by TLB/VPID) |
| Overall memory-intensive workload overhead | 15–40% | 2–5% |
Tip
Bottom line: EPT/NPT trades a slower TLB miss path (20 vs 4 memory accesses) for dramatically fewer VM exits. Since TLB hit rates are typically >99%, this is an overwhelmingly good trade. EPT is why modern VMs can run memory-intensive workloads with barely measurable overhead.
6. The Full Picture: Modern CPU Virtualization Stack
Note
Timeline recap: 1974 — Popek-Goldberg theorem published. 1999 — VMware ships binary translation (first practical x86 virtualization). 2003 — Xen ships paravirtualization. 2005 — Intel VT-x released. 2006 — AMD-V released. 2008 — Intel EPT / AMD NPT released. Today — hardware-assisted virtualization with EPT is the universal standard.
