Skip to content
Book a call
Menu

Virtualization11 min read

Hypervisors (KVM, QEMU, virtio)

The software layer that makes virtual machines possible

Type 1 vs Type 2 Hypervisors

Two Fundamental Architectures

Every hypervisor falls into one of two categories based on where it sits relative to the hardware. The distinction determines overhead, performance, and use case.

Type 1 — Bare-Metal

Runs directly on hardware — no host OS underneath.

  • The hypervisor is the operating system
  • Direct access to CPU, memory, NIC, storage
  • Lower overhead — no host OS context switches
  • Used in datacenters and enterprise production

Examples: VMware ESXi, Xen, Microsoft Hyper-V

Type 2 — Hosted

Runs as an application on top of a host OS.

  • The hypervisor is a regular userspace process
  • Hardware access goes through the host OS kernel
  • Higher overhead — extra OS layer in the path
  • Used for desktop development and testing

Examples: VirtualBox, VMware Workstation, QEMU (without KVM)

Architecture Stack: Type 1

Hardware (CPU, RAM, NIC, Disk)
Type 1 Hypervisor (ESXi / Xen / Hyper-V)
VM 1
Guest OS

Apps

VM 2
Guest OS

Apps

VM 3
Guest OS

Apps

Architecture Stack: Type 2

Hardware (CPU, RAM, NIC, Disk)
Host OS (Windows / macOS / Linux)
Type 2 Hypervisor (VirtualBox / VMware Workstation)
VM 1
Guest OS

Apps

VM 2
Guest OS

Apps

Note

Key insight: Type 1 has one fewer layer. Every I/O request from a VM goes directly to the hypervisor, which talks to hardware. In Type 2, it must traverse: VM → Hypervisor → Host OS kernel → Hardware.

KVM: The Hybrid

Kernel-based Virtual Machine

KVM is a Linux kernel module (kvm.ko + kvm-intel.ko or kvm-amd.ko) that turns the Linux kernel itself into a Type 1 hypervisor. It was merged into mainline Linux in 2007 (kernel 2.6.20).

Why KVM is Technically Type 1

  • KVM runs inside the kernel, not as a userspace application
  • It uses hardware virtualization extensions (Intel VT-x / AMD-V) directly
  • The Linux kernel becomes the hypervisor — it is not running on top of an OS, it is the OS
  • But it retains all the power of a full Linux kernel: schedulers, drivers, filesystems, cgroups, namespaces

The /dev/kvm Interface

How Userspace Talks to KVM

KVM exposes itself via /dev/kvm — a character device file. Userspace programs (like QEMU) interact with KVM entirely through ioctl() system calls on file descriptors obtained from this device.

  1. open(/dev/kvm)

    Get system fd — query capabilities, API version

  2. ioctl(KVM_CREATE_VM)

    Returns a VM fd — represents one virtual machine

  3. ioctl(KVM_CREATE_VCPU)

    Returns a vCPU fd — one per virtual processor

  4. ioctl(KVM_RUN)

    Enter guest mode — CPU executes guest code natively

VMs as Linux Processes

The KVM Process Model

This is KVM's most elegant design decision:

  • Each VM is a regular Linux process (specifically, a QEMU process)
  • Each vCPU is a thread within that QEMU process
  • VM memory is just the process's address space (anonymous mmap'd memory)

This means VMs automatically get all Linux kernel capabilities:

Scheduling

CFS (Completely Fair Scheduler) schedules vCPU threads alongside all other processes. No custom scheduler needed.

Resource Limits

cgroups limit CPU, memory, I/O bandwidth per VM. Same tools you use for containers work for VMs.

Process Isolation

Standard Linux process isolation. kill, nice, taskset, numactl — all work on VMs.

KVM Architecture Diagram

Hardware (CPU with VT-x/AMD-V, RAM, Devices)
Linux Kernel + KVM Module (= Type 1 Hypervisor)
QEMU Process (PID 1234) = VM 1
Thread 1 = vCPU 0
Thread 2 = vCPU 1
I/O Thread
QEMU Process (PID 5678) = VM 2
Thread 1 = vCPU 0
Thread 2 = vCPU 1
I/O Thread
console
console

    $ ps aux | grep qemu

    libvirt+ 1234 45.2 25.0 ... qemu-system-x86_64 -name vm1 -smp 2 -m 4096 ...

    libvirt+ 5678 32.1 12.5 ... qemu-system-x86_64 -name vm2 -smp 2 -m 2048 ...


    $ ls -la /dev/kvm

    crw-rw---- 1 root kvm 10, 232 Mar 31 10:00 /dev/kvm


    $ ls /proc/1234/task/

    1234  1235  1236  1237
    # Thread 1234 = main, 1235 = vCPU 0, 1236 = vCPU 1, 1237 = I/O

QEMU: The Device Emulator

What QEMU Actually Does

QEMU (Quick Emulator) is a userspace program that can emulate an entire computer system: CPU, memory, disk controllers, network cards, USB, display adapters, BIOS/UEFI firmware.

QEMU Alone vs QEMU + KVM

QEMU Alone (TCG Mode)
  • Uses Tiny Code Generator (TCG) — a JIT binary translator
  • Translates every guest instruction to host instructions at runtime
  • ~5-10x overhead on CPU-bound workloads
  • Can emulate different architectures (e.g., ARM on x86)
  • No hardware virtualization extensions required

Use case: cross-architecture development, firmware testing

QEMU + KVM
  • CPU runs guest code natively via hardware VT-x/AMD-V
  • KVM handles CPU virtualization — near-zero CPU overhead
  • QEMU handles device emulation only (I/O path)
  • Same architecture only (x86 guest on x86 host)
  • Requires VT-x/AMD-V support in CPU

Use case: production VMs, cloud infrastructure

What QEMU Emulates (Even with KVM)

Storage

  • IDE / AHCI controllers
  • SCSI controllers (LSI, megasas)
  • NVMe controllers
  • Floppy drives
  • virtio-blk, virtio-scsi

Networking

  • e1000 / e1000e (Intel GbE)
  • rtl8139 (Realtek)
  • virtio-net
  • TAP/bridge backends
  • User-mode networking (SLiRP)

Other Devices

  • USB controllers (UHCI, EHCI, xHCI)
  • Display (VGA, QXL, virtio-gpu)
  • Sound (AC97, HDA)
  • BIOS (SeaBIOS) / UEFI (OVMF)
  • Serial, parallel ports

The QEMU Process Lifecycle

  1. Start QEMU

    qemu-system-x86_64 process launches with VM config (CPU, RAM, devices)

  2. Open /dev/kvm

    open("/dev/kvm") → get system fd, check API version and extensions

  3. Create VM

    ioctl(KVM_CREATE_VM) → allocate memory regions, set up address space

  4. Create vCPUs

    ioctl(KVM_CREATE_VCPU) per vCPU → spawn one thread per vCPU

  5. Enter Guest Mode

    Each vCPU thread calls ioctl(KVM_RUN) in a loop

  6. VM Exit Handling

    On I/O, MMIO, or exception → KVM_RUN returns → QEMU handles it → re-enter guest

Note

The KVM_RUN loop: Each vCPU thread spends most of its time in guest mode (ring 0 of the guest, non-root mode of VT-x). When the guest does something the hardware can't handle alone — like an I/O port access, an MMIO write, or a HLT instruction — the CPU triggers a VM exit, returning control to the KVM module, which either handles it in-kernel or bounces it to QEMU userspace.

VM Exit Flow

  1. Guest Code

    CPU in VMX non-root mode, executing guest instructions natively

  2. VM Exit

    Guest accesses I/O port, MMIO region, or executes privileged instruction

  3. KVM (Kernel)

    Handles simple exits in-kernel (e.g., CR register access, MSR read/write)

  4. QEMU (Userspace)

    Handles complex exits — device I/O emulation, MMIO to emulated hardware

  5. VM Entry

    KVM_RUN again → VMLAUNCH/VMRESUME → back to guest code

virtio: Paravirtualized Devices

The Problem with Emulated Hardware

When QEMU emulates a real hardware device (e.g., an Intel e1000 NIC), every single I/O register access from the guest triggers a VM exit. A single network packet might cause dozens of exits. Each exit involves:

  1. CPU saves guest state to VMCS (VM Control Structure)
  2. CPU transitions from VMX non-root → VMX root mode
  3. KVM module runs exit handler
  4. If I/O-related, control passes to QEMU userspace (kernel → user context switch)
  5. QEMU emulates the hardware register access
  6. QEMU calls KVM_RUN again (user → kernel context switch)
  7. CPU restores guest state from VMCS, re-enters VMX non-root mode

Cost: Each VM exit/entry cycle takes ~1-2 microseconds. At high I/O rates, this dominates performance.

The Solution: virtio

virtio is a standardized paravirtualization framework (OASIS standard). Instead of emulating real hardware, both the guest and host agree on an efficient virtual device protocol.

  • The guest knows it is virtualized — no pretending to be real hardware
  • Communication uses shared memory ring buffers (virtqueues) instead of I/O port traps
  • Batches of I/O operations per single notification — amortizes VM exit cost

virtio Device Family

virtio-net

Virtual network card. Guest places packets into a TX virtqueue, host picks them up. Incoming packets placed in RX virtqueue.

Supports: checksum offload, TSO/GSO, multiqueue, RSS, VLAN.

virtio-blk

Virtual block device. Guest submits read/write requests directly to a virtqueue — no SCSI/IDE command translation.

Simpler and faster than emulated IDE. One virtqueue for requests.

virtio-scsi

Virtual SCSI controller. More feature-rich than virtio-blk: supports many LUNs, hot-plug, SCSI passthrough.

Preferred for VMs needing many disks.

Others

  • virtio-gpu — 3D-accelerated virtual display
  • virtio-serial — guest-host communication channel
  • virtio-balloon — dynamic memory management
  • virtio-rng — entropy source for guest
  • virtio-fs — shared filesystem (via FUSE)

Warning

Guest drivers required: The guest OS must have virtio drivers installed. Linux has had them built-in since kernel 2.6.25. For Windows guests, you need to install the virtio-win driver package (Red Hat provides ISO images).

virtio Architecture: The Virtqueue

How a Virtqueue Works

A virtqueue is a shared-memory ring buffer with three regions:

  • Descriptor Table — array of buffer descriptors (address, length, flags, next pointer)
  • Available Ring — guest writes descriptor indices here (producer: guest)
  • Used Ring — host writes completed descriptor indices here (producer: host)

  1. Guest Prepares Buffers

    Guest driver fills descriptor table entries with data buffer addresses

  2. Guest Publishes

    Guest adds descriptor indices to the Available Ring

  3. Guest Kicks

    Guest writes to a notification register (triggers VM exit to signal host)

  4. Host Processes

    Host reads descriptors, processes I/O (sends packet, writes block)

  5. Host Completes

    Host adds completed descriptors to Used Ring, injects interrupt into guest

Guest VM
Guest Kernel
virtio Driver (e.g., virtio-net, virtio-blk)

↓ writes to shared memory ↓

⇅ Shared Memory: Virtqueue (Descriptor Table + Available Ring + Used Ring) ⇅

Host
QEMU Process
virtio Backend (device emulation)

↓ actual I/O ↓

Tip

Key optimization: Multiple I/O operations can be batched into the virtqueue before a single "kick" notification. This means one VM exit can process many packets/blocks, amortizing the exit cost dramatically.

vhost: Kernel Data-Plane

The Remaining Bottleneck

Even with virtio, there's still a problem: the host side of the virtqueue is processed by QEMU in userspace. Every batch of I/O requires:

  1. VM exit from guest to KVM (kernel)
  2. KVM returns to QEMU (kernel → userspace context switch)
  3. QEMU processes the virtqueue
  4. QEMU makes syscalls to do actual I/O (userspace → kernel context switch)

That's two extra context switches per I/O batch just because QEMU sits in userspace.

vhost Moves the Data-Plane to the Kernel

Without vhost (QEMU handles data)

Data path:

  1. Guest → virtqueue → VM exit
  2. KVM → return to QEMU userspace
  3. QEMU reads virtqueue
  4. QEMU → syscall → kernel for actual I/O
  5. Kernel → back to QEMU
  6. QEMU injects interrupt → guest

Context switches: kernel → user → kernel → user

With vhost (kernel handles data)

Data path:

  1. Guest → virtqueue → VM exit
  2. KVM signals vhost kernel thread
  3. vhost thread reads virtqueue directly
  4. vhost thread does actual I/O (already in kernel)
  5. vhost thread injects interrupt → guest

Context switches: stays in kernel the entire time

vhost Variants

vhost-net

Kernel module (vhost_net.ko) that handles virtio-net data plane in kernel space. The vhost-net kernel thread directly moves packets between the guest's virtqueue and a TAP device — no QEMU involvement for data.

Benefit: ~2x network throughput improvement over QEMU-handled virtio-net.

vhost-scsi

Kernel module that handles virtio-scsi data plane. Directly passes SCSI commands from guest virtqueue to the host's target subsystem (LIO/TCM).

Benefit: Lower latency, higher IOPS for block storage.

vhost-user

Same virtqueue bypass concept, but the data-plane runs in a separate userspace process instead of the kernel. Communication via Unix domain sockets and shared memory.

Used by: DPDK (for line-rate packet processing), Open vSwitch (OVS-DPDK), SPDK (for storage).

Control vs Data Plane Split

QEMU always handles the control plane: device setup, feature negotiation, live migration state. vhost only takes over the data plane — the hot path of moving bytes between guest and host.

This separation keeps QEMU's device model intact while removing it from the performance-critical path.

vhost-net Architecture

Hardware (NIC)
Linux Kernel
KVM Module

VM exits / entries

vhost-net Kernel Thread

Reads virtqueue directly
Writes to TAP fd
No userspace round-trip

TAP Device (tap0)

Bridge to physical NIC

QEMU Process (userspace)

Control plane only: setup, feature negotiation, live migration

Performance Comparison

Approach CPU Overhead I/O Overhead Guest Changes Needed Use Case
Full Emulation
(QEMU TCG)		 Very high (~5-10x)
Binary translation of every instruction		 Very high
All devices emulated in software		 None — guest is unaware of virtualization Cross-arch dev (ARM on x86), firmware debugging, no VT-x available
KVM + Emulated Devices
(e.g., e1000, IDE)		 Near-zero
Native CPU execution via VT-x		 High
Every I/O register access = VM exit + QEMU emulation		 None — guest uses standard drivers (e1000, IDE, etc.) Compatibility-first: unmodified guests, legacy OS, driver availability
KVM + virtio Near-zero
Native CPU execution via VT-x		 Low
Batched I/O via shared ring buffers, fewer VM exits		 virtio drivers required
(built into Linux, separate install for Windows)		 Production Linux VMs, cloud instances (AWS, GCP use virtio)
KVM + virtio + vhost Near-zero
Native CPU execution via VT-x		 Very low
Data-plane in kernel, no QEMU in hot path		 virtio drivers required + vhost kernel modules on host High-performance networking/storage, telco NFV, latency-sensitive workloads

Tip

Cloud providers and virtio: AWS (Nitro/ENA is virtio-based), GCP (uses virtio-net and virtio-scsi), and most OpenStack deployments default to virtio devices. If you've used a cloud VM, you've used virtio.

The Full Stack: From Slowest to Fastest

  1. Full Emulation

    TCG + emulated HW
    ~5-10x CPU, very high I/O overhead

  2. KVM + Emulated HW

    Native CPU, emulated I/O
    Near-zero CPU, high I/O overhead

  3. KVM + virtio

    Native CPU, paravirt I/O
    Near-zero CPU, low I/O overhead

  4. KVM + virtio + vhost

    Native CPU, kernel data-plane
    Near-zero CPU, very low I/O overhead

Summary: The Layered Optimization Story

  1. KVM solved CPU overhead — guest code runs natively on hardware
  2. virtio solved device emulation overhead — shared memory rings replace I/O port traps
  3. vhost solved context switch overhead — data-plane stays in kernel, QEMU exits the hot path

Each layer addresses a specific bottleneck. Together, they achieve near-native performance for both CPU and I/O workloads — which is why KVM+QEMU+virtio+vhost is the dominant virtualization stack in production today (powering AWS, GCP, OpenStack, and most Linux-based clouds).

Solidnines — solidnines.com