Linux Storage15 min read

ZFS

The integrated filesystem, volume manager, RAID, and integrity layer

Philosophy

ZFS was developed at Sun Microsystems starting in 2001 and marketed as “the last word in filesystems”. Today it lives on as OpenZFS, the community-maintained fork that runs on FreeBSD, illumos, Linux, and macOS.

Its design goal was radical for its time: collapse the classic storage stack — partitioner, volume manager, RAID engine, filesystem, integrity checker — into a single coherent system. The authors argued (correctly) that layered stacks like mdadm + LVM + ext4 + fsck are each unaware of the other's semantics: mdadm doesn't know which blocks contain file data, LVM doesn't know what redundancy the lower layer provides, and ext4 can't tell the RAID layer that a corrupt block is recoverable elsewhere. Each layer is forced to make pessimistic assumptions.

What ZFS merges into one system

Filesystem — POSIX directories, files, xattrs
Volume manager — pools of devices, datasets, zvols
RAID engine — mirror, raidz1/2/3, draid
Integrity layer — end-to-end checksums on every block
Snapshot & replication engine — CoW snapshots, send/receive
Caching tier — ARC (RAM), L2ARC (SSD), SLOG
Encryption layer — native AES-256-GCM at dataset granularity

Why it matters operationally

End-to-end checksums catch silent corruption that traditional stacks miss entirely — bit rot, misdirected writes, phantom writes, controller/cable errors, DRAM flips that escape ECC.
Copy-on-write gives free, atomic snapshots and crash consistency — the on-disk tree is always a valid previous state.
No fsck, ever. A ZFS pool cannot be half-committed. A crash simply means the uberblock points to the last-committed transaction group — everything in it is already consistent.

The Storage Hierarchy

Internalising the ZFS hierarchy is non-negotiable. Three levels sit between the raw disks and the filesystem you mount.

Datasets & zvols — pool/home, pool/vm-100-disk-0

Pool (zpool) — stripe across vdevs

vdevs — fault-tolerance unit (mirror / raidz / …)

Disks — physical block devices

Hardware (HBA / SAS / SATA / NVMe)

vdev — Virtual Device

A vdev is the fault-tolerance unit. From the pool's point of view each vdev is a black box with a stated redundancy level. If a vdev fails (beyond its redundancy) the entire pool fails. vdevs come in several flavours — data vdevs, and support vdevs for caching/logging/spares.

Pool (zpool)

A pool is one or more vdevs combined. Data is striped (RAID-0-like) across the data vdevs in the pool. The pool is the unit of import/export and the top of the administrative namespace.

Warning

Pool layout is largely irreversible. Historically you could only add data vdevs to a pool, never remove them. Recent OpenZFS allows removal of top-level mirror or single-disk vdevs via zpool remove, but raidz vdevs cannot be removed. Raidz vdev expansion (adding one disk to an existing raidz) only landed in OpenZFS 2.3. Plan the vdev layout carefully up front.

Dataset & zvol

A dataset is a named filesystem within the pool — pool/home, pool/home/alice. Datasets form a hierarchical tree; children inherit properties from their parent by default. Each dataset has its own mountpoint, compression, quota, snapshots, etc.

A zvol is a dataset exposed as a block device (e.g. /dev/zvol/pool/vm-100-disk-0) — used for VM disks, iSCSI LUNs, swap, or anything that wants raw block storage on top of the pool.

console

bash

root@host:~# zpool status tank
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 02:14:33 with 0 errors on 2026-04-07

config:

        NAME                      STATE     READ WRITE CKSUM
        tank                      ONLINE       0     0     0
          raidz2-0                ONLINE       0     0     0
            ata-WDC_WD80EFBX-...  ONLINE       0     0     0
            ata-WDC_WD80EFBX-...  ONLINE       0     0     0
            ata-WDC_WD80EFBX-...  ONLINE       0     0     0
            ata-WDC_WD80EFBX-...  ONLINE       0     0     0
            ata-WDC_WD80EFBX-...  ONLINE       0     0     0
            ata-WDC_WD80EFBX-...  ONLINE       0     0     0
        special
          mirror-1                ONLINE       0     0     0
            nvme0n1p1             ONLINE       0     0     0
            nvme1n1p1             ONLINE       0     0     0
        logs
          mirror-2                ONLINE       0     0     0
            nvme0n1p2             ONLINE       0     0     0
            nvme1n1p2             ONLINE       0     0     0
        cache
          nvme0n1p3               ONLINE       0     0     0

errors: No known data errors

Copy-on-Write, Everywhere

ZFS never overwrites live data in place. Every write path allocates fresh blocks, writes the new indirect blocks bottom-up, and then atomically updates the pool's uberblock — a small on-disk pointer — to point at the new tree root.

App writes
New data blocks allocated from free space
Indirect blocks
Updated bottom-up with new pointers & checksums
New uberblock
Atomic pointer swap to new tree root
Old blocks freed
Unless pinned by a snapshot

Consequences

Always consistent on disk — a crash leaves the last committed transaction group intact; partial writes don't exist. No fsck.
Snapshots are free — just keep a pointer to a previous tree root. No data copy.
Efficient replication — zfs send walks block-pointer deltas, not file contents.
Fragmentation can accumulate on heavily-churned pools with little free space — keep pools below ~80% full.

Checksums on Everything

Every block in a ZFS pool has a checksum. Critically, the checksum is stored in the parent block pointer — not next to the data. This detaches the integrity guarantee from the data itself: a misdirected write that lands in the wrong location, or a block that's been silently overwritten, will fail verification because the expected checksum lives elsewhere in the tree.

Note

Default is fletcher4 — a fast non-cryptographic checksum designed for this workload. SHA-256, SHA-512, Skein, Edon-R, and BLAKE3 are available for stronger guarantees or when dedup demands collision resistance.

What this catches that traditional stacks miss

Failure mode	Traditional stack	ZFS
Disk reports bad sector (URE)	Detected, RAID recovers	Detected, self-heals
Silent bit rot (block returns wrong data, no error)	Undetected	Detected, self-heals from redundancy
Misdirected write (firmware bug, landed at wrong LBA)	Undetected	Detected via parent checksum
Phantom write (write acknowledged, never reached platter)	Undetected	Detected on next read
In-flight corruption on HBA/cable/DRAM	Undetected	Detected end-to-end

Self-healing read path

On a read, ZFS computes the checksum of the returned block and compares it against the value stored in the parent pointer. If they disagree and the vdev has redundancy (mirror / raidz), ZFS:

Reads a redundant copy (mirror peer, or raidz reconstruction)
Verifies that copy's checksum too
Returns good data to the application
Rewrites the bad copy in place — silent rescue, recorded in CKSUM counters

vdev Types

Data vdevs

Type	Redundancy	Usable capacity	Notes
single disk	None	100%	No self-healing possible. Loss = pool loss.
mirror	Survives N−1 failures (N-way)	1/N	2-way typical; 3-way for paranoid or large rebuild windows.
raidz1	1 disk loss	(N−1)/N	Like RAID 5, without the write hole (full-stripe CoW).
raidz2	2 disk loss	(N−2)/N	Like RAID 6. Recommended for 6+ disk arrays.
raidz3	3 disk loss	(N−3)/N	For very large arrays with long rebuild times.
draid	Configurable	Depends on config	Distributed RAID — reserves spare capacity across all disks, resilver reads/writes many disks in parallel. Much faster rebuild.

Note

raidz vs legacy RAID 5: raidz writes always span a full stripe — there is no partial-stripe read-modify-write, and therefore no write hole. Stripe width is variable; small writes use narrow stripes. This is possible because ZFS owns the filesystem layer and knows which blocks are live.

mirror

Every write goes to every disk. Reads can come from any copy.

Fastest random IO (any disk answers reads)
Fastest rebuild (plain disk-to-disk copy of live data)
Worst capacity efficiency
Best for VM storage, databases

raidz1

Single-parity. One disk's worth of capacity goes to parity.

Good capacity efficiency with 4–5 disks
Slow random reads (wide stripe)
Risky for large disks — rebuild time puts URE probability uncomfortably high

raidz2

Double-parity. Survives any 2 disks failing.

Default choice for bulk storage arrays of 6+ disks
Handles a URE during rebuild of a failed disk
Capacity efficiency ≈ (N−2)/N

raidz3 / draid

Triple parity / distributed RAID for very large arrays.

raidz3: for 12+ disks with long rebuild windows
draid: distributes spare capacity → rebuild reads/writes many disks in parallel
Used in large JBODs, backup targets, nearline

Support vdevs

vdev	Purpose	Loss impact
special	Dedicated SSD vdev for metadata and optionally small data blocks. Massive speedup for metadata-heavy workloads (`ls -lR`, rsync, find).	Pool loss if redundancy fails — mirror it.
log (SLOG)	Low-latency device for the ZFS Intent Log. Handles sync writes only. Very fast NVMe with PLP is ideal.	Losing SLOG = lose sync writes in flight at crash time. Pool survives.
cache (L2ARC)	SSD tier below the RAM-based ARC. Read cache only. Optional.	Pool unaffected — safe to lose.
spare	Hot spare. Auto-attached when a disk fails (if autoreplace / hot-spare policy enabled).	Pool unaffected.
dedup	Dedicated vdev for the dedup table (OpenZFS 2.3+).	Pool loss if lost — mirror it.

Caching: ARC, L2ARC, ZIL, SLOG

ARC — Adaptive Replacement Cache

The ARC is ZFS's RAM-based read cache. Unlike the Linux page cache it uses the ARC algorithm, which balances recency and frequency dynamically with scan-resistant ghost lists — a sequential find / -type f does not evict your hot working set.

ARC grows opportunistically towards a tunable arc_max (default on Linux is half of RAM) and releases memory under pressure. Monitor it:

console

bash

root@host:~# arc_summary | head -30
ZFS Subsystem Report                           Sun Apr 20 10:12:04 2026
Linux 6.8.0-52-generic                                 2.2.3-1~bookworm
Machine: nas (x86_64)                                               2.2.3-1

ARC status:
        Total memory available:                                128.0 GiB
        ARC size (current):                       89.2 %       57.0 GiB
        Target size (adaptive):                   89.6 %       57.3 GiB
        Min size (hard limit):                     6.2 %        4.0 GiB
        Max size (high water):                    50.0 %       64.0 GiB

ARC hit ratio:                                    97.4 %     21.1 M
        Demand data hit ratio:                    96.1 %      8.2 M
        Demand metadata hit ratio:                99.8 %     12.4 M

L2ARC

L2ARC is an SSD-backed extension of the ARC, used when the working set exceeds RAM. It's only useful for read-heavy workloads that miss ARC. It costs RAM — the ARC needs pointers tracking every L2ARC entry, roughly 70–100 bytes per block. Oversizing L2ARC relative to RAM is a classic mistake.

ZIL — ZFS Intent Log

The ZIL is where synchronous writes are durably logged before the next transaction group commits them to the pool proper. Every pool has a ZIL; by default it lives inline on the data disks as scattered ZIL blocks.

SLOG — Separate LOG

A SLOG is a dedicated (fast) log vdev that moves the ZIL off the data disks. Typical workloads: NFS servers with many sync writes, databases with O_SYNC/fsync, iSCSI with writeback disabled.

Warning

The SLOG is NOT a general-purpose write cache.

Async writes never touch the ZIL or SLOG — they go straight to the in-RAM transaction group and are flushed to pool on the next TXG sync (typically every 5 s).
Only sync writes (and large sync writes only above zfs_immediate_write_sz) use the ZIL.
The ZIL is read only during crash recovery, to replay any committed sync writes that hadn't made it to the pool.
SLOG device must have power-loss protection (PLP) if you want the durability guarantee to be meaningful. Consumer NVMe is usually a bad choice.

Dataset Properties Worth Knowing

Property	Recommended	Why
`compression`	`lz4` (or `zstd`)	Almost always on. lz4 is near-free on modern CPUs, ~1.5–2× on typical data, compresses-or-punts (no penalty on incompressible blocks). zstd for better ratio at higher CPU cost.
`atime`	`off`	Disables access-time updates. Removes a constant write workload for many-file trees.
`xattr`	`sa`	Stores extended attributes in the inode (system attribute) rather than as hidden files. Faster, less metadata churn. Required for SELinux, ACLs.
`recordsize`	`128K` default	Maximum block size for files. Smaller (e.g. `16K`) for PostgreSQL, `8K` for MySQL InnoDB. Larger (`1M`) for media/backup targets.
`volblocksize`	Depends	Block size for zvols. Set at creation only. Match the guest's expected block size (often 16K for VM disks; 8K for DB VMs).
`dedup`	`off` unless measured	Expensive. Needs roughly 1–5 GB RAM per 1 TB of unique data for the DDT. Compression + clones usually win.
`encryption`	`aes-256-gcm`	Native dataset-level encryption. Keys managed per-dataset. Enables raw replication (`send -w`) to untrusted destinations.
`sync`	`standard`	Honour `O_SYNC` / `fsync`. `always` for DBs wanting extra durability; `disabled` for speed at crash-consistency risk.
`quota` / `refquota`	As needed	`quota` includes snapshots/descendants; `refquota` limits only the live dataset.
`reservation` / `refreservation`	As needed	Guaranteed space. `refreservation=auto` on thick zvols prevents ENOSPC on the pool from killing guest writes.
`primarycache` / `secondarycache`	`all` default	Per-dataset ARC/L2ARC policy. Set `metadata` for DB datasets to avoid polluting ARC with cold pages.

console

bash

root@host:~# zfs create -o compression=zstd -o atime=off -o recordsize=16K tank/postgres
root@host:~# zfs get compression,atime,recordsize,xattr tank/postgres
NAME           PROPERTY     VALUE     SOURCE
tank/postgres  compression  zstd      local
tank/postgres  atime        off       local
tank/postgres  recordsize   16K       local
tank/postgres  xattr        sa        inherited from tank

Snapshots, Clones, Send/Receive

Snapshots

A snapshot is a read-only, atomic, instant point-in-time view of a dataset. Because ZFS is CoW, taking one is just writing a new snapshot record that pins the current tree root. Deleting newer data doesn't free blocks that snapshots still reference.

console

bash

root@host:~# zfs snapshot tank/home@2026-04-20
root@host:~# zfs list -t snapshot -r tank/home
NAME                        USED  AVAIL     REFER  MOUNTPOINT
tank/home@2026-04-19       8.2M      -     42.1G  -
tank/home@2026-04-20          0      -     42.1G  -

user@host:~$ cd ~/.zfs/snapshot/2026-04-20
user@host:~$ ls -la
drwxr-xr-x  42 alice alice   42 Apr 20 09:14 .
...

Tip

Set snapdir=visible on datasets where users benefit from self-service file restore from .zfs/snapshot/. Pair with a snapshot scheduler like zfs-auto-snapshot, sanoid, or zrepl.

Rollback

zfs rollback tank/home@2026-04-20 reverts the dataset to the snapshot — and destroys any snapshots taken after it. -r allows destroying intermediate snapshots; use with care.

Clones

A clone is a writable snapshot — zfs clone tank/home@snap tank/home-copy creates a new dataset that shares blocks with the snapshot until it diverges. Perfect for testing schema migrations against a production snapshot, or instant VM template provisioning. zfs promote swaps origin and clone, letting you delete the original dataset.

Send / Receive

Replication in ZFS walks the block-pointer delta directly — no file tree traversal, no file-by-file comparison. It is orders of magnitude more efficient than rsync on filesystems with many files.

Mode	Command	Use
Full	`zfs send tank/home@snap \| zfs receive back/home`	Initial seed
Incremental	`zfs send -i @prev tank/home@snap \| ssh dest "zfs receive back/home"`	Routine sync
Recursive	`zfs send -R tank@snap \| ...`	All descendants, properties, snapshots
Raw / encrypted	`zfs send -w ...`	Ciphertext-only replication to an untrusted target — the remote cannot read your data
Resumable	`zfs send -t <token>`	Resume an interrupted send (requires `-s` on receive side)

Maintenance: Scrub, Resilver, Monitoring

Scrub

zpool scrub tank reads every live block in the pool, recomputes its checksum, and repairs any mismatch from redundant copies. This is the mechanism by which ZFS detects and heals bit rot before it propagates into your backups.

Tip

Run scrubs monthly. A common cadence is weekly on SSD pools, monthly on HDD pools. Without scrub you don't know a cold file has rotted until you try to read it — possibly after the redundancy has already quietly degraded on another block.

Resilver

Resilvering is the ZFS term for rebuilding a vdev after disk replacement. Crucially, resilver only reads/writes live (allocated) blocks — unlike traditional RAID which rebuilds the entire disk surface regardless of whether it's in use. A half-full pool resilvers roughly half as fast as RAID would, and the window for a second failure hitting live data is correspondingly smaller.

console

bash

root@host:~# zpool replace tank ata-BAD-serial /dev/disk/by-id/ata-NEW-serial
root@host:~# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.
  scan: resilver in progress since Sun Apr 20 09:00:00 2026
        312G scanned at 1.2G/s, 118G issued at 480M/s, 8.4T total
        29.5G resilvered, 1.37% done, 04:12:21 to go

Routine monitoring

`zpool status -v`	Health, errors, current scrub/resilver, per-device CKSUM counts
`zpool iostat -v 1`	Per-vdev IO stats, latency distribution (with `-l`)
`zpool list -v`	Pool and vdev capacity, fragmentation
`zfs list -t all`	All datasets, volumes, snapshots
`arc_summary`	ARC hit ratio, sizing, memory pressure
`zpool events`	Event log — checksum errors, IO errors, resilvers, vdev state changes

Tuning Knobs

`ashift` — set at pool creation, cannot change

ashift is the log₂ of the smallest IO unit ZFS will issue to a vdev. It cannot be changed after pool creation without rebuilding. Match it to the disk's physical sector size:

ashift	Sector size	When to use
9	512 B	Very old disks only
12	4 KiB	Default for modern HDDs and most SSDs
13	8 KiB	NVMe that prefer 8K pages, some enterprise SSDs
14	16 KiB	Certain modern NVMe with 16K IU

Warning

Wrong ashift hurts. Too low (9 on a 4K disk) causes massive write amplification. Too high (13 on a 4K disk) wastes space for small files. Disks often lie about their sector size (reporting 512 for backwards compat). Check with smartctl -i or manufacturer datasheet, then pin ashift=12 explicitly at pool creation.

Other knobs worth knowing

recordsize (dataset) — already covered; tune per workload.
volblocksize (zvol, creation-time only) — match guest FS / DB page size.
primarycache=all|metadata|none — per-dataset ARC policy.
logbias=latency|throughput — how sync writes use the ZIL. throughput bypasses SLOG for large writes.
zfs_arc_max (module parameter) — hard cap on ARC size. Often pinned on systems co-hosting KVM/containers to prevent ARC↔guest tug-of-war.

ZFS on Linux (OpenZFS)

ZFS originated at Sun (Solaris). Oracle later relicensed Solaris ZFS as closed source; the community picked up the last CDDL-licensed snapshot and has developed OpenZFS ever since. OpenZFS is now the upstream for all ZFS ports.

Warning

Licensing: CDDL (ZFS) is incompatible with GPLv2 (Linux kernel). That's why ZFS on Linux cannot be merged into the mainline kernel and ships as an out-of-tree module (DKMS or prebuilt per-kernel). Canonical has ruled internally that shipping the module binary with Ubuntu is acceptable; the FSF disagrees. Practically: it works, but upgrades need care.

Distro support

Ubuntu	Shipped in installer, `zfsutils-linux` + `zfs-dkms`. Root-on-ZFS supported in the GUI installer.
Debian	In `contrib`. `apt install zfs-dkms zfsutils-linux`. No root-on-ZFS in the default installer.
Proxmox VE	First-class. Installer offers ZFS mirror / raidz root directly. Proxmox kernel ships with pre-built ZFS modules.
RHEL / Rocky / Alma	Via OpenZFS repo. DKMS-only; expect to manage kernel/module compatibility manually.
Arch	AUR + `zfs-linux` / `zfs-dkms`. Kernel upgrades frequently break the module — use the LTS kernel and pin.

Note

The OpenZFS project tracks each upstream kernel release and updates the kernel-facing shim as needed. The module-to-kernel interface is not a stable kernel API — blind kernel upgrades can brick ZFS until a compatible OpenZFS release lands.

When to Use ZFS — and When Not To

Good fit

Storage servers / NAS — integrity + snapshots + replication
Backup targets — compression, dedup (if measured), send/recv
Proxmox hosts — built-in, local+replicated VM storage with snapshots
Any workload that values data integrity over peak throughput
Long-term archival — scrubs catch bit rot before it spreads

Poor fit (or: be careful)

Laptops — root-on-ZFS adds real complexity around kernel upgrades, hibernation, initramfs
Low-RAM systems (< 4 GB) — ARC competes with everything, performance suffers
Latency-critical DBs with their own integrity — PG/MySQL can outperform on ext4/XFS if you accept the integrity story; if you use ZFS, tune recordsize and primarycache=metadata
Pure peak throughput on fast NVMe — the CoW tax is real; XFS is faster for some sequential workloads

Tip

Proxmox cross-reference: Proxmox uses ZFS heavily — see Virt 10 — Storage & HA for how pveperf, pvesm, and replication plug into ZFS pools for VM disks and backups.

Philosophy

Why it matters operationally

The Storage Hierarchy

vdev — Virtual Device

Pool (zpool)

Dataset & zvol

Copy-on-Write, Everywhere

App writes

Indirect blocks

New uberblock

Old blocks freed

Checksums on Everything

What this catches that traditional stacks miss

Self-healing read path

vdev Types

Data vdevs

Support vdevs

Caching: ARC, L2ARC, ZIL, SLOG

ARC — Adaptive Replacement Cache

L2ARC

ZIL — ZFS Intent Log

SLOG — Separate LOG

Dataset Properties Worth Knowing

Snapshots, Clones, Send/Receive

Snapshots

Rollback

Clones

Send / Receive

Maintenance: Scrub, Resilver, Monitoring

Scrub

Resilver

Routine monitoring

Tuning Knobs

ashift — set at pool creation, cannot change

Other knobs worth knowing

ZFS on Linux (OpenZFS)

Distro support

When to Use ZFS — and When Not To

`ashift` — set at pool creation, cannot change