Virtualization6 min read

Union Filesystems & OverlayFS

How multiple read-only layers merge into a single writable filesystem view

The Union FS Concept

Core Idea

A union filesystem merges multiple directories into a single unified view. Lower directories are read-only; an upper directory captures all writes. To the process inside the container, it looks like one normal filesystem.

Union Mount — Merged View

upperdir (read-write)

text

bash


          new-file.txt

          .wh.deleted-file.txt

Container's writes go here

lowerdir 2 (read-only)

text

bash


          app/

            server.py

            config.yml

lowerdir 1 (read-only)

text

bash


          bin/ lib/ usr/

          deleted-file.txt

↓ merges into ↓

merged/ (what the container sees)

text

bash


        bin/ lib/ usr/

        app/

          server.py

          config.yml

        new-file.txt

        deleted-file.txt ← hidden by whiteout

OverlayFS / overlay2

OverlayFS is a kernel-level union filesystem that has been the default storage driver for Docker and containerd since Docker 18.x. It is mature, performant, and works on all major Linux filesystems (ext4, xfs).

Four Directories

Directory	Mode	Purpose
`lowerdir`	Read-only	One or more image layers, colon-separated, rightmost = bottom-most layer
`upperdir`	Read-write	Captures all modifications (new files, changes, whiteouts for deletions)
`workdir`	Internal	Used by the kernel for atomic copy-up operations; must be on the same filesystem as upperdir
`merged`	Mount point	The unified view — this is what the container process sees as its root filesystem

Manual Overlay Mount

console


    $ mount -t overlay overlay \

      -o lowerdir=/layer3:/layer2:/layer1,upperdir=/upper,workdir=/work \

      /merged


    # lowerdir order: /layer3 is on top, /layer1 is on bottom

    # Files in /layer3 shadow same-named files in /layer2 and /layer1


    $ ls /merged

    bin  etc  lib  app  usr  var  # unified view of all layers


    $ echo "hello" > /merged/new-file.txt

    $ ls /upper

    new-file.txt  # write landed in upperdir, not in any lowerdir

Read Path

Check upperdir
Look for the file in the read-write upper layer first
Search lowerdirs
If not in upper, search lowerdirs top-to-bottom (left-to-right in mount options)
First match wins
Upper layers shadow (hide) files with the same path in lower layers

Tip

Read performance: Reading a file from any layer is the same speed — there is no decompression or reconstruction. OverlayFS layers are already extracted on disk. The kernel simply resolves which file to serve.

Write Path (Copy-Up)

When a process modifies a file that exists in a lower (read-only) layer, OverlayFS performs a copy-up operation:

The entire file is copied from the lowerdir to the upperdir (not just the changed bytes)
The modification is applied to the upperdir copy
Subsequent reads of this file now come from upperdir (it shadows the lowerdir version)
The original lowerdir file remains unchanged — it is never modified

Warning

Performance implication: Copy-up is a one-time cost per file, but it copies the entire file — not just the changed region. Modifying a 1 GB database file from a lower layer triggers a full 1 GB copy to upperdir. This is why databases should use volumes, not the container filesystem.

Note

Only the file is copied, not the layer. Copy-up operates at file granularity. Modifying one file does not copy the entire image layer — just that single file.

Delete Path (Whiteout)

Since lowerdir files cannot be modified or removed, OverlayFS uses whiteout files to represent deletions.

File Whiteout

Deleting a file /app/config.yml from a lower layer creates a character device with major/minor 0/0 named /upper/app/.wh.config.yml.

console


        $ rm /merged/app/config.yml


        # What actually happened in upperdir:

        $ ls -la /upper/app/

        c--------- 1 root root 0, 0 ... .wh.config.yml


        # The lowerdir file is untouched:

        $ ls /layer3/app/config.yml

        /layer3/app/config.yml  # still there

Opaque Directory Whiteout

To hide an entire directory's contents from lower layers, a special file .wh..wh..opq is placed inside the directory in upperdir.

console


        # Lower layer has /var/log/ with many files

        # Upper layer recreates /var/log/ as empty:


        $ ls -la /upper/var/log/

        c--------- 1 root root 0, 0 ... .wh..wh..opq


        # Merged view: /var/log/ appears empty

        # (all lower layer contents are hidden)

Docker's overlay2 Structure

Docker's overlay2 storage driver maps image layers and container writable layers onto the OverlayFS directories:

text

bash


    /var/lib/docker/overlay2/

    ├── abc123def456.../ ← image layer 1

    │   ├── diff/ ← actual filesystem content for this layer

    │   ├── link ← shortened symlink name (used in mount options to avoid arg length limits)

    │   └── lower ← pointer to layers below (colon-separated link names)

    ├── 789ghi012jkl.../ ← image layer 2

    │   ├── diff/

    │   ├── link

    │   └── lower

    ├── mno345pqr678.../ ← container read-write layer

    │   ├── diff/ ← container's modifications (the upperdir)

    │   ├── work/ ← OverlayFS workdir for atomic ops

    │   ├── merged/ ← unified view = container's rootfs

    │   └── lower ← points to all image layers below

    └── l/ ← directory of shortened symlinks to layer diff/ dirs

         ├── ABCD → ../abc123def456/diff

         ├── EFGH → ../789ghi012jkl/diff

         └── IJKL → ../mno345pqr678/diff

console


    # Inspect a running container's mount:

    $ docker inspect --format='{{.GraphDriver.Data.MergedDir}}' my-container

    /var/lib/docker/overlay2/mno345pqr678/merged


    $ mount | grep overlay

    overlay on /var/lib/docker/overlay2/mno345pqr678/merged type overlay
(rw,lowerdir=/var/lib/docker/overlay2/l/EFGH:/var/lib/docker/overlay2/l/ABCD,
upperdir=/var/lib/docker/overlay2/mno345pqr678/diff,
workdir=/var/lib/docker/overlay2/mno345pqr678/work)

Container Read-Write Layer

Ephemeral by Design

The container's upperdir is destroyed when docker rm is called
All writes — logs, temp files, database changes — are lost
This is intentional: containers are disposable
docker commit can snapshot the upperdir into a new image layer (but don't do this in production)

Why Volumes Exist

Volumes bypass the overlay filesystem entirely
They are bind-mounted or managed directories on the host
Data persists independently of the container lifecycle
No copy-up overhead — direct read/write to the underlying filesystem
Can be shared between multiple containers

Note

Copy-on-write overhead in practice: If a container writes to a 500 MB file from the image (e.g., a bundled SQLite database), the entire 500 MB is first copied to upperdir. This doubles the storage usage for that file and introduces latency on the first write. For any significant I/O, use a volume.

Storage Driver Comparison

Driver	Mechanism	Pros	Cons	Status
overlay2	OverlayFS (kernel)	Fast, efficient, native kernel support, simple setup	Copy-up overhead on first write to lower-layer files	Default, recommended
devicemapper	LVM thin provisioning	Block-level CoW (no file-level copy-up), good for large files	Complex setup (requires direct-lvm for production), slower image builds	Deprecated
btrfs	Btrfs snapshots	Native snapshots, built-in compression, checksumming	Requires btrfs filesystem, less mature on some distros	Niche
zfs	ZFS clones	Excellent snapshotting, handles many layers well, data integrity	Requires ZFS, memory-heavy (ARC cache), not in mainline kernel	Niche

Tip

How to check your storage driver:

console


    $ docker info --format '{{.Driver}}'

    overlay2


    $ docker info --format '{{.DriverStatus}}'

    [[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true]]

Warning

d_type support is required. OverlayFS needs d_type (directory entry type) support in the backing filesystem. All modern ext4 and xfs filesystems have this enabled by default. Running on an old xfs without ftype=1 will cause silent data corruption.

The Union FS Concept

OverlayFS / overlay2

Four Directories

Manual Overlay Mount

Read Path

Check upperdir

Search lowerdirs

First match wins

Write Path (Copy-Up)

Delete Path (Whiteout)

Docker's overlay2 Structure

Container Read-Write Layer

Storage Driver Comparison