Linux Software RAID (mdadm)

Managing arrays with the Linux MD subsystem

What mdadm Is

mdadm (multi-disk admin) is the userspace administration tool for the Linux kernel's MD (Multiple Device) subsystem. MD is the in-kernel driver that implements the actual RAID behaviour — mdadm just configures and monitors it.

• Implemented in drivers/md/ in the kernel tree (raid0.c, raid1.c, raid5.c, etc.)
• Supports RAID 0, 1, 4, 5, 6, 10, plus linear concatenation, multipath, and faulty (for testing)
• Exposes arrays as /dev/mdN block devices — indistinguishable to upper layers from any other block device
• Configuration state lives in superblocks on each member, not in a central controller

Creating Arrays

# mdadm --create /dev/md0 \
    --level=5 \
    --raid-devices=4 \
    --chunk=512 \
    --bitmap=internal \
    /dev/sd[abcd]1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Key flags:

• --level — raid0, raid1, raid5, raid6, raid10, linear
• --raid-devices — number of active members
• --spare-devices — hot spares (see below)
• --chunk — stripe chunk size in KiB. 512 is a common default for spinning disks; 64–128 for SSDs. Large chunks favour sequential workloads; small chunks favour random I/O.
• --bitmap=internal — enable write-intent bitmap (discussed later)
• --metadata=1.2 — superblock version (1.2 is the default)

The Array Superblock

Every member disk stores a superblock describing the array: UUID, level, role of this disk, event count, checksum. On boot, the kernel scans attached disks, reads superblocks, groups them by UUID, and assembles arrays automatically.

/proc/mdstat — The Status Window

A plain-text file exported by the kernel. Reflects the live state of every assembled array.

$ cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sdc1[2] sdb1[1] sda1[0]
      23441679360 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/59 pages [0KB], 65536KB chunk

unused devices: <none>

$ cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid5 sde1[4] sdc1[2] sdb1[1] sda1[0]
      23441679360 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
      [===>.................]  recovery = 17.8% (1391420928/7813893120) finish=423.7min speed=252618K/sec
      bitmap: 0/59 pages [0KB], 65536KB chunk

unused devices: <none>

Reading the status line:

• [4/3] — 4 expected members, 3 currently up
• [UUU_] — one character per slot: U = up, _ = missing/failed. Position is role index, not physical enumeration.
• [F] after a device name in the member list = faulty (e.g. sdc1[2](F))
• (S) after a device = spare
• recovery = X% — sync/rebuild progress; resync, check, or reshape also appear here
• Bitmap line appears only when a write-intent bitmap is attached

Inspecting Arrays and Members

# mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Wed Apr  9 14:22:31 2025
        Raid Level : raid5
        Array Size : 23441679360 (21.83 TiB 24.00 TB)
     Used Dev Size : 7813893120 (7.28 TiB 8.00 TB)
      Raid Devices : 4
     Total Devices : 4
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Mon Apr 20 11:04:58 2026
             State : clean
    Active Devices : 4
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : nas:0  (local to host nas)
              UUID : 8f2a1e3c:4b5d6e7f:8090a1b2:c3d4e5f6
            Events : 47213

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1

# mdadm --examine /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 8f2a1e3c:4b5d6e7f:8090a1b2:c3d4e5f6
           Name : nas:0
  Creation Time : Wed Apr  9 14:22:31 2025
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 15627786240 sectors (7.28 TiB 8.00 TB)
     Array Size : 23441679360 KiB (21.83 TiB 24.00 TB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 1a2b3c4d:5e6f7a8b:9c0d1e2f:3a4b5c6d

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Apr 20 11:04:58 2026
       Checksum : e7d1a92b - correct
         Events : 47213

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

Persistent Configuration

The superblocks are sufficient to assemble an array, but mdadm.conf tells the system which arrays to look for, what to name them, and where to send alerts.

# mdadm --detail --scan >> /etc/mdadm/mdadm.conf
# cat /etc/mdadm/mdadm.conf
DEVICE partitions
HOMEHOST <system>
MAILADDR root
ARRAY /dev/md0 metadata=1.2 name=nas:0 UUID=8f2a1e3c:4b5d6e7f:8090a1b2:c3d4e5f6

# update-initramfs -u               # Debian/Ubuntu
# dracut -f                         # RHEL/Fedora

Monitoring and Alerting

MD can push events to a monitoring daemon which sends email on failure. Without this, a failed disk sits unnoticed until the second disk fails.

# mdadm --monitor --daemonise --mail=root --delay=300 /dev/md0

# /etc/mdadm/mdadm.conf
MAILADDR ops@example.com
MAILFROM nas-root@example.com

# systemctl enable --now mdmonitor.service

Events reported: Fail, FailSpare, SpareActive, NewArray, DegradedArray, MoveSpare, TestMessage.

Hot Spares

A spare sits idle until an active member fails; MD then kicks off rebuild automatically onto the spare.

# mdadm --create /dev/md0 \
    --level=5 --raid-devices=4 --spare-devices=1 \
    /dev/sd[abcd]1 /dev/sde1

# mdadm /dev/md0 --add-spare /dev/sdf1

Global spares (spare-group)

A single spare can be pooled across multiple arrays. Tag each array with the same spare-group name in mdadm.conf; MD's monitor daemon moves spares between arrays on demand.

# /etc/mdadm/mdadm.conf
ARRAY /dev/md0 spare-group=pool1 UUID=...
ARRAY /dev/md1 spare-group=pool1 UUID=...

Replacing a Failed Disk

1. Identify the failed disk. /proc/mdstat shows which slot ([UU_U]), mdadm --detail gives the device node, dmesg has the kernel error log. Cross-check with smartctl -a /dev/sdX and the disk serial via ls -l /dev/disk/by-id/.

   # ls -l /dev/disk/by-id/ | grep sdc
   ata-WDC_WD80EFAX-68LHPN0_7HK5L0RC -> ../../sdc

2. Mark as failed (if not already). MD may still show the drive as active if it's flaky but not hard-failed.

   # mdadm /dev/md0 --fail /dev/sdc1

3. Remove from the array.

   # mdadm /dev/md0 --remove /dev/sdc1

4. Physically replace the disk. With hot-swap backplanes, pull and insert. Without, shut down, replace, and boot.
5. Partition the new disk identically. Copy the partition table from a surviving member.

   # MBR
   # sfdisk -d /dev/sda | sfdisk /dev/sdc
   # GPT (sgdisk is safer for GPT)
   # sgdisk --replicate=/dev/sdc /dev/sda
   # sgdisk --randomize-guids /dev/sdc

6. Add the new partition to the array. If a hot spare is configured, MD may rebuild automatically onto the spare; this --add makes the replacement a new spare.

   # mdadm /dev/md0 --add /dev/sdc1

7. Watch the rebuild.

   # watch -n1 'cat /proc/mdstat'

Rebuild Speed Tuning

MD throttles resync/rebuild/check so it doesn't starve foreground I/O. Two knobs in /proc/sys/dev/raid/:

# echo 500000 > /proc/sys/dev/raid/speed_limit_max
# echo 50000  > /proc/sys/dev/raid/speed_limit_min
# Persist via sysctl.d:
# cat > /etc/sysctl.d/90-mdadm.conf <<EOF
dev.raid.speed_limit_min = 50000
dev.raid.speed_limit_max = 500000
EOF

Stripe cache size (/sys/block/mdN/md/stripe_cache_size, for RAID 5/6) controls how many stripes are held in memory during writes. Default 256; bumping to 4096 or 8192 often helps RAID 5/6 write throughput at the cost of RAM (stripe_cache_size × chunk × (N−1) bytes).

Scrubbing — Periodic Integrity Checks

Scrubbing reads every stripe and verifies that data and parity agree (or that mirrors match). Without scrubbing, a silent read error on an otherwise-unused sector only gets noticed during rebuild — by which time it's too late.

# cat /sys/block/md0/md/mismatch_cnt
0

On Debian/Ubuntu, /etc/cron.d/mdadm runs a monthly checkarray on the first Sunday. On RHEL, mdcheck timers do the same. Non-zero mismatch_cnt after a check is a red flag worth investigating, especially on RAID 1 where it often indicates a flaky member.

Growing and Reshaping Arrays

MD supports online reshape — adding disks, changing RAID level, expanding after a disk swap — without taking the array offline. It is slow (hours to days for a full reshape) and risky.

# mdadm /dev/md0 --add /dev/sde1
# mdadm --grow /dev/md0 --raid-devices=5 \
    --backup-file=/root/md0-reshape.bak

# mdadm --grow /dev/md0 --size=max
# resize2fs /dev/md0            # or xfs_growfs, or pvresize, etc

Write-Intent Bitmap

A bitmap that tracks which regions of the array have pending writes. On unclean shutdown or a disk re-adding after brief disconnection, MD only has to resync the dirty regions instead of the entire array — typically a few GB instead of many TB.

# mdadm --grow --bitmap=internal /dev/md0
# mdadm --grow --bitmap=none     /dev/md0   # remove it

Journal Device — Closing the Write Hole

For RAID 4/5/6, MD supports attaching a dedicated journal device (typically a small, fast SSD with power-loss protection). Stripe updates are written to the journal first, then committed to the array. On crash, the journal is replayed — no partial stripe, no write hole.

# mdadm --create /dev/md0 \
    --level=5 --raid-devices=4 \
    --write-journal=/dev/nvme0n1p1 \
    /dev/sd[abcd]1

# mdadm --grow /dev/md0 --add-journal=/dev/nvme0n1p1

Layered Stacks — Common Patterns

MD rarely stands alone. Typical compositions:

Pattern A — Classic NAS

Simple, predictable. Good for single-purpose bulk storage.

Pattern B — Encrypted with LVM flexibility

RAID first, encryption once on the aggregate device (single key, parallelism across CPU cores), LVM for flexible volume management. Standard on Debian/Ubuntu encrypted installs with RAID.

Pattern C — LVM-managed RAID (no mdadm)

LVM can create RAID directly using the dm-raid kernel target (same MD code underneath). Unified management via lvcreate --type raid5. Fewer moving parts for greenfield LVM deployments.

Pattern D — RAID direct to FS (no LVM)

Common for /boot and simple mirrored system drives. Minimum layers, maximum simplicity.

Stopping, Assembling, Zeroing

# umount /mnt/data
# mdadm --stop /dev/md0

# mdadm --assemble --scan                # use mdadm.conf
# mdadm --assemble /dev/md0 /dev/sd[abcd]1   # explicit

# mdadm --stop /dev/md0
# mdadm --zero-superblock /dev/sd[abcd]1