Storage & High Availability
Storage backends, ZFS, Ceph, clustering, and automatic failover
Storage Types
Proxmox supports many storage backends. The choice determines whether you get snapshots, thin provisioning, and whether the storage can be shared across cluster nodes.
| Type | Content Types | Shared | Snapshots | Thin Provision | Notes |
|---|---|---|---|---|---|
| Directory | All (images, ISOs, backups, templates) | No (unless NFS-backed) | qcow2 only | qcow2 only | Simplest — just a path on the filesystem |
| LVM | Disk images, CT rootfs | No | No | No | Raw block devices, good performance |
| LVM-thin | Disk images, CT rootfs | No | Yes | Yes | Thin provisioning via dm-thin — Proxmox default |
| ZFS (local) | All | No | Yes | Yes | Feature-rich, RAM-hungry (1 GB per TB rule) |
| Ceph RBD | Disk images | Yes | Yes | Yes | Distributed block storage, no SPOF |
| CephFS | All | Yes | Yes | Yes | Distributed filesystem, needs MDS |
| NFS | All | Yes | qcow2 only | qcow2 only | Simple shared storage over network |
| iSCSI | Disk images | Yes | No | Depends on target | Block over TCP/IP — SAN access |
| GlusterFS | All | Yes | qcow2 only | qcow2 only | Distributed filesystem (less common now) |
Tip
For a single-node setup, LVM-thin or ZFS are the best choices. For clusters needing shared storage, Ceph (built-in to Proxmox) or NFS (external server) are most common.
Disk Formats
- Simple block image — byte-for-byte disk representation
- Best performance — no translation overhead
- No snapshots, no compression, no encryption
- File size = allocated disk size (no thin provisioning)
- Used natively by LVM, LVM-thin, Ceph RBD, ZFS zvols
- QEMU Copy-On-Write format v2
- Backing files — snapshot chains, linked clones
- Snapshots — internal snapshot support
- Compression — zlib/zstd per-cluster
- LUKS encryption — built-in at-rest encryption
- ~5-10% performance overhead vs raw
- Used on directory/NFS/GlusterFS storage
Note
On LVM-thin, ZFS, and Ceph, snapshots are handled by the storage layer (not qcow2). These backends use raw format and still get snapshots natively. qcow2 snapshots are only needed on directory-based or NFS storage.
ZFS Basics
What is ZFS?
A combined filesystem + volume manager. Manages disks, RAID, filesystems, snapshots, compression, checksums, and caching in a single integrated stack. Originally from Sun/Solaris, now available on Linux via OpenZFS.
ZFS Architecture
RAID Levels (vdev types)
| vdev Type | Parity Disks | Min Disks | Tolerated Failures | Usable Capacity |
|---|---|---|---|---|
| mirror | N/A (mirroring) | 2 | N-1 (in the mirror) | 50% (2-way mirror) |
| raidz1 | 1 | 3 | 1 | (N-1)/N |
| raidz2 | 2 | 4 | 2 | (N-2)/N |
| raidz3 | 3 | 5 | 3 | (N-3)/N |
Key Features
Copy-on-Write (CoW)
Data is never overwritten in place. New blocks are written to free space, then metadata pointers are updated atomically. This makes snapshots instant and free — they just preserve old block pointers.
Checksums
Every block has a checksum stored in its parent block (Merkle tree). Detects silent data corruption (bit rot). With mirrors/raidz, ZFS auto-repairs corrupted blocks.
ARC Cache
Adaptive Replacement Cache — uses available RAM for read caching. Rule of thumb: 1 GB RAM per 1 TB of storage. Can add an L2ARC (SSD) for overflow.
Send/Recv Replication
zfs send streams a snapshot (or incremental delta) to another pool — local or remote.
Basis for backup strategies and disaster recovery.
Common ZFS Commands
# Create a mirrored pool
zpool create tank mirror /dev/sda /dev/sdb
# Create a dataset with compression
zfs create -o compression=lz4 tank/data
# Take a snapshot
zfs snapshot tank/data@2024-01-15
# List snapshots
zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
tank/data@2024-01-15 128K - 1.23G -
# Incremental replication to remote host
zfs send -i tank/data@snap1 tank/data@snap2 | ssh backup zfs recv backup/data
# Pool status (health, errors, scrub progress)
zpool status tank
pool: tank
state: ONLINE
scan: scrub repaired 0B in 01:23:45 with 0 errors
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
Warning
ZFS is RAM-hungry. The ARC will consume most available RAM by default.
For Proxmox, set zfs_arc_max to limit it:
echo "options zfs zfs_arc_max=4294967296" > /etc/modprobe.d/zfs.conf (4 GB limit).
Ceph Integration
What is Ceph?
Distributed storage system with no single point of failure. Data is replicated across multiple nodes and disks. Proxmox has first-class Ceph integration — you can deploy and manage Ceph directly from the Proxmox web UI.
Ceph Components
Ceph Storage in Proxmox
RBD (RADOS Block Device)
- Block storage for VM disks and CT rootfs
- Thin provisioned, snapshots, clones
- Shared across all cluster nodes — enables live migration
- Data striped across OSDs, replicated (default 3x)
- Proxmox creates a pool (e.g.,
rbd-pool) and adds it as storage
CephFS (Ceph Filesystem)
- POSIX filesystem for ISOs, backups, templates, snippets
- Requires at least one MDS daemon
- Shared across all nodes — all nodes see the same files
- Good for backup storage (vzdump), ISO library
- Not recommended for VM disks (use RBD instead)
Self-Healing on OSD Failure
OSD Dies
Disk failure or node crash
MON Detects
Heartbeat timeout, OSD marked "down"
Wait Period
10 min default — allows transient recovery
OSD Marked Out
CRUSH map updated, data re-replicated to surviving OSDs
Recovery Complete
All data back to target replication count (3x)
# Create Ceph OSDs from Proxmox CLI
pveceph osd create /dev/sdc
pveceph osd create /dev/sdd
# Create a Ceph pool
pveceph pool create vm-pool --pg_autoscale_mode on --size 3 --min_size 2
# Check Ceph cluster health
ceph status
cluster:
id: a1b2c3d4-...
health: HEALTH_OK
services:
mon: 3 daemons, quorum pve1,pve2,pve3
mgr: pve1(active), standbys: pve2
osd: 12 osds: 12 up, 12 in
data:
pools: 2 pools, 128 pgs
objects: 4.2k objects, 16 GiB
usage: 52 GiB used, 3.5 TiB / 3.6 TiB avail
Clustering
Corosync
Cluster Communication Layer
- Membership — tracks which nodes are alive (heartbeats)
- Messaging — reliable multicast/unicast between nodes
- Quorum — calculates whether the cluster has majority
- Uses UDP (default port 5405) on a dedicated cluster network
- Config:
/etc/corosync/corosync.conf(auto-managed by Proxmox)
pmxcfs — Proxmox Cluster Filesystem
How /etc/pve/ Works
/etc/pve/is a FUSE mount backed by a local SQLite database- The database is replicated in real time via Corosync
- All nodes see the same configs — VM configs, firewall rules, HA state, user permissions
- Max file size: 128 KB (it is a config store, not a general filesystem)
- If a node loses quorum,
/etc/pve/becomes read-only
/etc/pve/
corosync.conf
storage.cfg
user.cfg
datacenter.cfg
firewall/
cluster.fw
nodes/
pve1/
qemu-server/100.conf
lxc/200.conf
pve2/
qemu-server/101.conf
qemu-server/
100.conf
101.conf
lxc/
200.conf
Quorum
Why Quorum Matters
A cluster requires a majority of nodes to agree before making changes. This prevents split-brain — two halves of a partitioned cluster both thinking they are in charge.
| Cluster Size | Quorum (majority) | Max Failures |
|---|---|---|
| 2 nodes | 2 | 0 (need QDevice for 1-node tolerance) |
| 3 nodes | 2 | 1 |
| 4 nodes | 3 | 1 |
| 5 nodes | 3 | 2 |
| 7 nodes | 4 | 3 |
Warning
Without quorum: the partition goes read-only. VMs keep running but you cannot start, stop, or migrate them. This is by design — it prevents both halves from modifying the same VM.
Tip
A 2-node cluster has no fault tolerance by default (quorum = 2). Add a QDevice (lightweight third-party voter, e.g., a Raspberry Pi) to achieve 1-node tolerance.
Fencing (STONITH)
Shoot The Other Node In The Head
Before migrating a failed node's VMs, the cluster must guarantee the failed node is truly off. Otherwise, two copies of the same VM could run simultaneously — causing data corruption.
Hardware Fencing
- IPMI/BMC — remotely power off via out-of-band management
- HP iLO — Integrated Lights-Out
- Dell iDRAC — Integrated Dell Remote Access Controller
- PDU — Smart power distribution unit (cut power to port)
- Most reliable — works even if OS is hung
Software Fencing (Watchdog)
- Proxmox default: software watchdog
- HA manager resets the watchdog timer periodically
- If the node becomes unresponsive, watchdog fires and reboots the node
- Hardware watchdog (
/dev/watchdog) is more reliable than software - Sufficient for many deployments
High Availability
HA Manager
The HA manager continuously monitors HA-managed resources (VMs and CTs). When a node fails, it coordinates fencing and automatic failover.
Failover Sequence
- Node Failure Detected
Corosync stops receiving heartbeats from the failed node. After timeout (typically a few seconds), the node is declared dead.
- Remaining Nodes Achieve Quorum
The surviving nodes verify they still have majority. If quorum is lost (e.g., 1 of 2 nodes), failover does not proceed — safety first.
- Failed Node Fenced (STONITH)
The cluster ensures the failed node is truly powered off. Via IPMI, iLO, iDRAC, or watchdog reboot. This prevents two instances of the same VM from running simultaneously.
- HA Manager Identifies Affected Resources
All VMs and CTs that were running on the failed node and are marked as HA-managed are queued for restart on surviving nodes.
- Resources Started on Other Nodes
VMs/CTs are started on surviving nodes based on HA group preferences and node priority. Requires shared storage (Ceph, NFS, iSCSI) — the disk must be accessible from the new node.
- Recovery Complete
Total failover time: ~2-5 minutes (fence timeout + VM boot time). This is not live migration — VMs are cold-started, so there is downtime.
Note
HA failover is not live migration. The VMs on the failed node crash — they are then cold-booted on a surviving node. Applications inside the VM experience a hard restart. For zero-downtime, you need application-level HA (e.g., database replication, load-balanced web servers).
HA Groups
Controlling Placement
HA groups define which nodes a resource can run on and with what priority.
# Create an HA group
ha-manager groupadd prod-group --nodes pve1:2,pve2:1,pve3:1 --restricted --nofailback
# pve1 has priority 2 (preferred), pve2 and pve3 have priority 1
# --restricted: resource can ONLY run on nodes in this group
# --nofailback: don't auto-migrate back when preferred node recovers
# Add a VM to HA management
ha-manager add vm:100 --group prod-group --state started --max_restart 3 --max_relocate 2
Recovery Policies
| Policy | Behavior | Use Case |
|---|---|---|
| migrate | Live migrate to another node (graceful, minimal downtime) | Planned maintenance — drain a node before rebooting |
| relocate | Shut down on current node, start on another node (cold move) | When live migration is not possible (local storage, incompatible CPUs) |
| freeze | Do not attempt to recover — leave the resource in failed state | Resources that should not auto-restart (manual intervention required) |
HA State Diagram
# Check HA status for all resources
ha-manager status
quorum OK, master pve1
vm:100 pve1 started
vm:101 pve2 started
ct:200 pve1 started
# Manually migrate an HA resource
ha-manager migrate vm:100 pve3
# Set a resource to freeze (prevent auto-recovery)
ha-manager set vm:100 --state freeze
Tip
For production HA, use at least 3 nodes (for quorum), shared storage (Ceph recommended), and configure hardware fencing (IPMI/iLO/iDRAC) for reliable STONITH.
