ZFS
Drives
Disk Drives are not all created as equals - by far not.
For example, mixing a (typically cheaper) Shingled Magnetic Recording (SMR) drive into a ZFS mirror with a (better) Conventional Magnetic Recording (CMR) drive is a massive anti-pattern; this can cause writes at expected roughly ~200MB/s-ish to drop 10x down to ~20MB/s!
And putting a 5400-RPM and a 7200-RPM drive next to each other in a chassis creates an asynchronous vibration pattern.
Instead of buying different brands so they don't all fail at the exact same time, buy drives of similar-ish NAS drive models from the same brand. To break up the "same batch curse", while keeping the underlying drive controllers, RPMs, firmware behavior, and physical geometry identical, you could buy them from different vendors to get a few months variation; when buying 2nd hand, perhaps even some years apart.
SMART
First, check your disks's health!
TL;DR
sudo zpool create \
-o autoexpand=off \
-O compression=lz4 \
-O recordsize=256K \
-O dnodesize=auto \
-o ashift=12 \
-O atime=off \
-O acltype=off \
-O xattr=sa \
-O normalization=formD \
-O encryption=on \
-O keyformat=passphrase \
-O keylocation=prompt \
-O mountpoint=none \
bardioc mirror \
/dev/disk/by-id/ata-WDC_WD80PUZX-64NEAY0_VK0GUMLY \
/dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840N805
sudo mkdir -p /bardioc/private
sudo bash -c 'chattr +i /bardioc/private'
sudo zfs create -o mountpoint=legacy bardioc/private
sudo mkdir /bardioc/private/test
sudo chown $USER: /bardioc/private/test
echo "hello, world" >/bardioc/private/test/hello.txt
PS: The ZFS pool name tank often seen in other examples is just a convention, nothing more!
Pools
For ashift, just always keep ashift=12; do NOT run lsblk -t to adjust for PHY-SEC of 4096 with ashift=12 (2^12 = 4096) on HDDs, and for 512 use ashift=9 (2^9 = 512); see the ZFS documentation for more details about why doing that would be is wrong.
It's highly recommended to create a reservation to prevent performance deterioration when space gets real tight, but it doesn't have to be 20% anymore; for a 8 TB mirror just 100 GB is good enough (which would allow you to zfs destroy pool/reserved, to temporarily unblock a tight spot to figure out a longer term solution; if it were ever needed), like so:
sudo zfs create \
-o encryption=off \
-o refreservation=100G \
-o mountpoint=none \
-o canmount=off \
"bardioc/reserved"
Datasets
Create datasets in a pool with zfs create as in the TL;DR above.
NixOS Declarative Mounts
To manage ZFS mount points declaratively in NixOS (e.g. in your fileSystems configuration), follow these recommendations:
mountpoint=legacy: By default, ZFS manages dataset mounting itself using the dataset'smountpointproperty. However, the NixOS systemd-based mounting generator requires standard mount commands. To allow NixOS to mount datasets viafileSystemsentries, set the ZFS dataset mountpoint tolegacy:
sudo zfs set mountpoint=legacy bardioc/private
-
options = [ "nofail" ];: When mounting optional datasets (like data disks or backup targets that might be missing or disconnected), always includeoptions = [ "nofail" ];in your NixOSfileSystemsdefinition. Withoutnofail, systemd treats the mount as a critical boot dependency, causing the system to crash or enter an infinite emergency mode loop if the ZFS pool/disk is unavailable. Withnofail, systemd will gracefully skip the mount if the pool import fails or times out. -
Manual Mounting when
mountpoint=legacy: Datasets withmountpoint=legacycannot be mounted usingzfs mount(which returnscannot mount '...': legacy mountpoint; use mount(8) to mount this filesystem). Instead, use standard Linuxmount(8)commands or systemd: -
Using systemd service (Recommended): NixOS automatically generates systemd mount units for entries in
fileSystems. For example, a dataset configured for/bardioc/publicwill have a unit namedbardioc-public.mount. You can check status or start/restart it directly:sudo systemctl status bardioc-public.mount sudo systemctl start bardioc-public.mount -
Using standard
mount: Alternatively, specifyzfsas the filesystem type:sudo mount -t zfs bardioc/public /bardioc/public
Import
$ zpool status
no pools available
$ sudo zpool import
$ sudo zpool import bardioc
If encrypted datasets were not loaded automatically, load their keys:
sudo zfs load-key bardioc/private
Mount legacy datasets either via systemd or standard mount(8):
sudo systemctl start bardioc-private.mount
Or manually:
sudo mount -t zfs bardioc/private /bardioc/private
ZFS-8000-4J
documents how it handles if a device is missing; hint: it still works - that's the point!
(On reconnecting the same previously missing drive it will have to scrub for a few hours.)
See here how to automatically have the pool imported at boot, with passphrase.
Replace
It's faster, but not required, to replace a disk in a pool with all datasets unmounted.
$ sudo zpool status
pool: bardioc
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
scan: scrub repaired 0B in 1 days 00:34:11 with 0 errors on Sun Jul 12 17:52:43 2026
config:
NAME STATE READ WRITE CKSUM
bardioc DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-WDC_WD80PUZX-64NEAY0_VK0GUMLY ONLINE 0 0 0
2121792862421408442 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840N805-part1
errors: No known data errors
To replace this unavailable (removed) drive with another one, we can simply:
$ sudo zpool replace bardioc /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840N805-part1 /dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-1F0XZZUU
$ zpool status
pool: bardioc
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sat Jul 25 14:17:47 2026
631G / 3.18T scanned at 39.5G/s, 0B / 3.18T issued
0B resilvered, 0.00% done, no estimated completion time
config:
NAME STATE READ WRITE CKSUM
bardioc DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-WDC_WD80PUZX-64NEAY0_VK0GUMLY ONLINE 0 0 0
replacing-1 DEGRADED 0 0 0
2121792862421408442 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840N805-part1
ata-WDC_WD80EFPX-68C4ZN0_WD-1F0XZZUU ONLINE 0 0 0
errors: No known data errors
After it's done scanning (which is relatively quick) and started resilvering (which takes a long time), this will show:
$ zpool status
pool: bardioc
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sat Jul 25 14:17:47 2026
3.16T / 3.18T scanned at 3.95G/s, 22.6G / 3.18T issued at 28.2M/s
22.6G resilvered, 0.69% done, 1 days 08:37:09 to go
config:
NAME STATE READ WRITE CKSUM
bardioc DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-WDC_WD80PUZX-64NEAY0_VK0GUMLY ONLINE 0 0 0
replacing-1 DEGRADED 0 0 0
2121792862421408442 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840N805-part1
ata-WDC_WD80EFPX-68C4ZN0_WD-1F0XZZUU ONLINE 0 0 0 (resilvering)
errors: No known data errors
After the resilvering is done, the pool is already back in ONLINE state. It will automatically also scrub the pool:
pool: bardioc
state: ONLINE
scan: scrub in progress since Sat Jul 25 21:58:47 2026
3.18T / 3.18T scanned, 2.59T / 3.17T issued at 146M/s
0B repaired, 81.48% done, 01:10:36 to go
config:
NAME STATE READ WRITE CKSUM
bardioc ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD80PUZX-64NEAY0_VK0GUMLY ONLINE 0 0 0
ata-WDC_WD80EFPX-68C4ZN0_WD-1F0XZZUU ONLINE 0 0 0
errors: No known data errors
And eventually this scrub will also be completed:
pool: bardioc
state: ONLINE
scan: scrub repaired 0B in 06:36:24 with 0 errors on Sun Jul 26 04:35:11 2026
config:
NAME STATE READ WRITE CKSUM
bardioc ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD80PUZX-64NEAY0_VK0GUMLY ONLINE 0 0 0
ata-WDC_WD80EFPX-68C4ZN0_WD-1F0XZZUU ONLINE 0 0 0
errors: No known data errors
Status
zfs list
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
bardioc 888K 7.14T 192K none
bardioc/private 192K 7.14T 192K legacy
zpool status
$ zpool status
pool: bardioc
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
bardioc ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD80PUZX-64NEAY0_VK0GUMLY ONLINE 0 0 0
ata-ST8000AS0002-1NA17Z_Z840N805 ONLINE 0 0 0
errors: No known data errors
Note that READ, WRITE, and CKSUM represent the count of I/O errors for read operations, write operations, and checksum validation failures, respectively. Ideally, these should all be zero, indicating that the pool is healthy and there are no issues with the underlying storage devices. If you see non-zero values, it may indicate potential problems with the pool or the disks, and further investigation would be warranted to identify and resolve any issues.
zpool history
sudo zpool history bardioc
Encryption
Encryption is enabled in the TL;DR above; the key can be changed like this:
zfs change-key bardioc
zfs change-key -o keyformat=raw -o keylocation=zfs-key bardioc
It's instantaneous, and does not need to rewrite or re-encrypt any of your data.
Snapshots
We recommend using Sanoid for Snapshots, like this:
services.sanoid = {
enable = true;
interval = "*:0/15"; # For "frequently" snapshots.
datasets = lib.genAttrs (
map (fs: fs.device) (
lib.filter (fs: fs.fsType == "zfs") (lib.attrValues config.fileSystems)
)
) (_name: {
useTemplate = [ "default" ];
recursive = true;
});
templates = {
"default" = {
frequently = 8; # Keep this many snapshots @15min frequency.
hourly = 48;
daily = 90;
weekly = 30;
monthly = 24;
yearly = 100;
};
};
};
Difference between NixOS services.sanoid and autoSnapshot.enable (zfstools)
We recommend sanoid over zfstools because:
- Integration:
sanoidintegrates cleanly withsyncoidfor ZFS replication. - Configuration:
sanoidis configured purely declaratively in the Nix configuration viaservices.sanoid.datasets(or dynamically mapped fromconfig.fileSystems, as shown above).zfstoolsrelies on setting ZFS properties on datasets and requires you to imperatively runzfs set com.sun:auto-snapshot=true <dataset>on the CLI. - Policy Flexibility:
sanoidsupports customizable templates, retention policies (e.g. keeping varying numbers of hourly, daily, monthly snapshots), and pre/post-snapshot scripts.zfstoolshas hardcoded retention defaults that are harder to customize. - Maintenance:
sanoid(written in Perl) is actively maintained and the industry standard for ZFS snapshot management.zfstoolsis an older, relatively inactive Ruby-based tool.
Destroying Snapshots
Because ZFS snapshots are read-only copy-on-write references to blocks on disk, deleting files from a dataset (e.g. via rm -rf) will not free disk space if those files are still referenced by any existing snapshots. The physical space will only be reclaimed once all snapshots containing those files are destroyed or naturally expire.
To manually delete a single specific snapshot, use:
sudo zfs destroy bardioc/private@snapshot-name
If you want to destroy all snapshots up to a certain point in time (for example, to reclaim space immediately after a large deletion), you can destroy a range using the % syntax:
# Syntax: zfs destroy dataset@oldest_snap%newest_snap
sudo zfs destroy bardioc/private@autosnap_2026-01-01_00:00:00_yearly%autosnap_2026-07-19_18:00:00_hourly
Freeing Space
When destroying a snapshot containing large amounts of unique data, the operation might take time as ZFS traverses metadata and frees blocks. However, ZFS processes this cleanup asynchronously in the background.
zfs destroy is completely atomic and transactional (handled via ZFS Transaction Groups). Even if you interrupt the CLI command (e.g. by pressing Ctrl-C), no data or filesystem corruption will occur. The ZFS kernel module will either finish registering the deletion or safely roll it back.
You can monitor the background freeing progress with:
zpool list -o name,freeing
Or query the property directly for a specific pool:
zpool get freeing bardioc
Cache
TODO L2ARC with an SSD, for read caching? Or will it trash the SSD with too much writes if it's an old one?
Backup
rsync
rsync -aHAXx --info=progress2 --inplace /mnt/source/ /bardioc/private/...
-aHAXxThe Bulletproof Archive Mode:-
-a(archive) copies recursively and preserves symlinks, devices, permissions, and modification times. -
-Hpreserves hard links. -
-Apreserves ACLs (important since you set up posixacl). -
-Xpreserves extended attributes (xattr=sa). -
-xstays on a single filesystem (prevents rsync from accidentally backing up virtual dirs like /proc or crossing into other mounts). -
--inplace(Crucial for ZFS performance): By default, rsync creates a hidden temporary file, copies the data, and then moves it into place. --inplace forces rsync to write directly to the target file. On ZFS, this avoids massive write amplification, bypasses unnecessary fragmentation, and reduces transactional overhead. -
--info=progress2: Instead of scrolling a wall of text for every single tiny file (which slows down the terminal emulator), this gives you a single, modern, live-updating progress bar for the entire transfer size.
Optionally, depending on the use case, you may also want to add:
--numeric-ids: Tells rsync to map user/group IDs by their raw numbers rather than trying to resolve usernames. This is much cleaner when moving data between different OS installations or environments.
Replication
TODO syncoid
Google Drive
TODO How to backup to Google Drive? https://github.com/someone1/zfsbackup-go,
Monitoring
TODO https://github.com/pdf/zfs_exporter
Deduplication
Permanent ZFS block-level deduplication is very resource intensive. Unless you really know what you're doing, you most likely don't want it.
You may however occasionally wish to manually use a (not ZFS specific) file-level deduplication tool?
Scrub
Automatic scrubbing is enabled, note:
$ zpool status
pool: bardioc
state: ONLINE
scan: scrub in progress since Sat Jun 6 16:56:23 2026
852G / 852G scanned, 28.7G / 852G issued at 70.3M/s
0B repaired, 3.37% done, 03:20:02 to go
config:
NAME STATE READ WRITE CKSUM
bardioc ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD80PUZX-64NEAY0_VK0GUMLY ONLINE 0 0 0
ata-ST8000AS0002-1NA17Z_Z840N805 ONLINE 0 0 0
errors: No known data errors
and later, hopefully:
$ zpool status
pool: bardioc
state: ONLINE
scan: scrub repaired 0B in 03:14:43 with 0 errors on Sat Jun 6 20:11:06 2026
config:
NAME STATE READ WRITE CKSUM
bardioc ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD80PUZX-64NEAY0_VK0GUMLY ONLINE 0 0 0
ata-ST8000AS0002-1NA17Z_Z840N805 ONLINE 0 0 0
errors: No known data errors
You can also manually trigger scrubbing with zpool scrub -a -w.
UI
- https://github.com/webzfs/webzfs with https://github.com/webzfs/webzfs/issues/162
- https://github.com/45Drives/cockpit-zfs or https://github.com/brycelarge/cockpit-zfs-manager
- https://github.com/ad4mts/zfdash
- https://github.com/macgaver/zfsnas-chezmoi
Boot
NixOS can be made to boot from ZFS directly (modulo one VFAT for UEFI)
Distros
ZFS is well integrated into NixOS; on other distros it may be slightly more of a PITA.
https://www.truenas.com is a storage distro built around ZFS.