Recovery · XB Field Notes

ZFS snapshots give the filesystem a rewind button

Tue, 21 Mar 2028 00:00:00 +0000

A ZFS snapshot preserves a point-in-time dataset view through copy-on-write references instead of a full duplicate of its data. Existing blocks remain available while references need them. Creation still has metadata and synchronization work; neither zero allocation nor completion before a shell redraw is a measured guarantee. This review did not run ZFS or recover the earlier claimed fixture’s raw output.

That makes retained history useful for local recovery, but every snapshot of tank/data remains in the same pool. A failure that makes that pool unavailable can also make its snapshots unavailable. This is a shared failure boundary, not a claim that every controller fault atomically destroys all blocks. Independent backups or replication need their own retention and verified recovery.

A name for one consistent tree

Snapshots atomically capture the dataset’s filesystem state; recursive -r snapshots include descendants at the same instant. Without recursion, child datasets are omitted rather than automatically captured later. Filesystem consistency is not an application checkpoint: database buffers, logs and files outside the selected datasets still require the application’s supported backup or crash-recovery procedure. Coordinate all required datasets before treating the set as recoverable.

Names such as tank/data@daily-2026-09-16 can help retention, alongside creation times, properties and backup records. Naming alone does not prove eligibility for deletion. zfs list -t snapshot -o name,used,refer,creation -s creation inspects snapshots and their accounting; dependencies and holds must also be checked before a pruning decision.

Where the space is, in zfs list -o space

zfs list -o space tank/data selects NAME, AVAIL, USED, USEDSNAP, USEDDS, USEDREFRESERV and USEDCHILD; it does not include REFER. USED decomposes into the four used-by categories, not REFER plus USEDSNAP. REFER is all data accessible through the dataset and can include shared blocks. Snapshot USED measures uniquely referenced space; deleting another snapshot can change it, so individual values do not simply sum to all recoverable space.

The earlier thirty-snapshot, 64 MiB dataset and 2 MiB daily-churn figures lack a recovered ZFS trace. The 60 MiB versus 1920 MiB comparison is illustrative arithmetic, not verified allocation. Metadata, compression, block size, sharing and the exact retention timeline affect actual cost. A recent snapshot may have little unique data while the whole retained set still prevents substantial reclamation.

Holds, diff, and comparing versions

A user hold prevents destruction until released. zfs hold keep tank/data@daily-2026-09-16 sets one; zfs holds tank/data@daily-2026-09-16 inspects it, and zfs release keep tank/data@daily-2026-09-16 releases that tag. Holds can protect an in-progress transfer, but clone-origin dependencies are separate and need not appear as named hold tags. Release only after understanding the dependent operation.

zfs diff tank/data@daily-2026-09-15 tank/data@daily-2026-09-16 reports changes between the retained versions: created, removed, modified and renamed paths. Filesystem snapshots are mounted on demand beneath .zfs/snapshot/<name>; snapdir=visible exposes that directory in listings. Snapshot access and a destructive dataset rollback are different recovery choices, and neither verifies application correctness by itself.

When it breaks

A successful send/receive into independently protected storage can survive loss of the source pool. The shown incremental transfer requires an existing destination whose latest snapshot matches the source base; a bare incremental stream is not a complete standalone backup. A second pool on the same machine can still share power, controller or administrator risks. Verify received data and a usable restore before claiming protection.

Growing USEDSNAP warrants capacity and retention review, but eighty-percent occupancy is not a universal write-failure threshold. Quotas, reservations and actual pool availability matter. A clone depends on its origin snapshot; deleting that origin or rolling back past newer dependent versions needs deliberate dependency handling. Rollback can discard newer data, and destroying clones is not the only possible recovery strategy.

ZFS history has dependencies — mechanism diagram.

Check it quickly

This unexecuted standalone Bash recipe requires exclusively controlled disposable datasets, existing source snapshots, and a receiver previously seeded from the base. The destination must be unchanged since that base, mounted at the bound path without intervening mounts or symlinks; the hold tag must be unused. Before taking the target snapshot, prepare regular-file probe.txt in the source and keep a trusted external copy at expected. Listing is recursive; transfer covers only tank/data, not descendants.

(
 set -euo pipefail
 src='tank/data'
 dst='backup/data'
 base='daily-2026-09-15'
 next='daily-2026-09-16'
 tag='example-transfer'
 mount='/mnt/zfs-lab/received'
 expected='/mnt/zfs-lab/expected-v2.txt'

 zfs list -r -t snapshot -o name,used,refer,creation -s creation "$src" "$dst"
 zfs list -o name,avail,used,usedsnap,usedds,usedrefreserv,usedchild,refer "$src"
 zfs holds "$src@$base" "$src@$next"
 zfs diff "$src@$base" "$src@$next"

 latest=$(zfs list -H -r -t snapshot -o name -s createtxg "$dst" |
 awk -F@ -v dataset="$dst" '$1 == dataset { last=$0 } END { print last }')
 test "$latest" = "$dst@$base"
 source_guid=$(zfs get -Hp -o value guid "$src@$base")
 target_guid=$(zfs get -Hp -o value guid "$dst@$base")
 test -n "$source_guid"
 test "$source_guid" = "$target_guid"
 target_mount=$(zfs get -H -o value mountpoint "$dst")
 target_mounted=$(zfs get -H -o value mounted "$dst")
 test "$target_mount" = "$mount"
 test "$target_mounted" = yes
 test -f "$expected"
 test ! -L "$expected"

 zfs hold "$tag" "$src@$base" "$src@$next"
 zfs send -i "$src@$base" "$src@$next" | zfs receive "$dst"
 source_guid=$(zfs get -Hp -o value guid "$src@$next")
 target_guid=$(zfs get -Hp -o value guid "$dst@$next")
 test -n "$source_guid"
 test "$source_guid" = "$target_guid"
 test -f "$mount/.zfs/snapshot/$next/probe.txt"
 test ! -L "$mount/.zfs/snapshot/$next/probe.txt"
 cmp -- "$expected" "$mount/.zfs/snapshot/$next/probe.txt"
)

The latest receiver snapshot and base GUID must match; equal names alone are insufficient. Failed checks stop the subshell, and pipefail exposes either transfer failure. Received GUID and fixture-byte comparisons still do not establish a valid application restore: verify the complete manifest, metadata and application recovery separately. Any holds created remain after failure or success; after validation and dependency review, release only this exercise’s tag with zfs release example-transfer tank/data@daily-2026-09-15 tank/data@daily-2026-09-16. No force rollback or automatic cleanup is included.

Sources

zfs-snapshot(8), snapshot creation semantics and recursive snapshots, checked 2026-09-16.
zfs-list(8), the space column set and separate REFER property, checked 2026-09-16.
zfs-hold(8), holds, zfs holds, and the EBUSY returned by destroy, checked 2026-09-16.
zfs-send(8), incremental send streams and their base-snapshot requirements, checked 2026-09-16.
zfsprops(7), used-by decomposition, shared references, quotas and reservations, checked 2026-10-18.
zfs-clone(8), clone dependence on its origin snapshot, checked 2026-10-18.
zfs-rollback(8), discarded newer data and dependent clone handling, checked 2026-10-18.
zfs-receive(8), matching destination base for incremental receive, checked 2026-10-18.
PostgreSQL filesystem backup, consistent snapshot scope, WAL and multi-volume recovery requirements, checked 2026-10-18.
ZFS concepts, clone dependency on its origin snapshot and shared storage boundaries, checked 2026-10-18.
zfs-get(8), exact property reads used for GUID and mountpoint checks, checked 2026-10-19.

Backups are untested hope without proof

Wed, 02 Jun 2027 00:00:00 +0000

A completed backup job establishes that the backup process produced an artifact. A restore rehearsal establishes whether that artifact, the available tooling, and the required environment can produce a usable system. These are different claims. Recovery evidence should identify the backup used, the state recovered, the dependencies supplied, and the time needed to make the result useful to the application.

When it breaks

A valid database archive can depend on objects outside its scope. PostgreSQL pg_dump exports one database, while roles are cluster-wide objects. Restoring a dump with grants into an independent cluster can fail because the referenced role does not exist. Rehearsing only in the original cluster can hide that dependency because the missing recovery material is already present there.

Restore errors also need deliberate handling. By default, pg_restore can continue after SQL errors and report their count at the end. A half-restored schema should not be accepted merely because some tables can be queried. For this small fixture, a single transaction makes success atomic and makes an expected failure leave no partial restored tables.

The correct recovery point is another requirement. A perfectly restored backup may intentionally lack changes made afterward. A checksum proves agreement with the recorded artifact, not that the artifact includes the newest business event. Likewise, a row count alone can match while amounts, relationships, or privileges are wrong. Choose checks that express the application’s meaning.

A checksum does not supply a missing cluster role. Restore the same archive into a fresh target, then check the intended business state, constraints and effective privileges. The source’s PostgreSQL drill is not rerun.

Check it quickly

Run this script as a non-root Linux account with PostgreSQL 18 server and client programs and sha256sum available. It was executed on PostgreSQL 18.6. It creates separate temporary source and target clusters with private Unix sockets and no TCP listeners. The fixture role cannot log in; its grants are tested through controlled role switching rather than through real credentials.

#!/bin/sh
set -eu
umask 077
lab=$(mktemp -d /tmp/xbcatm-restore.XXXXXX)
mkdir "$lab/socket"
source=$lab/source
target=$lab/target
cleanup() {
 for cluster in "$target" "$source"; do
 if test -f "$cluster/postmaster.pid"; then
 pg_ctl -D "$cluster" -m fast -w stop >/dev/null
 fi
 done
}
trap cleanup EXIT
for cluster in "$source" "$target"; do
 initdb -D "$cluster" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject >/dev/null
 cat >> "$cluster/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$lab/socket'
CONF
done
printf 'port = 56432\n' >> "$source/postgresql.conf"
printf 'port = 56433\n' >> "$target/postgresql.conf"
pg_ctl -D "$source" -l "$lab/source.log" -w -t 15 start >/dev/null
pg_ctl -D "$target" -l "$lab/target.log" -w -t 15 start >/dev/null
export PGHOST=$lab/socket PGPORT=56432 PGDATABASE=postgres PGUSER=$(id -un)
export PGCONNECT_TIMEOUT=2
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
CREATE ROLE backup_reader NOLOGIN;
CREATE TABLE customers (id integer PRIMARY KEY,name text NOT NULL);
CREATE TABLE orders (id integer PRIMARY KEY,customer_id integer REFERENCES customers,amount_cents integer CHECK (amount_cents>0));
INSERT INTO customers VALUES (1,'Ada');
INSERT INTO orders VALUES (10,1,4210),(11,1,1865);
CREATE VIEW order_totals AS SELECT c.name,sum(o.amount_cents) AS total_cents FROM customers c JOIN orders o ON o.customer_id=c.id GROUP BY c.id,c.name;
GRANT SELECT ON customers,orders,order_totals TO backup_reader;
SQL
pg_dump -Fc -w -f "$lab/snapshot.dump" postgres
sha256sum "$lab/snapshot.dump" > "$lab/snapshot.sha256"
psql -X -v ON_ERROR_STOP=1 -qAt -c 'INSERT INTO orders VALUES (12,1,1000)'
printf 'source_after_backup_orders '
psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT count(*) FROM orders'
sha256sum -c "$lab/snapshot.sha256"
if PGPORT=56433 pg_restore --single-transaction --exit-on-error -w -d postgres "$lab/snapshot.dump" >"$lab/missing-role.log" 2>&1; then
 echo 'restore unexpectedly ignored a missing role' >&2
 exit 1
fi
grep -F 'role "backup_reader" does not exist' "$lab/missing-role.log"
test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT to_regclass('public.orders') IS NULL")" = t
echo failed_restore_rolled_back
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'CREATE ROLE backup_reader NOLOGIN'
PGPORT=56433 pg_restore --single-transaction --exit-on-error -w -d postgres "$lab/snapshot.dump"
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
SELECT count(*) FROM orders;
SELECT name,total_cents FROM order_totals;
SELECT has_table_privilege('backup_reader','orders','SELECT'),has_table_privilege('backup_reader','orders','INSERT');
DO $$ BEGIN
 IF (SELECT count(*) FROM orders)<>2 OR
 (SELECT string_agg(id||':'||amount_cents,',' ORDER BY id) FROM orders) IS DISTINCT FROM '10:4210,11:1865' OR
 (SELECT total_cents FROM order_totals WHERE name='Ada') IS DISTINCT FROM 6075 OR
 (SELECT count(*) FROM orders o LEFT JOIN customers c ON c.id=o.customer_id WHERE c.id IS NULL)<>0 THEN
 RAISE EXCEPTION 'restored business data mismatch';
 END IF;
END $$;
DO $$ BEGIN
 BEGIN
 INSERT INTO orders VALUES (98,999,100);
 RAISE EXCEPTION 'restored foreign key unexpectedly accepted an orphan';
 EXCEPTION WHEN foreign_key_violation THEN
 RAISE NOTICE 'restored foreign key rejected orphan';
 END;
END $$;
SET ROLE backup_reader;
SELECT id,amount_cents FROM orders ORDER BY id;
DO $$ BEGIN
 BEGIN
 INSERT INTO orders VALUES (99,1,100);
 RAISE EXCEPTION 'read-only role unexpectedly wrote data';
 EXCEPTION WHEN insufficient_privilege THEN
 RAISE NOTICE 'read-only role rejected write';
 END;
END $$;
SQL
printf 'lab_directory %s\n' "$lab"

The source contains one customer, two orders, a totals view, a foreign key, and a read-only role grant. A custom-format dump captures that state. The source then receives a third order. Its printed count of three confirms that later activity exists outside the chosen backup point, rather than allowing a test to confuse the current source with the saved snapshot.

The checksum check succeeds, but the first restore must still fail because backup_reader is absent on the target. The script checks that specific error and verifies the orders table was rolled back. This is the dependency failure that an artifact-only check would miss. It then creates the documented fixture role and repeats the restore with the same archive.

The recovered target must contain two orders with the original identifiers and amounts, and the view must report Ada|6075. The third order is correctly absent. An attempted orphan order must fail through the restored foreign key. Permission checks report allowed selection and denied insertion; switching to the reader role confirms that reads work and a real write attempt is rejected.

These assertions cover data values, an application query, a relationship constraint, and effective privileges. All experimental servers stop on exit, and the printed directory retains the dump and logs. The drill does not test external authentication, encryption-key recovery, or application startup, which must be added when those are part of the actual recovery dependency set.

Record a recovery result people can use

A production rehearsal should retrieve the artifact from its intended recovery source, rebuild the required environment, restore data, and exercise representative application behavior. Provisioning compatible software is a normal part of recovery; include its elapsed time and prerequisites instead of assuming a prepared host will always exist. Keep recovery access available when production systems are unavailable.

Measure recovery duration against the recovery-time objective, and compare the recovered endpoint with the recovery-point objective. Include retrieval, validation, and the application becoming usable, not just the database command’s runtime. For larger restores, choose transaction and parallelism options appropriate to the workload, with an explicit partial-failure policy.

Repeat the rehearsal after changes to backup tooling, schema, roles, retention, or infrastructure. Preserve the successful checks and any corrected failure as evidence. A repeatable restore procedure with known inputs and limits supports an operational decision; a green backup scheduler alone does not answer whether recovery is ready.

Sources

PostgreSQL 18 pg_dump, database-level scope and excluded cluster-wide objects, checked 2026-09-12.
PostgreSQL 18 pg_restore, error handling, single-transaction restoration, grants, and parallelism constraints, checked 2026-09-12.
CISA StopRansomware Guide, testing backup availability and integrity with recovery dependencies, checked 2026-09-12.

Backups are only useful after a restore rehearsal

Wed, 03 Jun 2026 00:00:00 +0000

“We have backups” is not yet a recovery capability. A useful backup has a known scope, a known freshness, a verification step, and a restore procedure that someone has actually run.

A recovery rehearsal starts with identifiable inputs and a retrieved, verified stored copy. Restore into a disposable isolated target, then record files, permissions, dependencies and elapsed time. A checksum or existing archive alone does not demonstrate recoverability.

Define the recovery unit

For a static site, the source repository and the generated artifact have different purposes. The repository is the authoritative input; the artifact is the quickest way to restore the last published version. Keep both distinctions clear.

At minimum, record:

the source commit or archive identifier;
the generator and theme versions;
the list of files included in the artifact;
the checksum of important downloads;
the retention period and storage location.

Do not include credentials, private keys, cookies, or access tokens in a website backup. Those belong in a separate secret-management process.

Verify before storing

An archive that cannot be opened is not a backup. A minimal verification loop looks like this:

set -eu
tar -czf site-2026-06-03.tar.gz public/
tar -tzf site-2026-06-03.tar.gz >/dev/null
sha256sum site-2026-06-03.tar.gz > site-2026-06-03.tar.gz.sha256

Creating a checksum records a baseline; it does not read back the stored copy. Retrieve the archive and checksum from backup storage, then verify them before extraction. A matching checksum detects accidental change but does not authenticate an archive if an attacker can replace both files. Listing members also does not prove that the right files were backed up.

Restore somewhere disposable

Use a trusted archive, extract as an unprivileged user into an empty temporary directory or isolated host, then check entry points and a representative resource. An untrusted archive needs stronger isolation because paths and links can affect extraction. Do not test by overwriting the only live copy.

set -eu
sha256sum -c site-2026-06-03.tar.gz.sha256
restore_dir=$(mktemp -d)
tar -xzf site-2026-06-03.tar.gz -C "$restore_dir"
test -s "$restore_dir/public/index.html"
test -s "$restore_dir/public/404.html"

The command is intentionally boring. A rehearsal should reveal missing files, wrong permissions, or undocumented dependencies before an incident does.

Measure freshness honestly

A backup schedule is meaningful only relative to the amount of work that can be lost. If the source changes once a month, daily snapshots may be unnecessary. If a resource is updated every hour, a monthly archive is not enough.

Write down the target recovery point and recovery time, then choose storage and checks that can meet those targets. Avoid claiming “continuous protection” when the last verified restore is old.

Keep at least one recoverable copy outside the live host’s failure and deletion boundary. Record the elapsed restore time, restored checksums, permissions, and any manual dependencies; two nonempty HTML files are a smoke test, not a complete recovery acceptance test.

Sources

GNU tar manual, archive creation and verification behavior, checked 2026-06-03.
Static-site release checklist, local companion resource.
GNU Coreutils: SHA-2 utilities, checksum creation and --check verification, checked 2026-10-15.