Release and recovery · XB Field Notes

The advantage of small systems

Mon, 27 Jul 2026 00:00:00 +0000

Small does not mean unfinished. It means each component has a clear job, dependencies are deliberate, and the path from request to result remains visible.

Name each request boundary, its evidence and its owner. Fewer components reduce the paths to understand; backups and recovery practice still carry the recovery promise.

Visibility is a feature

When operators can name every boundary, they can also identify which evidence is missing. Logs become more useful, alerts become more specific, and routine maintenance becomes less risky.

For a static website the important boundaries are usually easy to list:

DNS points the hostname to the intended edge.
TLS terminates at the intended proxy.
The proxy serves one known output directory.
The output contains only the files the site needs.
A 404 response is handled by the site, not by an accidental upstream.

Writing this list down is more valuable than adding a dashboard that does not answer any of those questions.

Complexity should earn its place

A new layer is worthwhile when it removes more uncertainty than it creates. A build step that fingerprints assets can be useful when caching is important. A database is useful when content needs transactions. Neither is automatically an improvement for a small, mostly static site.

The same rule applies to client-side JavaScript. If a page genuinely requires client-side scripting, choose the smallest implementation that meets the requirement and review its content security policy deliberately. For this site, that would first require revisiting the accepted zero-script architecture and its verification contract; it is not an exception to the current policy. If the page is a document, plain HTML is often easier to inspect and more resilient.

Make ownership explicit

Every file should have a reason to exist and a clear way to update it. A vendored theme should record its upstream version and license. A generated directory should be marked as generated. A resource download should say what it contains, when it was updated, and how its checksum was produced.

These small statements prevent future operators from treating an accidental artifact as a supported interface.

Leave room for the next change

The best small system is not the one with the fewest lines. It is the one where the next change has a visible place to go. A notes site can start with Markdown, a static generator, and a web server; if it later needs search or comments, the new requirement can be evaluated on its own rather than assumed from the beginning.

Small is not automatically fault-tolerant. A single edge, disk, or operator can still be a single point of failure. Choose acceptable recovery time and data loss, keep an independent backup, and rehearse recovery. The benefit of fewer components is a smaller set of failure paths to understand, not immunity to outages.

Sources

Hugo directory structure, directory conventions for content and assets, checked 2026-07-27.
Caddy file_server directive, static file serving defaults referenced by the design, checked 2026-07-27.

A static server should have one obvious root

Thu, 04 Jun 2026 00:00:00 +0000

The most useful property of a static deployment is that an operator can answer “which directory served this response?” without reading application code. Caddy’s root and file_server directives provide that boundary directly.

Make the serving directory explicit. Audit symlinks and read/traverse permissions, then verify the actual response at the edge.

Make the root explicit

A minimal site block can be kept close to the artifact layout:

www.example.test {
 root * /srv/example-site
 encode zstd gzip

 header {
 -Server
 X-Content-Type-Options nosniff
 X-Frame-Options DENY
 Referrer-Policy strict-origin-when-cross-origin
 }

 file_server
}

The hostname and path are placeholders; www.example.test is not a publicly issuable production name. Use an owned, resolvable hostname for production and plan local TLS trust separately for a test environment. The release command should publish a complete directory, verify it, and only then make that directory the configured root. Keep the prior known-good directory until the new one has passed smoke tests.

Check the edge, not only the file tree

Request an HTML page, a downloadable text file, and a missing path. Compare content types and status codes with the local artifact. A 404 page that returns 200 is a deployment bug even if the page looks polished.

When Caddy manages HTTPS, certificate provisioning and HTTP-to-HTTPS redirects are part of the edge behavior. Test the redirect from the bare HTTP listener and the certificate name from the public hostname. Do not infer success from a healthy process alone.

A root is a publication boundary, not a sandbox

Keep source control metadata, backups, credentials, and private configuration outside the published tree. Caddy follows symlinks inside the root, so a link can expose files outside it. Check both the artifact contents and the serving user’s read/traverse permissions. Also, merely placing 404.html in the tree does not configure a custom error handler; verify the missing-route status and error-page behavior you intend.

Sources

Caddy file_server directive, file serving, root, and index defaults, checked 2026-10-15.
Caddy Automatic HTTPS, automatic TLS and issuance behavior, checked 2026-10-15.
Static-site release checklist, local companion resource.
Caddy header directive, response header operations, checked 2026-10-15.
RFC 7034 — X-Frame-Options, the DENY framing policy, checked 2026-10-15.
MDN: X-Frame-Options, framing options, checked 2026-10-15.
RFC 6761 — Special-Use Domain Names, reserved .test names, checked 2026-10-15.

A release checklist for a static site

Thu, 30 Jul 2026 00:00:00 +0000

A static site is simple to publish, but simplicity does not remove the need for a release check. The useful checks are small enough to run on every release and specific enough to fail for a real reason.

The checks belong to the generated artifact. Clean output does not establish reproducibility; placeholder searches are review aids, with parser-based link checks as the gate. Serve the exact artifact before publishing, compare public GET bodies afterward in both languages, and keep the commit, generator version, and download checksums.

Build a clean artifact

Use a pinned generator version and build into a disposable output directory:

set -eu
hugo version
hugo --cleanDestinationDir --gc --minify --destination public

--gc collects unused build-cache resources; it does not clean stale published pages. --cleanDestinationDir removes stale destination files, so use it only on the disposable artifact directory, never a directory containing source or unrelated data. Compare the printed version with the project’s pin; printing a version does not enforce it. Record the build inputs and verify the resulting artifact. A clean destination does not establish byte-for-byte reproducibility: timestamps and other nondeterministic inputs need separate control and comparison. Do not edit generated HTML by hand after the build.

Check the artifact, not only the source

Inspect the generated tree and verify that expected entry points exist. Run this assertion block as a standalone shell script; it must stop before printing the inventory if any required file is missing or empty:

set -eu
test -s public/index.html
test -s public/404.html
test -s public/robots.txt
test -s public/sitemap.xml
find public -type f -print | sort

Then search for accidental placeholders, local paths, and development-only hosts:

rg -l 'localhost|127\.0\.0\.1|TODO|example\.com|/Users/' public --glob '*.html'

This is a review aid, not an automatic release gate: examples like this checklist deliberately match, including in translated pages. This HTML-only scan does not cover XML feeds or JSON search output. rg exits 1 when there are no matches and 2 on an error. Review each listed file, distinguish fenced examples from active links or attributes, and use a parser-based link check for the actual gate. A private path in a published link is a release mistake.

Exercise the public routes

An isolated production-like server pointed at this exact artifact gives the strongest pre-release routing check. Hugo’s development server rebuilds content; the following is a separate source preview, not evidence that the already checked public/ bytes were served. Run the first command in one terminal and the requests in another:

hugo server --bind 127.0.0.1 --port 1313 --baseURL http://127.0.0.1:1313/ \
 --disableFastRender --disableLiveReload --renderToMemory
curl -I http://127.0.0.1:1313/
curl -I http://127.0.0.1:1313/notes/
curl -I http://127.0.0.1:1313/resources/
curl -I http://127.0.0.1:1313/missing-page/

HEAD checks do not compare bodies. Repeat with GET against the staged server, then after publishing compare public bodies with the artifact and check both languages. Confirm that the missing route returns a 404 status in the production server. A pretty error page with a 200 status still breaks monitoring and caches.

Record the result

Keep the source commit, generator version, and checksum of each downloadable resource in the release record. This makes it possible to compare a live file with the source that produced it without relying on memory.

The downloadable version of this checklist is in the Resources section.

Sources

Hugo command reference, generator flags for a clean production build, checked 2026-07-30.
curl manual, HTTP verification and header inspection options, checked 2026-07-30.
Static-site release checklist, downloadable companion resource.
Hugo server command reference, development rebuilds, memory rendering, and live-reload controls, checked 2026-10-15.

Publish a static site by switching complete directories

Mon, 01 Jun 2026 00:00:00 +0000

Copying new files directly over a live static directory creates a period where visitors can receive HTML from one release and assets from another. A safer pattern stages one complete tree beside the live tree, verifies it, then changes directory names.

These two moves leave live absent between them; crashes can prolong the gap. Restore must be attempted and checked. Old asset URLs must remain reachable across separate browser requests.

Keep all release directories on one filesystem

Keep the live tree, staged tree, and rollback tree on the same filesystem. These fragments assume one Bash session, GNU Coreutils, and a single deployer with exclusive access to these names. Stop on a failed command or either side of the copy pipeline:

set -euo pipefail
df -P /srv/www

Create a uniquely named stage directory under that parent. Do not reuse an unresolved variable or broad wildcard as a deletion target.

stage=$(mktemp -d /srv/www/site.new.XXXXXX)

Populate and verify the stage

Copy the generated artifact into the empty directory, then check representative files. The final command displays a count; compare it with the verified artifact’s expected count or manifest rather than treating any number as a pass:

tar -C public -cf - . | tar -C "$stage" -xf -

test -s "$stage/index.html"
test -s "$stage/404.html"
find "$stage" -type f | wc -l

Run link and checksum checks against the staged path when the tooling supports a directory argument. Also verify that the serving user can traverse the directories and read the files: mktemp -d starts with restrictive permissions, and the extracted artifact must supply the intended modes. A stage that has not passed verification must never acquire the live name.

Switch names with an explicit rollback path

Resolve every path before the first move:

live=/srv/www/site
stamp=$(date -u +%Y%m%dT%H%M%SZ)
rollback=/srv/www/site.old-$stamp

test -d "$live" && test ! -L "$live" || exit 1
test -d "$stage" && test ! -L "$stage" || exit 1
test ! -e "$rollback" && test ! -L "$rollback" || exit 1
df -P "$live" "$stage"
live_device=$(stat -c %d -- "$live") || exit 1
stage_device=$(stat -c %d -- "$stage") || exit 1
test "$live_device" = "$stage_device" || exit 1

mv -T -- "$live" "$rollback" || exit 1
if mv -T -- "$stage" "$live"; then
 test -s "$live/index.html"
else
 mv -T -- "$rollback" "$live" || {
 printf 'Restore failed; inspect %s and %s before retrying.\n' "$rollback" "$live" >&2
 exit 1
 }
 exit 1
fi

This is a two-step switch, not one atomic replacement: the live path is absent between moves, and a crash can leave it absent until recovery. GNU mv -T prevents an existing directory from silently becoming a container for the moved tree; it does not serialize competing deployers. Do not proceed if the filesystem check differs. Platform-specific atomic exchange or symlink replacement can remove the missing-name interval, but must be tested with the actual server.

Verify the public edge

After the switch, request the homepage, one deep route, one fingerprinted asset, and a missing path. Compare public bytes with the staged artifact. Keep the rollback directory until that check and the observation window complete.

Even an atomic name switch does not make a browser’s HTML and asset requests one transaction. Keep older fingerprinted assets reachable for cached or in-flight HTML; merely retaining an unserved rollback directory does not do that. A filesystem rename also makes no promise of crash durability without the platform-specific persistence steps.

Sources

POSIX rename(), name replacement semantics, checked 2026-06-01.
Linux rename(2), filesystem and replacement behavior, checked 2026-10-15.
GNU tar manual, archive creation and extraction, checked 2026-06-01.
GNU Coreutils: Target directory, -T and directory nesting hazards, checked 2026-10-15.
GNU Coreutils: stat, filesystem device-number checks, checked 2026-10-15.

A release manifest should name every published byte

Thu, 16 Jul 2026 00:00:00 +0000

A static artifact is a directory, not one executable. Recording only the source commit or homepage checksum leaves gaps: a missing image, stale alias, or changed download can survive unnoticed. Enumerate every regular file in a frozen artifact before transport.

Freeze writers, reject symlinks and special files, and keep both manifests outside the artifact trees. A listed-file check detects missing or changed bytes; recomputing and comparing the complete lists also detects extras. Hashing the trusted source manifest supplies a compact identifier, with separate controls for authenticity and metadata.

Generate a stable list

Run this in Bash with GNU findutils and coreutils, from the directory containing public/. Keep the manifest outside the measured tree. The release must contain only directories and regular files: inspect and reject symlinks or special files first, since they are outside this manifest’s contract.

set -euo pipefail
manifest="$PWD/release.sha256"
(
 cd public
 nonregular=$(find . ! -type d ! -type f -print -quit)
 test -z "$nonregular"
 find . -type f -print0 | LC_ALL=C sort -z | xargs -0 -r sha256sum --
) > "$manifest"
test -s "$manifest"

Sort filenames before hashing. NUL-delimited paths preserve spaces, tabs, and embedded newlines between tools; GNU sha256sum escapes special names in its ordinary checkable output. Do not parse that output with awk '{print $2}': whitespace is legal in filenames. The separate assignment preserves a failed find exit status; testing only its output could mistake a failed scan for an empty result. Treat any failed scan or pipeline as an incomplete manifest and do not publish it. Stop writers while measuring; these commands do not create a filesystem snapshot.

Verify after transport

In one disposable Bash session on the destination, run the remaining fragments in order so the paths and set options stay in scope. Set the manifest and artifact paths to their actual locations:

set -euo pipefail
manifest=/path/to/release.sha256
artifact=/srv/www/site.new
(
 cd "$artifact"
 sha256sum -c "$manifest"
)

Checking listed files catches missing or changed bytes, but not extra files. Recompute the destination manifest with the same procedure and compare the complete records:

observed=$(mktemp "${manifest}.observed.XXXXXX")
trap 'rm -f "$observed"' EXIT
(
 cd "$artifact"
 nonregular=$(find . ! -type d ! -type f -print -quit)
 test -z "$nonregular"
 find . -type f -print0 | LC_ALL=C sort -z | xargs -0 -r sha256sum --
) > "$observed"
diff -u "$manifest" "$observed"

No diff and exit status zero means the regular-file paths and hashes agree. An extra stale file now fails too. The temporary output remains outside the artifact so the check never includes itself. Empty directories, permissions, ownership, and timestamps are not measured; validate those separately if deployment depends on them.

Add an aggregate digest for release records

Hash the manifest itself for a compact release identifier:

sha256sum "$manifest"

Its meaning depends on fixed rules: relative paths, sorting locale, algorithm, tool format, and included metadata. Keep the trusted manifest with the source commit and builder version.

Do not confuse integrity with authenticity

Matching SHA-256 digests are strong practical evidence of matching bytes, not proof of authorship. An attacker able to replace both artifact and manifest can make the check pass. Signing and trusted distribution are separate controls; verify only a manifest obtained through the intended trusted channel.

Sources

GNU Coreutils: sha2 utilities, checksum generation and verification; reviewed through the linked upstream manual source, checked 2026-10-15.
GNU Coreutils manual source, untagged checksum escaping, --check, and NUL-delimited sorting, checked 2026-10-15.
sha256sum(1), command behavior and output format, checked 2026-07-16.
GNU Bash: Simple Command Expansion, assignment-only commands retain command-substitution exit status, checked 2026-10-18.

Read the public edge in two commands

Mon, 13 Jul 2026 00:00:00 +0000

A deployment is complete when public responses match the release contract, not when the last upload exits zero. Two read-only helpers from the operator kit expose useful evidence, but they do not prove the whole artifact was published correctly.

These are illustrative excerpts. Evaluate HTTP status and expiry separately; verify hostname, trust and body equality with the checks that actually test them.

Inspect the response headers

check-http-headers.sh performs a GET, discards the body, follows redirects, and prints the response headers for each hop plus the final status and URL:

check-http-headers.sh https://www.xbcatm.com/

This illustrative output shows the fields to inspect. It is not a fresh production observation:

HTTP/2 200
accept-ranges: bytes
content-security-policy: default-src 'self'; base-uri 'self'; form-action 'none'; frame-ancestors 'none'; object-src 'none'; script-src 'none'; style-src 'self'; img-src 'self' data:
content-type: text/html; charset=utf-8
last-modified: Mon, 24 Aug 2026 11:20:54 GMT
permissions-policy: accelerometer=(), camera=(), geolocation=(), gyroscope=(), microphone=(), payment=(), usb=()
referrer-policy: strict-origin-when-cross-origin
strict-transport-security: max-age=31536000; includeSubDomains
x-content-type-options: nosniff
x-frame-options: DENY

HTTP_STATUS 200
FINAL_URL https://www.xbcatm.com/

Check the expected status, final host and path, and the policy on the final response. The helper does not use curl’s --fail: an HTTP 404 or 500 can still leave the shell command successful. Treat its printed status as data to evaluate. A header mismatch warrants investigation of server, proxy, or cache configuration; it does not by itself prove the wrong output directory is live.

Inspect the certificate dates

check-tls-expiry.sh HOST [DAYS] extracts the certificate presented with SNI and checks whether its expiry is at least the requested number of seconds away:

check-tls-expiry.sh www.xbcatm.com 30

An illustrative excerpt, with the fingerprint line omitted:

subject=CN=www.xbcatm.com
issuer=C=US, O=Let's Encrypt, CN=YE1
notBefore=Aug 21 08:37:13 2026 GMT
notAfter=Nov 19 08:37:12 2026 GMT

certificate remains valid for at least 30 day(s)

The helper prints subject, issuer, dates, and SHA-256 fingerprint. It does not print the remaining day count. Its success message describes the expiry test only: the script does not enforce hostname or trust-chain validation, and -checkend does not validate notBefore. The subject CN is not sufficient hostname evidence; service identity is matched against the certificate’s subject alternative names. A valid wildcard may legitimately cover the host.

Use a separate verification when those properties matter:

openssl s_client -connect www.xbcatm.com:443 -servername www.xbcatm.com \
 -verify_hostname www.xbcatm.com -verify_return_error </dev/null

This uses the configured trust store and fails on certificate verification errors under OpenSSL’s matching rules. In OpenSSL 3.0, hostname verification can fall back to the subject CN when no suitable DNS subject alternative name is present. A successful command therefore does not by itself establish RFC 9525’s SAN-only identity policy: inspect the SAN extension and verify the intended client’s policy separately. Keep a job-level timeout around the TLS helper, which does not impose its own network deadline. Certificate inspection and automatic renewal configuration remain separate checks.

Use both around a release

Before: save the expected policy, final URL, validators, and certificate dates with a UTC collection time.
Deploy the already validated artifact.
After: repeat the probes and fetch representative bodies for comparison with that artifact.

Headers and certificate dates cannot establish byte equality. Last-Modified may stay unchanged or be rewritten during copying. The release smoke test saves bodies and compares them directly; a complete manifest extends that check beyond one page. Browser behavior, geographic edge differences, and load-dependent failures need separate tests. Record which paths and observation points were actually checked.

Sources

curl manual, GET, redirect, header output, and HTTP failure behavior, checked 2026-10-15.
OpenSSL s_client, SNI, explicit hostname checks, and failure on verification errors, checked 2026-10-15.
OpenSSL verification options, hostname matching and subject-CN fallback, checked 2026-10-15.
OpenSSL x509, certificate display and expiry-only -checkend behavior, checked 2026-10-15.
RFC 9525 — Service Identity in TLS, subject alternative names and wildcard identity matching, checked 2026-10-15.
Caddy Automatic HTTPS, automatic certificate management and renewal configuration, checked 2026-10-15.
MDN: Content Security Policy guide, CSP directive background, checked 2026-07-13.
RFC 9110 — HTTP Semantics, response metadata and validators do not establish a complete artifact match, checked 2026-10-15.

A release is not complete until the public edge agrees

Thu, 09 Jul 2026 00:00:00 +0000

Static publishing removes application runtime complexity, but it does not remove the boundary between a generated directory and a public response. A release check should cover both.

Use the same artifact before and after selection. The download row refers to public/downloads/check-http-headers.sh in the article. Supported content codings are decoded before cmp; server transformations need a documented rule. Status and header checks are separate. Retain the previous directory for a deliberate rollback.

Start from a clean artifact

Build with the pinned generator into a clean destination. Record the source commit, generator version, output file list, and checksums of files that users download. A clean build catches stale files that a copy-over deployment can accidentally preserve.

set -eu
make verify
find public -type f -print | sort

The file list is evidence, not decoration. If a route disappears, the diff should explain whether the source was removed or the generator failed to emit it.

Exercise three response classes

Before publishing, test the staged artifact with the intended server configuration. After switching directories, repeat these checks at the public edge; probing the old live site cannot validate the new release. Request one HTML page, one static resource, and one missing path. Inspect status, content type, cache policy, and security headers:

set -eu
curl --compressed -fsS -D page.headers -o page.body https://www.example.test/
curl --compressed -fsS -D asset.headers -o asset.body \
 https://www.example.test/downloads/check-http-headers.sh
curl --compressed -sS -D missing.headers -o missing.body \
 https://www.example.test/does-not-exist

cmp public/index.html page.body
cmp public/downloads/check-http-headers.sh asset.body
cmp public/404.html missing.body

Run each fragment as a standalone shell script or in a disposable shell session. set -eu stops this sequential script after a failed build, transfer, or comparison, so a later successful command cannot hide an earlier failure. The missing-path request intentionally does not use -f: a 404 is expected and should still be captured. Check the final status in each header file: 200, 200, then 404; the byte comparisons do not automate that status check. No redirect is followed here. --compressed decodes supported content codings before comparison; a deliberate server-side HTML transformation needs its own documented comparison rule. cmp is silent on equality and returns nonzero on a difference. Repeat for /zh/ and its error page rather than assuming language parity.

Keep rollback simple

Do not delete the previous directory until the new one has passed the checks. A rollback should select a known-good directory, not reconstruct one from memory during an incident. The release checklist and HTTP header check are deliberately small enough to run from a clean shell.

Sources

RFC 9110 — HTTP Semantics, status and representation semantics, checked 2026-07-09.
curl manual, response inspection options, checked 2026-07-09.
Static-site release checklist, local companion resource.

DNS changes need a cache-aware rollback window

Thu, 18 Jun 2026 00:00:00 +0000

Changing an address record is easy; predicting when every resolver stops using the old answer is not. The time-to-live value travels with a DNS response and controls its ordinary cache freshness. Lowering the TTL at the moment of a migration does not rewrite answers that were cached earlier.

TTL travels with each answer. Prepare before the switch, keep both endpoints valid, and use old-endpoint traffic plus a margin to guide retirement and rollback.

Lower the TTL before the event

If an existing record has a one-day TTL, lower it at least one old TTL before the planned switch. That gives caches time to fetch the shorter value while the authoritative record still points to the old endpoint. The exact interval belongs in the change plan rather than in an improvised command during the migration.

Record authoritative answers separately. The following two probes compare the system-configured resolver with a public recursive resolver; neither is inherently an authoritative query:

dig +noall +answer www.example.test A
dig @1.1.1.1 +noall +answer www.example.test A

For authoritative evidence, query the zone’s actual nameserver without recursion and inspect the AA flag, status, answer, and authority sections. The remaining TTL shown by a recursive resolver is evidence of its current cache state; different resolvers can legitimately differ. Repeat for AAAA and any CNAME chain used by the service, not only A.

Keep both sides valid

During the switch, keep the old and new endpoints capable of serving the intended hostname. TLS names, redirects, and application configuration must agree on both sides. Removing the old endpoint immediately after the authoritative record changes creates an avoidable outage for clients that still hold the prior answer.

Negative answers are cached too. Creating a name that recently returned NXDOMAIN can take time to become visible because the negative response may remain in recursive caches according to the zone’s authority data.

Raise the TTL after observation

Once authoritative and representative recursive answers point to the new endpoint and the old cache window has elapsed, restore the normal TTL. Keep the change record with the old value, new value, authoritative answer, and rollback condition.

TTL is not a universal shutdown deadline. Resolvers can serve stale data during resolution failures, and long-lived connections or application caches may outlast DNS freshness. Observe old-endpoint traffic and retain a margin; changing DNS back also has a cache window. For negative answers, inspect the SOA-derived negative TTL rather than assuming the new A record’s TTL applies.

Sources

RFC 1034 — Domain Names: Concepts and Facilities, caching model, checked 2026-10-15.
RFC 1035 — Domain Names: Implementation and Specification, TTL field, checked 2026-10-15.
RFC 2308 — Negative Caching of DNS Queries, negative caching model and retained error TTL, checked 2026-10-15.
RFC 8767 — Serving Stale Data, stale answers during DNS resolution failures, checked 2026-10-15.
BIND 9 dig manual, server selection and recursion controls, checked 2026-10-15.

Restart policies need a rate limit

Mon, 03 Aug 2026 00:00:00 +0000

Automatic restart is useful for a long-running service, but an unbounded loop can turn a clear failure into a noisy one. The unit should describe both the normal process lifecycle and the amount of recovery that is acceptable.

Count starts within the configured window. A restart rate limit stops rapid churn; it neither proves health nor caps lifetime retries.

Track setup failures

For a long-running process, Type=exec makes failures to execute the configured program visible during start-up rather than treating a fork as success. Pair it with an explicit ExecStart and the least-privileged User that can perform the work.

[Unit]
Description=Example foreground site service
StartLimitIntervalSec=60s
StartLimitBurst=5

[Service]
Type=exec
User=site
ExecStart=/srv/site/bin/server
Restart=on-failure
RestartSec=5s

Type=exec confirms execution, not application readiness. Keep the program in the foreground; use Type=notify when the application supports explicit readiness notification. Restart=on-failure is a recovery policy, not a health check. If the process stays alive while serving bad responses, systemd will not know that the site is broken; an external smoke test still has a job.

Bound the loop

Systemd applies start-rate limiting through StartLimitIntervalSec= and StartLimitBurst=. These directives belong in [Unit], while RestartSec belongs in [Service]. The example allows five starts within 60 seconds; rapid failures separated by five-second restart delays can therefore hit the limit. A longer delay can keep a loop below that rate forever: this is not a lifetime retry budget. Choose limits from the expected failure pattern. When the limit is hit, the journal should contain the evidence needed to distinguish “restarted successfully” from “gave up after repeated failure”.

After installing or editing the unit on an authorized test host, validate the file, reload the manager, and inspect its effective configuration before restarting. The commands below change service state and can interrupt traffic:

systemd-analyze verify /etc/systemd/system/example.service
systemctl daemon-reload
systemctl cat example.service
systemctl show example.service -p Restart -p RestartUSec \
 -p StartLimitIntervalUSec -p StartLimitBurst
systemctl restart example.service
systemctl status --no-pager example.service
journalctl -u example.service -n 80 --no-pager

Do not use a restart policy to conceal missing files, invalid permissions, or a bad environment. Fix the first failure, then test the recovery path.

Once start limiting stops automatic restarts, waiting for the interval alone does not schedule another attempt. After fixing the cause, either start it after the interval or deliberately use systemctl reset-failed example.service to clear the counter before starting. Test missing-executable and rapid-crash cases in an isolated service, and retain the first failure log rather than only the final “start limit hit” message.

Sources

systemd.service, service type, restart, and start-rate limiter, checked 2026-08-03.
systemd.unit, common unit sections and drop-in semantics, checked 2026-08-03.
systemd.service upstream manual (v257), Type=exec/oneshot, readiness, activation timeout, and restart semantics, checked 2026-10-15.
systemd.unit upstream manual (v257), start-limit section placement, rate window, and reset-failed behavior, checked 2026-10-15.

A maintenance timer should leave its last result visible

Tue, 04 Aug 2026 00:00:00 +0000

Cron-like syntax is not the whole contract for a maintenance task. The command needs a clear service identity, a bounded runtime, and evidence that distinguishes “not scheduled” from “ran and failed”. A systemd timer and oneshot service keep those concerns separate.

A fired timer is only a scheduling event. Judge this run by its service result, exit status and fresh timestamp.

Define the work as a service

Use an explicit user and absolute command path:

# /etc/systemd/system/site-check.service
[Unit]
Description=Read-only public site check

[Service]
Type=oneshot
User=sitecheck
ExecStart=/usr/local/bin/check-public-site
TimeoutStartSec=2min

TimeoutStartSec limits the oneshot activation, which includes its command runtime; oneshot services otherwise have no start timeout by default. Timeout handling can then require a stop timeout and cannot instantly end uninterruptible kernel I/O, so the helper also needs bounded network requests. The helper should return non-zero when its contract fails and must not modify the service it checks.

Schedule the service

# /etc/systemd/system/site-check.timer
[Unit]
Description=Run the public site check every hour

[Timer]
OnCalendar=hourly
Persistent=true
RandomizedDelaySec=5min
Unit=site-check.service

[Install]
WantedBy=timers.target

Persistent=true triggers one catch-up activation when the timer is activated after missing at least one calendar event; it does not replay every missed hour. The catch-up is still subject to the randomized delay. RandomizedDelaySec spreads routine work across a window; it is not appropriate when the task must run at an exact instant.

Validate before enabling

After creating both unit files, validate them and reload the manager before starting the service manually. These are administrative operations on the intended test host, not read-only checks:

systemd-analyze verify /etc/systemd/system/site-check.service \
 /etc/systemd/system/site-check.timer
systemd-analyze calendar hourly
systemctl daemon-reload
systemctl start site-check.service
systemctl status site-check.service
journalctl -u site-check.service -n 50 --no-pager
systemctl show site-check.service -p Result -p ExecMainCode -p ExecMainStatus

Enable the timer only after the service succeeds with the intended user, environment, paths, and network access:

systemctl enable --now site-check.timer
systemctl list-timers site-check.timer

Alert on the service result

The timer firing is not success. Monitor the oneshot service’s exit status or journal result. Keep stdout concise and send detailed artifacts to a bounded path if they are needed for debugging.

A successful oneshot normally returns to inactive (dead); that alone is not a failure, and systemctl status can return nonzero for an inactive service. Correlate Result and exit status with a fresh journal timestamp so an old successful run is not mistaken for this one. Do not add RemainAfterExit=yes merely to keep it green: a recurring timer does not start a fresh instance while its service is already active. Calendar timing uses the host timezone unless specified, and normal timer accuracy adds scheduling slack beyond the random delay.

Sources

systemd.timer(5), calendar timers, persistence, and delay behavior, checked 2026-08-04.
systemd.time(7), calendar expression syntax, checked 2026-08-04.
systemd.service, oneshot service semantics, checked 2026-08-04.
systemd.service upstream manual (v257), Type=exec/oneshot, readiness, activation timeout, and restart semantics, checked 2026-10-15.
systemd.timer upstream manual (v257), persistence coalesces missed runs; active units, randomized delay, and accuracy, checked 2026-10-15.

Backups are only useful after a restore rehearsal

Wed, 03 Jun 2026 00:00:00 +0000

“We have backups” is not yet a recovery capability. A useful backup has a known scope, a known freshness, a verification step, and a restore procedure that someone has actually run.

A recovery rehearsal starts with identifiable inputs and a retrieved, verified stored copy. Restore into a disposable isolated target, then record files, permissions, dependencies and elapsed time. A checksum or existing archive alone does not demonstrate recoverability.

Define the recovery unit

For a static site, the source repository and the generated artifact have different purposes. The repository is the authoritative input; the artifact is the quickest way to restore the last published version. Keep both distinctions clear.

At minimum, record:

the source commit or archive identifier;
the generator and theme versions;
the list of files included in the artifact;
the checksum of important downloads;
the retention period and storage location.

Do not include credentials, private keys, cookies, or access tokens in a website backup. Those belong in a separate secret-management process.

Verify before storing

An archive that cannot be opened is not a backup. A minimal verification loop looks like this:

set -eu
tar -czf site-2026-06-03.tar.gz public/
tar -tzf site-2026-06-03.tar.gz >/dev/null
sha256sum site-2026-06-03.tar.gz > site-2026-06-03.tar.gz.sha256

Creating a checksum records a baseline; it does not read back the stored copy. Retrieve the archive and checksum from backup storage, then verify them before extraction. A matching checksum detects accidental change but does not authenticate an archive if an attacker can replace both files. Listing members also does not prove that the right files were backed up.

Restore somewhere disposable

Use a trusted archive, extract as an unprivileged user into an empty temporary directory or isolated host, then check entry points and a representative resource. An untrusted archive needs stronger isolation because paths and links can affect extraction. Do not test by overwriting the only live copy.

set -eu
sha256sum -c site-2026-06-03.tar.gz.sha256
restore_dir=$(mktemp -d)
tar -xzf site-2026-06-03.tar.gz -C "$restore_dir"
test -s "$restore_dir/public/index.html"
test -s "$restore_dir/public/404.html"

The command is intentionally boring. A rehearsal should reveal missing files, wrong permissions, or undocumented dependencies before an incident does.

Measure freshness honestly

A backup schedule is meaningful only relative to the amount of work that can be lost. If the source changes once a month, daily snapshots may be unnecessary. If a resource is updated every hour, a monthly archive is not enough.

Write down the target recovery point and recovery time, then choose storage and checks that can meet those targets. Avoid claiming “continuous protection” when the last verified restore is old.

Keep at least one recoverable copy outside the live host’s failure and deletion boundary. Record the elapsed restore time, restored checksums, permissions, and any manual dependencies; two nonempty HTML files are a smoke test, not a complete recovery acceptance test.

Sources

GNU tar manual, archive creation and verification behavior, checked 2026-06-03.
Static-site release checklist, local companion resource.
GNU Coreutils: SHA-2 utilities, checksum creation and --check verification, checked 2026-10-15.

Reliable defaults beat clever recovery

Mon, 20 Jul 2026 00:00:00 +0000

Small services often accumulate recovery logic before their normal path is fully understood. Every retry, alternate endpoint, and compatibility branch adds another state that must be observed and tested.

Keep the common path visible. Qualify and bound retries, then recover to a retained known-good state rather than a partial copy.

Make the common path boring

A useful default is explicit, easy to inspect, and safe to repeat. For a static site, that can be as simple as:

Build from a pinned toolchain.
Check the generated HTML, assets, and complete staged directory.
Publish that directory, then verify public responses against it.
Keep the previous directory available for rollback.

The sequence is less impressive than a multi-stage deployment system, but its state is visible. When something fails, the operator can answer whether the problem is in the source, the build, the copy, or the web server.

Retry only a bounded operation

Retries are useful when the failure may be transient and repeating the operation preserves its intended effect. Idempotence does not mean every response or incidental log entry is identical; after a lost response, the first attempt may already have succeeded. A download from an origin may be safe to retry; a database migration may not be. Treating every error as retryable hides the boundary that needs attention.

When a retry is appropriate, record the attempt count and the final reason. A log entry such as “publish failed: checksum mismatch” is more actionable than a sequence of silent retries followed by a generic timeout.

Prefer one authoritative state

Configuration, runtime state, and documentation should agree about which directory is live and which command produces it. If a release can be copied to three different paths depending on who runs it, recovery becomes guesswork.

For a static deployment, a short release record is enough:

release: 2026-07-20
source: git commit <commit-id>
builder: Hugo 0.165.0
output: public/
verification: links + headers + checksums

The placeholder commit ID is deliberately resolved at release time instead of being invented in documentation.

Recovery should restore a known state

Recovery is easiest when it returns the service to the last known-good directory, not when it chooses among several partially built alternatives. Keep the previous output until the new output has passed its checks, then remove old copies according to a retention policy.

The goal is not to eliminate failure. It is to make the failure small enough to explain and the recovery small enough to trust.

Bound retries by both attempts and elapsed time. Use backoff with jitter when many clients could retry together, and avoid stacking retries at every layer. Permission errors and deterministic bad input need correction, not a longer retry loop.

Sources

RFC 9110 — HTTP Semantics, section 9.2.2 on idempotent methods and retries, checked 2026-07-20.
Static-site release checklist, local companion resource.
AWS Builders’ Library: Timeouts, retries, and backoff with jitter, retry amplification, bounded retry budgets, backoff, and jitter, checked 2026-10-15.

A postmortem should leave a fact record

Tue, 07 Jul 2026 00:00:00 +0000

A postmortem earns trust when its claims can be re-checked. Record each observation’s source, collection time and timezone, command or query, and exit status where relevant. Logs, metrics, and clearly attributed human observations also count; distinguish confirmed facts from hypotheses and preserve redacted evidence rather than secrets. The four phases below keep a write-up reviewable at any later time.

Tie every phase to captured evidence. A post-fix sample is a bounded observation; causes, uncertainty and owned prevention still need explicit treatment.

Detect: the symptom, with evidence

Start with the smallest verifiable symptom and the command that showed it. Save the actual capture. The following is an illustrative excerpt, not evidence of a fresh probe:

$ check-http-headers.sh https://www.xbcatm.com/
HTTP/2 200
content-security-policy: ... script-src 'none' ...
last-modified: Mon, 24 Aug 2026 11:20:54 GMT

If the symptom is that a header changed, the output showing the previous policy and the new one belongs side by side. If it is a failure, include the exit status and the error line.

Scope: how wide, and how you know

State the blast radius with evidence, not adjectives. Define the boundary you can inspect:

Which hosts and paths show the symptom? Record each probed URL and observation time; one successful request cannot establish the state of every edge or visitor.
Does the cache differ from origin? Compare actual responses, cache metadata, and validators from both paths using the same request headers. Vary describes selection inputs, not proof that the response came from cache.
What delays recovery? For a DNS change, record the old TTL and resolver observations. For HTTP caches, record freshness and revalidation policy separately; a DNS TTL is not a universal rollback deadline.

Fix: the change and its intended effect

Record the actual configuration diff and deployment reference so the corrective action can be understood and, where safe, replayed. This header fragment is an expected observation, not a shell command or a complete policy. Prefer the smallest change that restores the contract, and name what the post-fix run should show:

Content-Security-Policy: script-src 'none'

Verify: the post-incident run

Repeat the detection command after the change and show that the observed output now matches the intended contract. A postmortem without this final section is a story; with it, it is a record that approves its own future re-check.

Close the record with contributing causes, what remains uncertain, and preventive actions with an owner, due date, and acceptance check. A successful probe confirms that sampled path at that time; it does not establish the cause or prevent recurrence by itself.

Sources

RFC 9110 — HTTP Semantics, response metadata and conditional requests, checked 2026-07-07.
artifact HTTP header check, read-only inspection used throughout this record.
Google SRE: Postmortem Culture, impact, contributing causes, evidence review, and preventive actions, checked 2026-10-15.
RFC 9111 — HTTP Caching, cache selection and freshness are distinct from DNS cache lifetimes, checked 2026-10-15.

Ship a static site and prove the edge agrees

Tue, 11 Aug 2026 00:00:00 +0000

A release is complete when the required public-edge checks agree with the artifact you chose, not merely when upload exits zero. This tutorial connects build, check, publish, and verify, while distinguishing local checks from deployment changes. Publishing changes the server; only the final probes are read-only network observations.

The commands assume Bash, GNU find/sort/sha256sum, rsync, Python 3, and the project’s pinned Hugo. Run build steps from the repository root. Before touching a live host, rehearse the directory-switch example in a disposable directory. Know the actual serve root, who owns deployment, the previous release target, and the rollback procedure. The /srv layout below is an example requiring an already configured deployment, not something this repository automatically installs.

The pointer switch changes the failure boundary. Before it, stop with the old release; after it, roll back explicitly and verify again. Disk checks and edge samples establish different facts.

1. Build one artifact from recorded inputs

Start from a clean source revision. Run the project build and record the installed generator version:

set -eu
git status --short
git rev-parse HEAD
hugo version
make build

An empty short status supports a clean-tree claim; the command itself does not enforce one. Inspect any changes before calling the build a release of that commit. make build writes data/buildinfo.toml with the short commit and current UTC build time, then generates public/. The rendered marker identifies the recorded inputs; it is not proof that a dirty checkout contained only that commit.

Pinning Hugo and the vendored theme reduces drift, but the build-time field means ordinary builds at different times need not be byte-identical. Build the chosen release once, validate it, and transfer that exact artifact rather than rebuild independently on the server.

2. Check the complete offline contract

make check checks generated visuals, offline-pack freshness, internal links, script-free output, source links, and downloadable checksums for the current artifact. For the full repository gate, use:

set -eu
make verify
make check-upstream

The full gate also runs regression tests, bilingual-post checks, and review-ledger freshness, then rebuilds and checks output. Therefore create the release manifest only after the last build. Build success and a current review ledger do not establish technical correctness of every article or prove the public server uses the intended headers. Page counts and timings vary; retain actual output rather than comparing to a copied example count.

3. Name the artifact bytes in a manifest

Before hashing, verify that public/ contains only directories and regular files, and stop all writers until staging is complete. The commands below do not create a snapshot or check for every special-file type. Keep the manifest outside public/ so it cannot hash itself. NUL-delimited filenames preserve spaces and other shell-sensitive characters:

set -euo pipefail
manifest="$(pwd)/release.sha256"
(
 cd public
 find . -type f -print0 | LC_ALL=C sort -z | xargs -0 sha256sum
) > "$manifest"
(cd public && sha256sum -c "$manifest")
sha256sum "$manifest"

GNU checksum lines contain the digest before the filename. A successful sha256sum -c verifies listed files, not that no extra files exist. A new empty release directory avoids stale destination files, but does not detect extra source files added after hashing. To establish complete regular-file inventory parity, recompute the staged manifest with the same path/sort/hash rules and compare the full lists before selection. Inspect symlinks separately: find -type f does not include them, and serving an unintended symlink can expose files outside the intended root. Keep the manifest and its own digest with the release record; a checksum proves consistency against that manifest, not authenticity of an untrusted manifest.

4. Stage and select a complete release

Assume Caddy already serves /srv/www/example-site/current, which is a symlink into releases/, with correct ownership and read permissions. The releases parent must already exist. Serialize deployers and choose a unique release ID. Run this block as a standalone Bash script on the deployment machine, from the directory containing the validated public/ and release.sha256. Check the artifact’s ownership and readable permissions against the deployment policy before running it. It establishes its own manifest path and fail-fast policy; a failed precondition, copy, or checksum must stop before pointer selection. It refuses symlinks inside the staged artifact and is a mutating deployment step.

set -euo pipefail
base=/srv/www/example-site
manifest="$(pwd)/release.sha256"
test -f "$manifest"
test -d public
release_id=20261015T120000Z
release="$base/releases/$release_id"
test -L "$base/current"
previous=$(readlink "$base/current")
mkdir "$release" # Fail if this release already exists.
rsync -a public/ "$release/"
(cd "$release" && sha256sum -c "$manifest")
links=$(find "$release" -type l -print -quit)
if [ -n "$links" ]; then
 echo "Refusing symlinks in the staged artifact" >&2
 exit 1
fi
# Ownership and readable permissions must match the deployment policy.
python3 - "$base" "$release_id" <<'PYTHON'
from pathlib import Path
import os, sys, uuid
base = Path(sys.argv[1])
release = base / "releases" / sys.argv[2]
current = base / "current"
if not current.is_symlink() or not release.is_dir():
 raise SystemExit("Expected current symlink and a complete release directory")
pending = base / (".current-" + uuid.uuid4().hex)
pending.symlink_to(os.path.relpath(release, base), target_is_directory=True)
os.replace(pending, current)
print("selected:", os.readlink(current))
PYTHON
printf 'Previous target: %s\n' "$previous"

The symlink replacement uses one same-filesystem rename because the pending link and current share a parent. An ordinary mv new existing-directory does not exchange non-empty directories. Do not convert an existing real-directory serve root with this snippet; plan that migration separately. Validate the actual server’s root resolution and filesystem behavior in staging.

The old target stays on disk, but that alone does not make an old asset URL reachable after switching. Retain fingerprinted asset bytes in the newly served namespace or a stable asset store. Multiple browser requests can span releases, and caches may keep old HTML. Atomic namespace replacement is not cross-request snapshot isolation or a power-loss durability guarantee.

5. Verify delivered bytes and policy

After switching, compare a sampled public body with the corresponding artifact file, using a fixed representation selection:

set -euo pipefail
url=https://www.xbcatm.com/
status=$(curl -sS --max-time 20 -H 'Accept-Encoding: identity' \
 -D served.headers -o served.html -w '%{http_code}' "$url")
test "$status" = 200
cmp public/index.html served.html
sh public/downloads/check-http-headers.sh "$url"

Run this block from the same artifact directory used for staging. The header helper is the trusted copy inside that artifact; a source checkout is not required for this step. cmp succeeding establishes exact parity for this homepage sample. An unexpected redirect, proxy-generated 200 error page, CDN transformation, or encoded response needs investigation. Inspect served.headers, including Content-Encoding, cache age, CSP, and content type. The header helper prints responses and final status; it does not enforce all your expectations.

Repeat for Chinese content, a nested article, CSS, an SVG, a download, redirects, and the error route. Document the URL-to-file mapping, including directory indexes. The shown listed-file check establishes consistency for those files, not a complete staged inventory or what all public cache nodes deliver. A complete edge claim requires broader retrieval and representation-aware checks; a few probes remain samples.

If a post-switch check fails, the new release is still selected until you act. Use a new temporary symlink targeting the recorded previous value, atomically replace current again, and repeat the probes. Do not merely move the new tree away, and do not delete evidence or the previous release while diagnosing.

6. Check TLS identity and the expiry window separately

From the repository root, the expiry helper is:

sh static/downloads/check-tls-expiry.sh www.xbcatm.com 30

It extracts a certificate, prints dates, and checks the remaining expiry interval. It is not a full TLS identity test: its current implementation does not make hostname/trust verification failure fatal. Follow the public-edge trust tutorial for an explicit verified handshake and SAN check. Do not infer trust-chain or revocation success from “valid for at least 30 days.”

A near-expiry certificate is a separate release concern. Preserve the actual notAfter, trust result, and client environment rather than copying a historical issuer or date into the release log.

7. Include the DNS cache window when DNS changes

A content-only release need not change DNS. If the host or edge changes, preserve service at the old destination while caches expire. Lowering TTL at the moment of the change does not shorten entries already cached with the old TTL. Resolver serve-stale behavior, negative caching, and client caches can complicate the window; TTL is not a universal deadline by which every client has moved. CAA changes affect issuance policy, not routing to the new content.

8. Keep a rehearsed restore path

The previous directory is useful for fast rollback, but it is not an independent backup. A disk failure, accidental cleanup, or compromised deployment account may affect every local release. Retain source, build inputs, configuration, and an appropriate separate artifact backup; rehearse restoring them in isolation. One successful rehearsal is evidence about that recovery path at that time, not a permanent guarantee for later changes.

9. Record the outcome precisely

Keep the source revision and clean-tree status, build metadata and tool versions, manifest digest, selected and previous targets, probe timestamps, HTTP/TLS observations, and any rollback. State whether verification covered samples, all mapped URLs, or multiple vantage points. If a gate failed, record the failure and recovery rather than “it worked.”

Before selection, a failure can leave the old release untouched. After selection, recovery is an explicit operation with its own verification. This distinction makes the release path auditable without pretending every failure automatically restores the old tree.

Sources

rsync manual, archive copying and source trailing-slash semantics, checked 2026-10-15.
RFC 8767: Serving Stale Data, DNS serve-stale behavior beyond TTL expiry, checked 2026-10-15.
GNU sha256sum manual, Debian package, checksum output and verification flags, checked 2026-10-15.
Python os.replace, atomic replacement and filesystem limits, checked 2026-10-15.
Caddy file_server, root and symlink handling, checked 2026-10-15.
curl manual, status capture, headers, and content decoding, checked 2026-10-15.
OpenSSL s_client, explicit certificate verification, checked 2026-10-15.
Reproducible Builds: definitions, artifact identity versus reproducibility, checked 2026-10-15.