Operations · XB Field Notes

Startup probes delay other probes until initialization succeeds

Sun, 22 Aug 2027 00:00:00 +0000

A startup probe gives a container a separate initialization contract. While that configured probe has not succeeded, Kubernetes does not execute the container’s liveness or readiness probes. This is stronger and more specific than ignoring some liveness failures during a grace period: those other checks have not begun. Once startup succeeds, ordinary ongoing health and readiness behavior takes over.

The threshold-times-period product helps planning but is not an exact restart time. Startup success does not force readiness or guarantee continuing health. The retained Kubernetes integration was not rerun.

Give each stage an observable condition

The application process starts before its startup probe can succeed. A Running container may therefore still have started reported as false in container status and may not make its Pod Ready. Running describes execution; successful startup describes completion of the configured startup check. An HTTP listener that opens early can expose progress without yet claiming that initialization is complete.

Choose a startup condition that represents necessary local preparation, such as loading an index or validating required initial state. A port opening is sufficient only if that is the actual readiness prerequisite you intend to observe. If the listener appears before expensive setup finishes, checking only that listener can release liveness protection too early and recreate the original restart loop.

After a successful startup check, kubelet stops that startup probe for the current container instance. It runs liveness and readiness according to their separate configuration. Startup success is not a permanent health certificate: the application can later become unready or require a restart. A new container instance must complete its own startup check again.

When it breaks

A startup probe that never succeeds eventually reaches its failure threshold. Kubelet then terminates the container and applies its restart policy. Startup protection is therefore not permission to initialize forever. Set period, timeout and failure threshold from measured initialization behavior, allowing for the environment’s realistic resource contention and expected recovery time.

The product of failureThreshold and periodSeconds is useful when planning the startup allowance, but do not present it as an exact stopwatch for every observed restart. Initial delay, probe execution and scheduling, termination grace and restart backoff also affect the timeline. Record the actual attempts and state changes when investigating a slow deployment.

Do not make successful readiness a prerequisite for liveness unless that dependency is explicitly represented by startup configuration. Readiness and liveness otherwise answer separate ongoing questions. Equally, passing startup does not force readiness to succeed; a later readiness check can still reject traffic eligibility for a different reason, while the process remains alive.

Changing checks requires application support. An endpoint that always returns success cannot verify a missing initialization step merely because it is named startup. Keep its state transition deliberate, expose enough diagnostic information to explain waiting, and avoid placing credentials or sensitive initialization details in responses readable by unintended clients. The fixture below uses only counters and synthetic state.

Check it quickly

The fixture and observations below are retained from the original article. This review did not rerun the Kubernetes integration; reported success is historical, not a new execution result.

Use a disposable Kubernetes cluster, kubectl and python:3.11-slim-bookworm available to its nodes. Save the script as startup-gate.py and run python3 startup-gate.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. Namespace creation and deletion permissions are required. The recorded server was Kubernetes 1.34.11. The startup probe allows sixty failed attempts at one-second periods for this short controlled exercise.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-startup-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 server = '''from http.server import BaseHTTPRequestHandler,HTTPServer
from pathlib import Path
import json
counts={'startup':0,'live':0,'ready':0}
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 key=self.path.removeprefix('/'); code=200
 if key in counts:
 counts[key]+=1
 if key=='startup' and not Path('/tmp/start').exists(): code=503
 body=json.dumps(counts).encode(); self.send_response(code)
 self.send_header('Content-Length',str(len(body))); self.end_headers(); self.wfile.write(body)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''
 podspec = spec(server)
 c = podspec['containers'][0]
 c['startupProbe'] = {'httpGet': {'path': '/startup', 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 60}
 for field, path in [('livenessProbe', '/live'), ('readinessProbe', '/ready')]:
 c[field] = {'httpGet': {'path': path, 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 2}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'slow'}, 'spec': podspec})
 wait_for(lambda: obj(namespace, 'pod', 'slow'), lambda p:
 any('running' in c['state'] for c in p.get('status', {}).get('containerStatuses', [])))
 read = 'from urllib.request import urlopen; print(urlopen("http://127.0.0.1:8080/stats",timeout=3).read().decode())'
 def counts():
 result = k('exec', '-n', namespace, 'slow', '--', 'python3', '-c', read, expected=None)
 return json.loads(result.stdout) if result.returncode == 0 else None
 before = wait_for(counts, lambda d: d and d['startup'] >= 3)
 assert before['live'] == before['ready'] == 0
 starting = obj(namespace, 'pod', 'slow')
 assert not pod_ready(starting)
 assert starting['status']['containerStatuses'][0]['started'] is False
 assert starting['status']['containerStatuses'][0]['restartCount'] == 0
 k('exec', '-n', namespace, 'slow', '--', 'python3', '-c',
 'from pathlib import Path; Path("/tmp/start").touch()')
 ready = wait_for(lambda: obj(namespace, 'pod', 'slow'), pod_ready)
 assert ready['status']['containerStatuses'][0]['started'] is True
 enabled = wait_for(counts, lambda d: d and d['live'] > 0 and d['ready'] > 0)
 time.sleep(3)
 after = counts()
 assert after['startup'] == enabled['startup']
 assert after['live'] > enabled['live'] and after['ready'] > enabled['ready']
 assert obj(namespace, 'pod', 'slow')['status']['containerStatuses'][0]['restartCount'] == 0
 print(json.dumps({'before_startup_success': before, 'after_startup_success': after,
 'other_probes_suppressed_before_startup': True,
 'startup_probe_stops_after_success': True, 'container_restarts': 0}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The HTTP server counts requests separately for startup, live and ready paths. Its statistics path is observational and does not increment those counters. Startup initially returns 503 until a local release file appears. After three observed startup requests, both other counters are zero, the container has not restarted, and its started field is false.

Creating the release file makes the next startup request succeed. The Pod becomes Ready, started becomes true, and the live and ready counters begin increasing. After another three seconds, the startup counter stays fixed while both ongoing counters increase. The recorded final counts were four, four and five; exact ongoing counts can vary with timing.

The original article reports that all assertions passed without a restart, and the namespace was removed. This verifies suppression and transition using actual HTTP calls. It does not measure a production boot-time percentile or exercise startup-budget exhaustion. Use those additional measurements when choosing real thresholds, while keeping startup completion distinct from continuing service health.

Sources

Kubernetes: Configure Probes, startup failure budgets and probe configuration, checked 2026-10-15.
Kubernetes: Pod Probes, suppression of liveness and readiness until startup succeeds, checked 2026-10-15.

Change windows fit people, not servers

Thu, 29 Jul 2027 00:00:00 +0000

A change that races to zero traffic but lands when nobody is watching trades one risk for another. The best window is one where the operators responsible can actually attend.

Traffic is only part of the math

Deploying at the absolute quietest hour minimizes how many users feel the change, but it collides with the people problem: the operator is most likely asleep, unreachable, or already covering another incident. If the rollout breaks at 3 a.m. and nobody is watching, a silent failure can run for hours before a human reads it. Least-traffic is a good starting estimate, but it is a traffic answer, not a risk answer. Balance reduced exposure against the ability to react, and prefer a window where you can actually observe the result.

Watch the human clock

The window has to fit the humans, not just the chart. Consider who must notice a broken deploy: the SRE on call, the feature owner, the support team, and schedule their attention into the slot. Also watch what else is scheduled — a parallel release, a job run, an adjacent change — that could tangle with yours and turn a clean rollout into a confusing blame-fest. A slot where the people who must read the result are awake, reachable, and not already overloaded beats a quieter one where nobody is watching.

Let the window be a habit

A recurring staffed window can help coordinate risky changes and make observation predictable. It is not a universal requirement to batch every release: small, automated, independently reversible changes may use a continuous-delivery workflow. Choose the cadence from actual risk and response coverage. Automate builds, smoke tests, and canaries, while preserving an accountable owner and a tested recovery path. A calendar slot should support those safeguards, not replace them.

When it breaks

An urgent fix arrives outside the window and there is no emergency lane. The team either waits while the incident burns, or someone pushes straight to production with no watcher and no rollback note. Without a documented break-glass path — who approves, who watches, what gets written down — every exception becomes a secret deploy. Rehearse an explicit emergency procedure alongside the normal schedule.

A second failure is the stampede inside the window. Four services land in the same two hours, their signals blend together, and when the error rate climbs nobody can name the culprit. Rollback becomes a negotiation instead of a button press. Serialize changes that share failure signals or dependencies, and require every entry to name its owner, signal, and way back.

A third failure is the cross-timezone misread. The window says Wednesday 10:00 in Berlin, but a teammate in Los Angeles reads it as local time and ships while responders sleep. Daylight saving quietly shifts the slot by another hour. Store every window with its zone, render it locally, and confirm a human is present before counting a slot as staffed.

One instant, different human clocks. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

Use Python 3.9 or newer with IANA time-zone data available to zoneinfo; the module does not bundle that data. Save the following as change_window_check.py and run python3 change_window_check.py. Run without -O, -OO, or a nonzero PYTHONOPTIMIZE setting so the assertions execute. It uses only the standard library, treats every window as timezone-aware, and decides between allow and needs-approval. The table holds same-day Wednesday slots in Berlin and New York. Freeze dates use an explicitly chosen UTC policy day, regardless of the timezone used to express the input instant. The bad samples are negative controls.

from datetime import datetime, time
from zoneinfo import ZoneInfo

WINDOWS = [
 {"name": "berlin-wed", "weekday": 2, "start": "09:00", "end": "11:00", "tz": "Europe/Berlin"},
 {"name": "newyork-wed", "weekday": 2, "start": "09:00", "end": "11:00", "tz": "America/New_York"},
]
FREEZE_DATES = frozenset({"2027-12-24"})
FREEZE_ZONE = ZoneInfo("UTC") # Organization-wide policy, independent of input representation.


def _hhmm(value):
 hour, minute = map(int, value.split(":"))
 return time(hour, minute)


def decide(moment, windows=WINDOWS, freeze=FREEZE_DATES):
 if moment.tzinfo is None or moment.utcoffset() is None:
 return "needs-approval", "naive datetime carries no zone; attach a ZoneInfo first"
 freeze_day = moment.astimezone(FREEZE_ZONE).date()
 if freeze_day.isoformat() in freeze:
 return "needs-approval", f"freeze date {freeze_day} in UTC; emergency path only"
 for window in windows:
 local = moment.astimezone(ZoneInfo(window["tz"]))
 if local.weekday() == window["weekday"] and _hhmm(window["start"]) <= local.time() < _hhmm(window["end"]):
 return "allow", f"inside {window['name']} ({local:%Y-%m-%d %H:%M %Z})"
 return "needs-approval", f"outside all windows ({moment:%Y-%m-%d %H:%M %Z})"


def check(actual, expected, label):
 status, reason = actual
 assert status == expected, f"{label}: got {status} ({reason})"
 print(f"{label}: {status} -- {reason}")


def main():
 berlin = ZoneInfo("Europe/Berlin")
 newyork = ZoneInfo("America/New_York")
 good_berlin = datetime(2027, 8, 4, 9, 30, tzinfo=berlin)
 good_ny = datetime(2027, 8, 4, 9, 30, tzinfo=newyork)
 check(decide(good_berlin), "allow", "good/berlin-wed-morning")
 check(decide(good_ny), "allow", "good/newyork-wed-morning")
 # Same UTC instant, different local reading: 07:30 UTC is 09:30 in Berlin but 03:30 in New York.
 same_instant = datetime(2027, 8, 4, 7, 30, tzinfo=ZoneInfo("UTC"))
 check(decide(same_instant), "allow", "good/same-instant-via-berlin")
 evening = datetime(2027, 8, 4, 22, 10, tzinfo=berlin)
 check(decide(evening), "needs-approval", "bad/wednesday-night")
 wrong_day = datetime(2027, 8, 5, 9, 30, tzinfo=berlin)
 check(decide(wrong_day), "needs-approval", "bad/thursday-morning")
 frozen = datetime(2027, 12, 24, 9, 30, tzinfo=berlin)
 check(decide(frozen), "needs-approval", "bad/freeze-date")
 naive = datetime(2027, 8, 4, 9, 30)
 check(decide(naive), "needs-approval", "bad/naive-datetime")
 misread = datetime(2027, 8, 4, 9, 30, tzinfo=ZoneInfo("America/Los_Angeles"))
 check(decide(misread), "needs-approval", "bad/timezone-misread")
 check(decide(datetime(2027, 8, 4, 9, 0, tzinfo=berlin)), "allow", "good/start-inclusive")
 check(decide(datetime(2027, 8, 4, 11, 0, tzinfo=berlin)), "needs-approval", "bad/end-exclusive")
 winter = datetime(2027, 1, 6, 8, 30, tzinfo=ZoneInfo("UTC"))
 check(decide(winter), "allow", "good/winter-berlin")
 # This Wednesday would otherwise pass: the freeze test cannot pass by accident.
 frozen_wednesday = frozenset({"2027-08-04"})
 shifted = same_instant.astimezone(ZoneInfo("Etc/GMT+12"))
 assert shifted == same_instant and shifted.date() != same_instant.date()
 check(decide(same_instant, freeze=frozen_wednesday), "needs-approval", "bad/UTC-freeze")
 check(decide(shifted, freeze=frozen_wednesday), "needs-approval", "bad/same-freeze-other-zone")
 assert decide(same_instant, freeze=frozen_wednesday) == decide(shifted, freeze=frozen_wednesday)
 print("PASS: 5 allows and 8 approvals; policy dates do not depend on caller timezone")


if __name__ == "__main__":
 main()

The script prints five allow results and eight needs-approval results. Window starts are inclusive and ends exclusive. The summer and winter UTC cases reach Berlin at 09:30 despite different offsets. A Wednesday freeze rejects an otherwise eligible instant both in UTC and in a representation whose local date is the previous day. Checking the input date directly would incorrectly allow that second representation.

These revised assertions passed with Python 3.12.14. Allow means only that this schedule predicate passed; it grants no deployment authorization and verifies no staffing. This same-day fixture does not handle overnight windows or validate ambiguous local-time construction. Convert a known instant, then separately check approvals, the on-call roster, observation time, and recovery readiness.

Sources

Google SRE: release engineering, release as an engineered practice, checked 2026-10-15.
12Factor: build release run, separating build from release, checked 2026-10-15.
Python datetime, aware timestamps and conversion between timezone representations, checked 2026-10-15.
Python zoneinfo, IANA timezone rules and daylight-saving transitions, checked 2026-10-15.

A rota needs continuous coverage and a load review

Fri, 16 Jul 2027 00:00:00 +0000

A rota can distribute responsibility, but equal shift counts do not guarantee equal burden. Review nighttime pages, incident effort, recovery time, and reliable backup alongside the schedule.

A job that is shared

On-call work is a job that belongs to a team, not to one hero. A rotation turns ‘whoever was last awake’ into a defined roster and a schedule, so responsibility is explicit and no one is assumed to carry the whole queue indefinitely. The schedule itself — a weekly handoff, a backup/secondary on-call, a follow-the-sun structure across sites — is how you keep the obligation sustainable without letting it turn into a possession. Don’t let the urgency of tonight’s page decide who owns tomorrow; the rotation decides on purpose, ahead of time.

Handoff is half the craft

Handoff is where a rotation most often breaks. When the incoming person inherits a blank slate, what should be a handoff becomes a re-investigation. Two habits close that gap: a written handoff (what is broken, what has been tried, what the next person should own) and an overlap window so the incoming engineer is shadowed by the outgoing one before they leave. A rotation that swaps names but not context turns the same incident into a fresh one every week. The craft is not the schedule; it is the continuity the schedule protects.

Rehearse before the bell rings

The time to rehearse is before the pager rings, not during. Run the runbook in quiet hours, drill the restore path (not just the diagnosis), and map each alert to a decision tree so a cold Sunday-morning page is a checklist rather than a riddle. Rehearsal improves the team’s options under pressure: the cheap, repeatable steps taken calmly in daylight are what keep the team from improvising during the real incident. That rehearsal is the difference between ‘we will figure it out’ and a boring, repeatable process.

When it breaks

Single-person coverage looks simple until it breaks the person. One engineer holding the pager for months stops sleeping well, dreads every notification, and eventually burns out or quits; Google’s SRE guidance emphasizes sustainable workload and staffing. The fix is a bounded rotation with a maximum tour length and a real off-call week where the previous primary is not the silent backup.

A broken escalation chain fails differently. If the secondary never acknowledges, contacts are stale, or paging the manager reaches a mailbox nobody reads, a 3 a.m. incident waits while the primary debugs alone. Rehearse the chain like a restore: page the secondary in business hours and confirm a human answers within minutes.

Alert fatigue is the third failure. Hundreds of noisy, unactionable alerts teach the team the pager lies, so the one real page is swiped away with the rest. Every alert needs an owner, a runbook link, and a threshold tuned after the last false positive.

Missing handoffs complete the set. The name changes on Monday with no written state, so Tuesday repeats Monday’s investigation from scratch. Require a short note — symptoms, tried steps, next move — plus a fifteen-minute overlap before the old primary goes dark.

Keep both roles covered through every week. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

Use this paper-model exercise before publishing a schedule. It assigns fictional primary and secondary responders for twenty consecutive weeks. Two invented special-coverage weeks remain staffed; they are not assertions about any country’s holidays. A real continuously operated service still needs coverage when an engineer is on leave.

Save the snippet as rota_probe.py in an empty directory.
Run python3 rota_probe.py with Python 3.9 or newer; it touches no network or files. Run without -O, -OO, or a nonzero PYTHONOPTIMIZE setting so the assertions execute.
Check the weekly assignments, role counts, and final PASS line.

from collections import Counter
from datetime import date, timedelta

TEAM = ["ana", "ben", "cara", "dev", "elin"] # fictional names
START = date(2027, 8, 2)
WEEKS = 20
SPECIAL_WEEKS = {START + timedelta(weeks=1), START + timedelta(weeks=4)}
rota = [(START + timedelta(weeks=i), TEAM[i % 5], TEAM[(i + 1) % 5])
 for i in range(WEEKS)]
assert len(rota) == WEEKS
assert len({week for week, _, _ in rota}) == WEEKS
assert all(primary != secondary for _, primary, secondary in rota)
assert all(primary in TEAM and secondary in TEAM for _, primary, secondary in rota)
assert all(b[0] - a[0] == timedelta(days=7) for a, b in zip(rota, rota[1:]))
assert all(a[1] != b[1] for a, b in zip(rota, rota[1:]))
primary_counts = Counter(primary for _, primary, _ in rota)
secondary_counts = Counter(secondary for _, _, secondary in rota)
assert set(primary_counts.values()) == {4}
assert set(secondary_counts.values()) == {4}
for week, primary, secondary in rota:
 marker = " special-coverage" if week in SPECIAL_WEEKS else ""
 print(f"{week}: primary={primary} secondary={secondary}{marker}")
assert sum(week in SPECIAL_WEEKS for week, _, _ in rota) == 2
print("primary:", dict(sorted(primary_counts.items())))
print("secondary:", dict(sorted(secondary_counts.items())))
print("PASS: 20 covered weeks, distinct backup, four turns per role; special weeks remain staffed")

Every week has distinct primary and secondary names, including both marked weeks. Each engineer receives four turns in each role, and adjacent weeks have different primaries. These assertions passed in Python 3.12.14 during this review. They establish the stated roster properties only; equal counts do not measure equal overnight interruptions, overlapping incidents, availability, time zones, compensation, or recovery time. Obtain team agreement on those constraints and test the actual escalation contacts before activating a real rota.

Sources

SRE Workbook: on-call, structuring fair on-call rotations, checked 2026-10-15.
SRE book: being on-call, on-call practice and limits, checked 2026-10-15.

Backups are untested hope without proof

Wed, 02 Jun 2027 00:00:00 +0000

A completed backup job establishes that the backup process produced an artifact. A restore rehearsal establishes whether that artifact, the available tooling, and the required environment can produce a usable system. These are different claims. Recovery evidence should identify the backup used, the state recovered, the dependencies supplied, and the time needed to make the result useful to the application.

When it breaks

A valid database archive can depend on objects outside its scope. PostgreSQL pg_dump exports one database, while roles are cluster-wide objects. Restoring a dump with grants into an independent cluster can fail because the referenced role does not exist. Rehearsing only in the original cluster can hide that dependency because the missing recovery material is already present there.

Restore errors also need deliberate handling. By default, pg_restore can continue after SQL errors and report their count at the end. A half-restored schema should not be accepted merely because some tables can be queried. For this small fixture, a single transaction makes success atomic and makes an expected failure leave no partial restored tables.

The correct recovery point is another requirement. A perfectly restored backup may intentionally lack changes made afterward. A checksum proves agreement with the recorded artifact, not that the artifact includes the newest business event. Likewise, a row count alone can match while amounts, relationships, or privileges are wrong. Choose checks that express the application’s meaning.

A checksum does not supply a missing cluster role. Restore the same archive into a fresh target, then check the intended business state, constraints and effective privileges. The source’s PostgreSQL drill is not rerun.

Check it quickly

Run this script as a non-root Linux account with PostgreSQL 18 server and client programs and sha256sum available. It was executed on PostgreSQL 18.6. It creates separate temporary source and target clusters with private Unix sockets and no TCP listeners. The fixture role cannot log in; its grants are tested through controlled role switching rather than through real credentials.

#!/bin/sh
set -eu
umask 077
lab=$(mktemp -d /tmp/xbcatm-restore.XXXXXX)
mkdir "$lab/socket"
source=$lab/source
target=$lab/target
cleanup() {
 for cluster in "$target" "$source"; do
 if test -f "$cluster/postmaster.pid"; then
 pg_ctl -D "$cluster" -m fast -w stop >/dev/null
 fi
 done
}
trap cleanup EXIT
for cluster in "$source" "$target"; do
 initdb -D "$cluster" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject >/dev/null
 cat >> "$cluster/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$lab/socket'
CONF
done
printf 'port = 56432\n' >> "$source/postgresql.conf"
printf 'port = 56433\n' >> "$target/postgresql.conf"
pg_ctl -D "$source" -l "$lab/source.log" -w -t 15 start >/dev/null
pg_ctl -D "$target" -l "$lab/target.log" -w -t 15 start >/dev/null
export PGHOST=$lab/socket PGPORT=56432 PGDATABASE=postgres PGUSER=$(id -un)
export PGCONNECT_TIMEOUT=2
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
CREATE ROLE backup_reader NOLOGIN;
CREATE TABLE customers (id integer PRIMARY KEY,name text NOT NULL);
CREATE TABLE orders (id integer PRIMARY KEY,customer_id integer REFERENCES customers,amount_cents integer CHECK (amount_cents>0));
INSERT INTO customers VALUES (1,'Ada');
INSERT INTO orders VALUES (10,1,4210),(11,1,1865);
CREATE VIEW order_totals AS SELECT c.name,sum(o.amount_cents) AS total_cents FROM customers c JOIN orders o ON o.customer_id=c.id GROUP BY c.id,c.name;
GRANT SELECT ON customers,orders,order_totals TO backup_reader;
SQL
pg_dump -Fc -w -f "$lab/snapshot.dump" postgres
sha256sum "$lab/snapshot.dump" > "$lab/snapshot.sha256"
psql -X -v ON_ERROR_STOP=1 -qAt -c 'INSERT INTO orders VALUES (12,1,1000)'
printf 'source_after_backup_orders '
psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT count(*) FROM orders'
sha256sum -c "$lab/snapshot.sha256"
if PGPORT=56433 pg_restore --single-transaction --exit-on-error -w -d postgres "$lab/snapshot.dump" >"$lab/missing-role.log" 2>&1; then
 echo 'restore unexpectedly ignored a missing role' >&2
 exit 1
fi
grep -F 'role "backup_reader" does not exist' "$lab/missing-role.log"
test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT to_regclass('public.orders') IS NULL")" = t
echo failed_restore_rolled_back
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'CREATE ROLE backup_reader NOLOGIN'
PGPORT=56433 pg_restore --single-transaction --exit-on-error -w -d postgres "$lab/snapshot.dump"
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
SELECT count(*) FROM orders;
SELECT name,total_cents FROM order_totals;
SELECT has_table_privilege('backup_reader','orders','SELECT'),has_table_privilege('backup_reader','orders','INSERT');
DO $$ BEGIN
 IF (SELECT count(*) FROM orders)<>2 OR
 (SELECT string_agg(id||':'||amount_cents,',' ORDER BY id) FROM orders) IS DISTINCT FROM '10:4210,11:1865' OR
 (SELECT total_cents FROM order_totals WHERE name='Ada') IS DISTINCT FROM 6075 OR
 (SELECT count(*) FROM orders o LEFT JOIN customers c ON c.id=o.customer_id WHERE c.id IS NULL)<>0 THEN
 RAISE EXCEPTION 'restored business data mismatch';
 END IF;
END $$;
DO $$ BEGIN
 BEGIN
 INSERT INTO orders VALUES (98,999,100);
 RAISE EXCEPTION 'restored foreign key unexpectedly accepted an orphan';
 EXCEPTION WHEN foreign_key_violation THEN
 RAISE NOTICE 'restored foreign key rejected orphan';
 END;
END $$;
SET ROLE backup_reader;
SELECT id,amount_cents FROM orders ORDER BY id;
DO $$ BEGIN
 BEGIN
 INSERT INTO orders VALUES (99,1,100);
 RAISE EXCEPTION 'read-only role unexpectedly wrote data';
 EXCEPTION WHEN insufficient_privilege THEN
 RAISE NOTICE 'read-only role rejected write';
 END;
END $$;
SQL
printf 'lab_directory %s\n' "$lab"

The source contains one customer, two orders, a totals view, a foreign key, and a read-only role grant. A custom-format dump captures that state. The source then receives a third order. Its printed count of three confirms that later activity exists outside the chosen backup point, rather than allowing a test to confuse the current source with the saved snapshot.

The checksum check succeeds, but the first restore must still fail because backup_reader is absent on the target. The script checks that specific error and verifies the orders table was rolled back. This is the dependency failure that an artifact-only check would miss. It then creates the documented fixture role and repeats the restore with the same archive.

The recovered target must contain two orders with the original identifiers and amounts, and the view must report Ada|6075. The third order is correctly absent. An attempted orphan order must fail through the restored foreign key. Permission checks report allowed selection and denied insertion; switching to the reader role confirms that reads work and a real write attempt is rejected.

These assertions cover data values, an application query, a relationship constraint, and effective privileges. All experimental servers stop on exit, and the printed directory retains the dump and logs. The drill does not test external authentication, encryption-key recovery, or application startup, which must be added when those are part of the actual recovery dependency set.

Record a recovery result people can use

A production rehearsal should retrieve the artifact from its intended recovery source, rebuild the required environment, restore data, and exercise representative application behavior. Provisioning compatible software is a normal part of recovery; include its elapsed time and prerequisites instead of assuming a prepared host will always exist. Keep recovery access available when production systems are unavailable.

Measure recovery duration against the recovery-time objective, and compare the recovered endpoint with the recovery-point objective. Include retrieval, validation, and the application becoming usable, not just the database command’s runtime. For larger restores, choose transaction and parallelism options appropriate to the workload, with an explicit partial-failure policy.

Repeat the rehearsal after changes to backup tooling, schema, roles, retention, or infrastructure. Preserve the successful checks and any corrected failure as evidence. A repeatable restore procedure with known inputs and limits supports an operational decision; a green backup scheduler alone does not answer whether recovery is ready.

Sources

PostgreSQL 18 pg_dump, database-level scope and excluded cluster-wide objects, checked 2026-09-12.
PostgreSQL 18 pg_restore, error handling, single-transaction restoration, grants, and parallelism constraints, checked 2026-09-12.
CISA StopRansomware Guide, testing backup availability and integrity with recovery dependencies, checked 2026-09-12.

Bucket versioning and lifecycles manage cost

Sat, 08 May 2027 00:00:00 +0000

S3 Versioning retains multiple versions under one key, making many accidental overwrites and ordinary deletes recoverable while the required version still exists. It does not protect against every deletion or provide an independent backup by itself. Retention, permissions, encryption-key availability, and restoration procedures determine whether the saved history can actually recover the application.

Distinguish versioning states

In a versioning-enabled bucket, a PUT creates a new version ID and preserves earlier data versions. Each version stores the whole object, not merely its changed bytes. Repeatedly overwriting a large object therefore has a different storage profile from a block-level incremental snapshot, even when the visible key count stays constant.

An object that predates enablement keeps its null version ID. New versioned writes do not overwrite that null version. Once enabled, a bucket cannot return to the never-versioned state, although versioning can be suspended. Under suspension, subsequent writes use the null-version slot and can replace an existing null version. Specify which state applies before describing overwrite safety.

Null is a real version identifier, not a guarantee of permanent retention. A null version that becomes noncurrent can be subject to noncurrent expiration. It can also be explicitly targeted for deletion when authorized. A listing containing null therefore does not prove that old data is exempt from lifecycle cleanup.

A delete marker changes visibility

An ordinary DELETE without a version ID in an enabled bucket creates a delete marker. When that marker is current, a versionless GET normally returns 404; explicitly requesting a delete-marker version returns 405. Earlier data remains available by its data-version ID, subject to access, storage-class, and encryption requirements.

Delete markers have no object payload but are not free zero-byte storage entries: S3 charges for the key-name bytes they store. Removing the specific current marker exposes the preceding version, which could itself be another marker. Alternatively, copying an explicit earlier data version creates a new current data version while preserving history. A marker cannot be copied as if it were object data.

Deleting a particular data version is a different, permanent operation. Versioning alone cannot restore that deleted version. MFA Delete, when configured, adds authentication for permanent version deletion and changes to versioning state; it is not automatically active whenever versioning is enabled. AWS does not support using MFA Delete with lifecycle configurations.

Lifecycle expresses a retention tradeoff

Current-version Expiration and NoncurrentVersionExpiration do different jobs. In an enabled bucket, expiring current data generally creates a marker and makes that data noncurrent. Noncurrent expiration permanently removes eligible older data versions. Eligibility uses time since becoming noncurrent, and optional retained-version counts add another condition. It is not simply the object’s original upload age.

Lifecycle execution is asynchronous. A thirty-day rule is not proof that exactly thirty versions remain at every instant. Rule filters, write frequency, retained-count settings, replication status, and Object Lock can affect what qualifies. Transitions to colder storage retain data with different storage, retrieval, and minimum-duration costs; calculate those costs against the actual object sizes and recovery needs.

When it breaks

Growing noncurrent bytes can raise costs while ordinary listings appear stable. Inspect version inventory and lifecycle scope rather than expecting a fixed version-to-key ratio to diagnose every case. A bucket does not need to be physically “full” for this to become expensive, and lifecycle is not a hard spending cap or the only possible retention control.

Overaggressive expiration creates the opposite problem: the required recovery version is permanently removed. Validate retention against the longest plausible detection delay and restore requirements. HEAD can inspect metadata but does not download and validate object contents; a successful metadata request alone is not a completed restoration test.

Version visibility — mechanism diagram.

Check it quickly

Run this Python 3 model with assertions enabled. Retained historical executions cover these exact model bytes; this source review did not rerun them. It models an enabled bucket’s version stack entirely in memory, including a preexisting null data version. No AWS account, credentials, or remote object is used. Marker storage billing, real timestamps, permissions, lifecycle scheduling, and concurrent writes are deliberately outside the model.

versions = [("null", b"old"), ("v1", b"new")]
marker = ("d1", None)
versions.append(marker)
def get_current():
 payload = versions[-1][1]
 return (404, None) if payload is None else (200, payload)
assert get_current() == (404, None)
assert sum(len(data) for _, data in versions if data is not None) == 6
versions.remove(marker)
assert get_current() == (200, b"new")
print("remove current marker: previous data visible")
versions.append(("d2", None))
# Copy an explicit data version to create a new current version.
old = next(data for identifier, data in versions if identifier == "null")
versions.append(("v2", old))
assert get_current() == (200, b"old")
assert any(identifier == "d2" for identifier, _ in versions)
print("copy explicit data version: new current version; history retained")
# Noncurrent null versions are not inherently protected from deletion.
versions[:] = [entry for entry in versions if entry[0] != "null"]
assert get_current() == (200, b"old")
print("remove old null version: current copy remains")

The three outputs distinguish exposing a previous version, copying explicit historical data, and removing a noncurrent null version. Stored payload bytes remain through marker creation; this does not count the marker’s billable key bytes. For real recovery, inspect the exact key across paginated data-version and delete-marker listings, select an explicit data version, retrieve it, and validate its contents before changing production state.

Sources

AWS S3 versioning, how versioning and the null version work, checked 2026-09-16.
AWS S3 delete markers, what a delete marker does and how to undo it, checked 2026-09-16.
AWS S3 lifecycle management, current and noncurrent expiration and transition rules, checked 2026-09-16.
AWS expiring objects, eligibility, permanent noncurrent deletion and asynchronous execution, checked 2026-10-15.
AWS MFA Delete, optional protection and lifecycle constraints, checked 2026-10-15.
AWS restoring previous versions, explicit-version copying and restoration, checked 2026-10-15.

A pinned base fixes one input; the release needs the rest

Fri, 30 Apr 2027 00:00:00 +0000

Pinning a container base by digest fixes the referenced image object instead of relying on a movable tag. That removes one source of unexpected build changes, but it does not make every derived image or running instance identical. Application source, build arguments, fetched dependencies, target platform, runtime configuration, and persistent data can still change what gets built and how it behaves.

When it breaks

Two Dockerfiles can use the same FROM digest and produce different applications. Even the same Dockerfile can produce different output when a build argument changes or an unpinned download resolves new content. Declaring the base immutable therefore does not prove that the complete build is deterministic. The full set of inputs and the transformations applied to them determine that property.

Running the exact same application image with different environment values can select different endpoints, feature behavior, or storage. A writable bind mount can also replace files that were tested inside the image. Keeping configuration outside the image is often useful, but the configuration still needs versioning and review when it contributes to the release’s intended behavior.

An ordinary container also has a writable filesystem layer. An emergency edit can change one instance while leaving the image and other instances untouched. Replacement from the unchanged image then loses the edit. This is why immutable deployment practice combines artifact replacement with controlled runtime state; naming a base digest alone does not enforce that operational discipline.

A pinned base is one controlled input. Read-only roots constrain selected writes, while mounts, data and settings remain separate. Base updates need deliberate review, rebuild and tests. The Docker example remains blocked and was not rerun.

Check it quickly

Run the script using host Python and a Docker builder with the pinned base available. The earlier recorded run used Docker 29.3.1 and Linux arm64 Python 3.11.16; this review did not rerun Docker. Container networking and build RUN networking are disabled. Image resolution can still contact a registry; no build step downloads packages. The fixture removes its temporary containers and tags and never edits an existing service:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(image):return json.loads(docker('image','inspect',image).stdout)[0]

name='xbcatm-pinning-'+uuid.uuid4().hex[:10];tags=[name+':one',name+':two'];container=name+'-running'
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory)
 (root/'Dockerfile').write_text(f'FROM {BASE}\nARG MESSAGE\nRUN printf "%s" "$MESSAGE" > /built-message\nENV MODE=default\nCOPY app.py /app.py\nCMD ["python", "/app.py"]\n')
 (root/'app.py').write_text('from pathlib import Path\nimport os\nprint(Path("/built-message").read_text(),os.environ["MODE"])\n')
 layers=inspect(BASE)['RootFS']['Layers']
 for tag,message in zip(tags,['one','two']):
 docker('build','--network','none','--build-arg','MESSAGE='+message,'-t',tag,str(root))
 assert inspect(tag)['RootFS']['Layers'][:len(layers)]==layers
 assert inspect(tags[0])['Id']!=inspect(tags[1])['Id']
 assert docker('run','--rm','--network','none',tags[0]).stdout.strip()=='one default'
 assert docker('run','--rm','--network','none',tags[1]).stdout.strip()=='two default'
 assert docker('run','--rm','--network','none','-e','MODE=changed',tags[0]).stdout.strip()=='one changed'
 print('both builds use the same pinned base layers; different build arguments produce different artifacts')
 print('the identical application image behaves differently when runtime configuration changes')
 docker('run','-d','--name',container,'--network','none',tags[0],'python','-c','import time;time.sleep(60)')
 docker('exec',container,'python','-c','from pathlib import Path;Path("/built-message").write_text("edited")')
 assert docker('exec',container,'python','/app.py').stdout.strip()=='edited default'
 assert docker('run','--rm','--network','none',tags[0]).stdout.strip()=='one default'
 failed=docker('run','--rm','--network','none','--read-only',tags[0],'python','-c','from pathlib import Path;Path("/built-message").write_text("edited")',check=False)
 assert failed.returncode!=0 and 'Read-only file system' in failed.stderr
 print('a writable container can drift while its image stays unchanged; a read-only root rejects that particular write')
finally:
 docker('rm','-f',container,check=False);docker('image','rm',*tags,check=False)

Both builds use the same base reference and application source. The only selected build difference is MESSAGE, which the Dockerfile writes to /built-message. One image prints one default; the other prints two default. Their complete image identities differ, while inspection confirms that their ordered base filesystem-layer prefix is identical.

That pair establishes a bounded claim: the base remained fixed, but the derived artifact changed because another explicit input changed. It does not attempt to prove bit-for-bit reproducibility of repeated builds with every input held constant. Such a test would also need to control timestamps, dependency retrieval, toolchain behavior, and metadata emitted by the builder.

Next, the first application image runs with MODE overridden to changed. It prints one changed without rebuilding the image. This separates artifact identity from release configuration. A registry digest can identify application bytes, but it cannot by itself tell an incident investigator which environment values, credentials, mounted configuration, or external services the process actually used.

The script then starts a long-running container from the first image and edits its message file. Running the application inside that container prints edited default, while a fresh container from the unchanged image still prints one default. The repair exists only in the modified container’s writable state and will not automatically propagate into replacements.

A final run enables a read-only root filesystem and attempts the same file edit. It fails with a read-only filesystem error. This demonstrates a useful enforcement mechanism for that image-backed path, not a guarantee that every possible runtime state is frozen. Explicit writable mounts, external databases, environment settings, and the host kernel remain separate parts of the execution environment.

Record a release, then replace its instances

Keep a release record connecting the tested application digest, target platform, configuration revision, and required data compatibility. An OCI index can contain manifests for several platforms, so pinning the index still allows the runtime to select a platform-specific image. Record the selection when architecture affects reproducibility or diagnosis.

Build required changes into a new artifact, validate it, and replace instances through the deployment mechanism. Put durable data in storage with its own lifecycle and recovery plan. Avoid treating the container’s writable layer as the only durable copy, and avoid silently changing shared configuration underneath instances that supposedly belong to one fixed release.

Digest pinning also means new base fixes do not arrive automatically. Track available updates, review a changed digest, rebuild dependent applications, and test before promotion. Retaining a known input is valuable, but retaining an obsolete input forever is not the same as maintaining a secure release.

When investigating drift, compare artifact, configuration, mounts, and runtime changes separately. Each has a different owner and evidence trail. Replacing ad hoc instance edits with reviewed releases makes those differences explainable, while acknowledging that identical image bytes do not imply identical external state or identical behavior under every workload.

Sources

Docker build best practices, base-image digest pinning and deliberate update workflows, checked 2026-10-15.
Docker container run, runtime environment overrides and writable/read-only filesystems, checked 2026-10-15.
The Twelve-Factor App: build, release, run, separating a build artifact from release configuration and execution, checked 2026-10-15.
OCI image index 1.1.1, platform selection within an image index, checked 2026-10-15.
Docker container run reference, read-only roots and separately writable volumes, checked 2026-10-15.
Docker build networking, --network selects networking for RUN instructions, checked 2026-10-15.

Idle timeouts recycle edge connections

Mon, 26 Apr 2027 00:00:00 +0000

An idle timeout closes a connection after a defined period without qualifying activity. It is a resource policy, not proof that the client disappeared. Quiet connections can belong to healthy clients; the useful question is which activity resets which timer, and what the protocol allows after closure.

The historical Go HTTP/1.1 exercise reuses one connection for two /ping requests and a 350 ms /slow handler even with IdleTimeout 200 ms. After the response, it waits for StateClosed before a GET uses a new identity. A separate incomplete-header probe uses ReadHeaderTimeout 500 ms and ends with EOF without invoking the handler. Client 2 s bounds are separate. No Nginx instance was tested, and this review did not rerun the source. Source exercise illustration; not rerun in this review.

Idle is defined at a particular boundary

A TCP connection consumes file descriptors, kernel buffers, and application or proxy bookkeeping. It does not normally require a distinct routing-table entry per connection. Reclaiming unused connections can bound retained resources, but legitimate reuse also avoids setup. Choose the policy for the workload instead of assuming that every quiet connection is abandoned.

Go’s HTTP Server.IdleTimeout bounds waiting for the next request on a keep-alive connection. It does not time out a handler merely because that handler is working without sending bytes. Request-header reading, body transfer, response writing, and application execution need their own relevant controls. A limit with the word “idle” should not be treated as a complete request deadline.

Nginx also distinguishes boundaries: its client keepalive_timeout controls retained client connections, while proxy_read_timeout limits gaps between successive reads from an upstream response, not the whole response duration. These settings observe different activity on different sides of the proxy. Matching their numeric values does not make their meanings identical.

Zero is not portable either. Nginx’s zero keepalive_timeout disables client keep-alive connections. Go’s zero IdleTimeout uses ReadTimeout; when both provide no positive limit, no idle timeout is established. Check the implementation rather than interpreting zero as universally meaning “off” or “immediately close.”

When it breaks

A threshold shorter than legitimate quiet periods forces reconnections or interrupts the relevant flow. An excessively generous one can retain more sockets than the service can afford. Record connection counts, close reasons, reuse, and workload patterns; silence alone does not identify a broken client or establish the best threshold.

HTTP keep-alive means connection reuse, not a periodic application heartbeat. TCP keepalive probes are another mechanism and are not application request bytes. An application heartbeat only affects a timer if that particular endpoint counts it as activity. A proxy, upstream, and client need not observe the same events, especially with streaming or multiplexed protocols.

Closure can race a client starting another request. A stale pooled connection therefore needs a recovery path, but reconnecting does not make every request safe to replay. HTTP explicitly permits asynchronous connection closure; retry decisions still depend on request semantics and the uncertainty about prior effects. “The server closes slightly early” is not automatically harmless for a mutation.

Check it quickly

Save this as idle_probe.go and run go run idle_probe.go. It uses only Go’s standard library and was checked with Go 1.27.0 on macOS. The test server listens on a local loopback address. Requests are plain HTTP/1.1; no public edge, Nginx instance, certificate, or production configuration is involved.

package main

import (
 "context"
 "fmt"
 "io"
 "net"
 "net/http"
 "net/http/httptest"
 "runtime"
 "strconv"
 "sync/atomic"
 "time"
)

type connectionKey struct{}

func check(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func main() {
 const idle = 200 * time.Millisecond
 const headers = 500 * time.Millisecond
 const work = 350 * time.Millisecond
 var serial, handled atomic.Int64
 closed := make(chan struct{}, 8)
 server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 handled.Add(1)
 if r.URL.Path == "/slow" {
 time.Sleep(work)
 }
 w.Header().Set("X-Connection-ID", strconv.FormatInt(r.Context().Value(connectionKey{}).(int64), 10))
 w.Header().Set("Content-Type", "text/plain")
 fmt.Fprintln(w, "ok")
 }))
 server.Config.IdleTimeout = idle
 server.Config.ReadHeaderTimeout = headers
 server.Config.ConnContext = func(ctx context.Context, _ net.Conn) context.Context {
 return context.WithValue(ctx, connectionKey{}, serial.Add(1))
 }
 server.Config.ConnState = func(_ net.Conn, state http.ConnState) {
 if state == http.StateClosed {
 closed <- struct{}{}
 }
 }
 server.Start()
 defer server.Close()
 transport := &http.Transport{MaxIdleConns: 1, MaxIdleConnsPerHost: 1}
 defer transport.CloseIdleConnections()
 client := &http.Client{Transport: transport, Timeout: 2 * time.Second}

 get := func(path string) int64 {
 response, err := client.Get(server.URL + path)
 check(err == nil, fmt.Sprint(err))
 body, readErr := io.ReadAll(response.Body)
 closeErr := response.Body.Close()
 check(readErr == nil && closeErr == nil, "response body did not finish cleanly")
 check(response.Proto == "HTTP/1.1" && response.StatusCode == 200, "unexpected protocol or status")
 check(string(body) == "ok\n", "unexpected response body")
 id, err := strconv.ParseInt(response.Header.Get("X-Connection-ID"), 10, 64)
 check(err == nil, "missing connection identity")
 return id
 }

 first := get("/ping")
 check(get("/ping") == first, "the fully consumed responses should allow reuse")
 started := time.Now()
 check(get("/slow") == first, "the active request should keep its connection")
 check(time.Since(started) >= work, "the handler did not complete its work")
 fmt.Println("two requests reuse one HTTP/1.1 connection; active work outlasts IdleTimeout")

 select {
 case <-closed:
 case <-time.After(2 * time.Second):
 panic("the idle connection was not closed")
 }
 check(get("/ping") != first, "the request after idle closure needs a new connection")
 fmt.Println("server reports idle closure; a later GET succeeds on a new connection")

 before := handled.Load()
 connection, err := net.DialTimeout("tcp", server.Listener.Addr().String(), time.Second)
 check(err == nil, fmt.Sprint(err))
 defer connection.Close()
 check(connection.SetDeadline(time.Now().Add(2*time.Second)) == nil, "could not bound raw probe")
 _, err = io.WriteString(connection, "GET /ping HTTP/1.1\r\nHost: probe\r\n")
 check(err == nil, fmt.Sprint(err))
 reply, err := io.ReadAll(connection)
 check(err == nil && len(reply) == 0, "partial headers should end with EOF in this server")
 check(handled.Load() == before, "incomplete headers must not invoke the handler")
 fmt.Println("incomplete request headers close without reaching the handler; ReadHeaderTimeout is separate")
 fmt.Println(runtime.Version())
}

The client reads each response completely and closes its body, allowing reuse. Two successive requests report the same server-assigned connection identity. A third request runs its handler for 350 milliseconds, longer than the 200-millisecond idle allowance, and still succeeds on that connection. Active handler time is not the between-request idle interval.

After the response finishes, the client sends nothing. The server reports the connection closed, and a later GET succeeds with a different identity. The example observes that transition directly instead of sleeping an assumed amount and declaring that cleanup must have occurred.

A separate raw connection sends an incomplete request header and waits for closure. The tested server returns EOF without invoking the handler; its 500-millisecond ReadHeaderTimeout is distinct from idle reuse policy. Client-side deadlines bound the test’s own waits, and deferred cleanup closes the owned transport and server.

The short durations make the exercise observable, not suitable as universal production values. These checks cover Go’s HTTP/1.1 behavior, not HTTP/2 streams, WebSockets, kernel keepalive timing, or every intermediary. Validate each real hop’s definition of activity before coordinating its idle policy.

Sources

Go net/http, idle and header timeouts, connection states and response-body reuse, checked 2026-09-09.
Go net/http/httptest, a task-owned loopback test server and its lifecycle, checked 2026-09-09.
Nginx HTTP core module, client keep-alive timeout semantics, checked 2026-09-09.
Nginx proxy module, per-read upstream response timeout, checked 2026-09-09.
RFC 9112: HTTP/1.1, persistent connections and asynchronous closure, checked 2026-09-09.

Error budgets are a real budget

Thu, 25 Mar 2027 00:00:00 +0000

An error budget makes a reliability agreement usable during planning. Product and operations choose an objective, measure failures against it, and agree what changes when the allowance is exhausted. The number supports that agreement; it does not predict whether tomorrow’s deployment will cause an incident. Shipping a release creates risk, while observed bad events consume the measured budget.

Request counts and policy cases come from the article’s synthetic fixture. The bars share a request-count scale. This drawing does not execute the code or grant deployment approval; the positive balance only makes a closed-incident case eligible for review.

Give the account a measurement rule

For a request-based objective, allowed bad events equal eligible requests multiplied by one minus the target. A 99.9% objective over three million requests permits three thousand bad requests. Specify which requests count, which outcomes are bad, the measurement location, and the window before attaching a release rule. A traffic-weighted allowance is not automatically a fixed number of downtime minutes.

A rolling thirty-day account drops old observations as new ones arrive. Its denominator changes when traffic changes, so the allowance can change even when the target stays fixed. A calendar-month account resets at a calendar boundary instead. Those approaches answer different questions; switching between them during an incident makes an apparently precise balance difficult to audit. Keep the definition versioned with the policy and show the window beside the balance.

Decide the response before the argument

An example policy might hold nonessential feature releases when the budget is overspent, prioritize reliability work, and allow urgent security or incident fixes through review. Name who owns the decision, what evidence permits an exception, and when ordinary release review can resume. Google’s published example is a policy for its example service, not a universal requirement to copy every threshold or escalation rule.

Remaining budget alone does not establish release readiness. An open incident, missing monitoring, a failed regression, or unsafe rollback can still prevent a change. Conversely, a necessary repair should not be blocked merely because the incident it repairs exhausted the budget. Record these decisions explicitly. The useful result is a shared response to customer impact, not an automated mechanism for blaming the team whose dashboard turned red.

When it breaks

A team can appear to recover simply because yesterday’s large incident leaves the rolling window. The arithmetic is correct, but it says nothing about whether the defect was repaired. Changing exclusions, silently replacing unavailable samples with successes, or averaging differently sized service percentages can also create a misleading surplus. Display measurement coverage and investigate definition changes before interpreting an improved balance.

Short-window burn rate supplies another view: the observed bad-event fraction divided by the allowed fraction. A high burn can warn before the longer account is exhausted. Estimating time to exhaustion from that rate assumes comparable traffic and continued behavior; it is not a promised deadline. Pair actionable alerts with the long-window account and account for sparse traffic, rather than paging on every isolated failure.

Check it quickly

Save the following as budget-policy.py and run it with Python 3. It uses synthetic daily request counts to verify rolling-window accounting and a deliberately small example policy. It does not change a deployment system or estimate the risk of a real release.

from fractions import Fraction

target = Fraction(999, 1000)
# Every synthetic daily bin has 100,000 eligible requests.
days = [(100000, 1300)] + [(100000, 70)] * 29

def account(window):
 eligible = sum(total for total, _ in window)
 bad = sum(errors for _, errors in window)
 allowed = eligible * (1 - target)
 return bad, allowed, allowed - bad

def feature_policy(remaining, incident_open):
 return "hold" if remaining < 0 or incident_open else "eligible for review"

before = account(days)
after = account(days[1:] + [(100000, 50)])
assert before == (3330, 3000, -330)
assert after == (2080, 3000, 920)
assert feature_policy(before[2], False) == "hold"
assert feature_policy(after[2], True) == "hold"
assert feature_policy(after[2], False) == "eligible for review"
for label, (bad, allowed, remaining) in [("before", before), ("after", after)]:
 print(f"{label}: bad={bad}, allowed={float(allowed):.0f}, remaining={float(remaining):.0f}, spent={float(bad / allowed):.2%}")
print("after rotation, incident still open:", feature_policy(after[2], True))
print("after rotation, incident closed:", feature_policy(after[2], False))

The first account has 3,330 bad requests against an allowance of 3,000: 111% spent and a balance of minus 330. Removing the oldest day and adding the new day leaves 2,080 bad requests, 69.33% spent, and 920 remaining. Both windows contain the same three million eligible requests in this fixture; real traffic need not stay constant.

The policy still prints hold while an incident is open, despite the positive new balance. Only the final case becomes eligible for review, which intentionally stops short of granting deployment approval. This separates a verified calculation from the operational judgment that uses it. Review the definition, coverage, incident evidence, and release checks together before acting on the account.

Sources

Google SRE Workbook: Example Error Budget Policy, release restrictions, exceptions, and ownership, checked 2026-09-13.
Google SRE: Service Level Objectives, measurement boundaries and reliability targets, checked 2026-09-13.
Google SRE Workbook: Alerting on SLOs, burn rate and alerting windows, checked 2026-09-13.

Incident response needs a timeline

Mon, 01 Mar 2027 00:00:00 +0000

During an incident, memory is unreliable, and clocks can disagree too. A timeline — who was paged, what was observed, what was tried, when the change landed — is what makes a response recordable and later reviewable.

Times and actors are transcribed from the inert source fixture. The shared date and UTC offset are display shorthand; underlying entries require full timestamps. Vertical spacing is schematic, not elapsed-time scale. A planned effect is not shown as a confirmed fix; no validator or incident system was run.

Write as it happens

Memory is the least reliable instrument in an incident, so record while the alert is fresh. Capture the timestamp of the first alert, the initial symptoms, each action tried and its observable result, and the moment the fix actually took effect. That running record is what keeps a team from repeating a failed fix during the response, and what later makes an honest postmortem possible instead of a plausible story. Make the timeline a live document from the minute the page fires: note the tool that alerted, the metric, the suspected and the confirmed cause, and record ‘same as before, did not help’ rather than silently re-trying.

Separate communication from execution

As an incident grows, debugging and keeping everyone informed compete for attention. Split operational and communication responsibilities: one person acts, another maintains the timeline and shares status, scope, and open questions. That second seat prevents the classic failure where everyone is heads-down troubleshooting and nobody is recording what was tried. A shared timeline also shortens onboarding: a teammate who arrives late reads the record and catches up, instead of interrupting the lead with questions or trusting a scattershot retelling. Update it as events land, with links to supporting logs and decisions.

Learning requires the record

A postmortem written from memory is a plausible story; one written from a timeline is a reconstruction of facts you can cross-check. Because the timeline pins actions to timestamps and outcomes, the review can focus on systemic gaps rather than individual blame: which alert did not exist, which runbook was wrong, which decision had no owner. That is the real payoff of recording from the first minute. Start at the first alert when possible, and label reconstructed entries if recording begins later. A partial, sourced record is still useful; do not fill gaps with assumed certainty.

When it breaks

A live timeline fails while the incident is still open, not after it. The first failure is a clock without a zone: one responder writes 10:02 from a laptop on local time while the alert pipeline logs in UTC, so identical-looking marks can represent different instants. Record date, offset, clock source, and any known skew. Even synchronized timestamps alone cannot establish causation.

The second failure is the next-morning rewrite. The team reconstructs the record from memory, keeps the fix, drops the two retries that did not help, and reorders events into a tidy story. That version hides exactly what the next responder needs: what was already tried and what observably changed.

The third failure is concurrent work without coordination. Two responders can both claim the rollback slot even if the editor preserves every line. Assign change owners and stable entry identifiers; preserve the original recording order and mark late additions. A chronological view should not silently rewrite the underlying evidence.

The fourth failure is the chat burial. The failover decision is approved deep in a thread with two hundred side messages, but nobody copies the decision, owner, and expected effect into the timeline. An hour later nobody can answer who decided what. Unlike the companion postmortem piece about blame-free learning after the fact, these are recording failures during the response itself.

Check it quickly

Save the following as timeline_check.py and run python3 timeline_check.py. It uses only the Python standard library, builds one good and one bad in-memory timeline, writes no files, and checks the three fields every live entry needs: t, actor, and action.

"""Live incident timeline validator: each event needs t/actor/action, ordered by time."""
from datetime import datetime


def parse_time(raw):
 text = str(raw or "").strip().replace("Z", "+00:00")
 # Require a full timestamp with date and timezone, not a bare wall-clock time.
 if "T" not in text or not text:
 raise ValueError(f"bad timestamp {raw!r}: use ISO-8601 with date and timezone")
 try:
 stamp = datetime.fromisoformat(text)
 except ValueError:
 raise ValueError(f"bad timestamp {raw!r}: use ISO-8601 with date and timezone") from None
 if stamp.tzinfo is None:
 raise ValueError(f"bad timestamp {raw!r}: missing timezone")
 return stamp


def check_timeline(events):
 errors = []
 seen = []
 for i, ev in enumerate(events):
 for field in ("t", "actor", "action"):
 if not isinstance(ev.get(field), str) or not ev[field].strip():
 errors.append(f"event {i} missing {field} or not a nonempty string")
 try:
 seen.append(parse_time(ev.get("t", "")))
 except ValueError as exc:
 errors.append(f"event {i} {exc}")
 seen.append(None)
 stamps = [s for s in seen if s is not None]
 # Only judge ordering over entries that parsed, so one bad clock
 # does not hide a second sequencing mistake.
 ordered = sorted(stamps)
 if stamps != ordered:
 errors.append("events out of order: review clock sources and late entries")
 return errors


GOOD = [
 {"t": "2027-03-01T10:01:00+00:00", "actor": "pager", "action": "alert fired: error-rate burn"},
 {"t": "2027-03-01T10:04:00+00:00", "actor": "ana", "action": "took timeline keeper seat, opened shared doc"},
 {"t": "2027-03-01T10:09:00+00:00", "actor": "lee", "action": "rolled back deploy 1a2b3c, error-rate still 8%"},
 {"t": "2027-03-01T10:14:00+00:00", "actor": "ana", "action": "decision: fail over to region-b, owner lee, effect visible by 10:20"},
]

BAD = [
 {"t": "10:09", "actor": "lee", "action": "rollback done?"},
 {"t": "2027-03-01T10:04:00+00:00", "actor": None, "action": "someone joined late, backfilled from memory"},
 {"t": "2027-03-01T10:02:00+00:00", "actor": "ana", "action": []},
]


def main():
 good_errors = check_timeline(GOOD)
 bad_errors = check_timeline(BAD)
 assert good_errors == [], f"good sample should pass, got: {good_errors}"
 assert any("missing actor" in e for e in bad_errors), f"bad sample should flag actor, got: {bad_errors}"
 assert any("missing action" in e for e in bad_errors), f"bad sample should flag action, got: {bad_errors}"
 assert any("bad timestamp" in e or "timezone" in e for e in bad_errors), f"bad sample should flag clock, got: {bad_errors}"
 assert any("out of order" in e for e in bad_errors), f"bad sample should flag order, got: {bad_errors}"
 print("PASS: good timeline has t/actor/action on every event in time order")
 print(f"FAIL as expected: bad timeline -> {'; '.join(bad_errors)}")
 print("summary: 1 passed check, 1 failed check (expected 1 and 1)")


if __name__ == "__main__":
 main()

The good sample passes because every event carries a full ISO-8601 timestamp with timezone, a named actor, and an observable action in time order. The bad sample fails together on a bare wall-clock time, a missing actor, a missing action, and backward stamps. Expect one PASS line, one FAIL line naming each defect, and a one-passed-one-failed summary. The revised script was run with Python 3.12.14 on Linux. Equal timestamps are allowed; this checks nondecreasing recorded times, not unique sequencing or truthful observations. Python accepts several ISO-8601 forms here; this is not a strict RFC 3339 validator. It checks recording discipline during the response, not the post-incident review in the companion postmortem article.

Sources

Google SRE: managing incidents, incident roles, living documents, and explicit handoffs, checked 2026-10-15.
Google SRE: postmortem culture, evidence-based and blameless learning, checked 2026-10-15.
Python datetime, aware timestamp parsing and comparisons, checked 2026-10-15.
RFC 3339 timestamps, offsets and limitations of textual ordering, checked 2026-10-15.

Failover is a practiced switch

Thu, 25 Feb 2027 00:00:00 +0000

A standby helps only if the promotion path works and the old writer cannot corrupt the new history. Failover is a decision, a procedure, and a rehearsal. A healthy-looking replica is not proof that clients can reach it or that it contains every acknowledged write. Define the required recovery time and acceptable data loss before deciding when to switch.

Design the decision in advance

Choose who can authorize promotion, what evidence demonstrates failure, and whether automation or an operator acts. A lost heartbeat can mean a network partition rather than a dead primary. Failure detection alone cannot authorize two independent writers. The system needs an enforced ownership mechanism, such as verified node isolation or resource-side fencing, with guarantees appropriate to its replication protocol.

For a single-writer service, the safety goal is at most one accepted writer, not uninterrupted availability of exactly one. Fencing may deliberately leave zero writers during a transition. The runbook should identify the candidate, its replication position, the acceptable loss boundary, how clients reconnect, and who stops the procedure if prerequisites fail. Automatic failover can shorten an outage but cannot manufacture missing replicated data.

Rehearse the actual path

Practice at a cadence matched to risk and after relevant changes. A drill validates the paths it actually exercises: a database promotion that bypasses DNS says little about cached addresses, and a test client using a different identity says little about production permissions. Include realistic client reconnection and checks of acknowledged writes where safe. Record scope, timings, lag, refusals, and recovery evidence, rather than treating the date of the last successful drill as a complete readiness score.

Plan the safe stopping point and recovery procedure before starting. Once the promoted node has accepted new writes, switching back is not necessarily a reversible configuration edit. The former primary may need reconciliation or rebuilding as a replica. Backups provide another recovery path, but their existence alone does not prove a usable restore.

When it breaks

Split brain appears when the old primary remains writable after promotion. Merely recording “fenced” in a controller is insufficient; the storage or isolation mechanism must enforce it. A stale candidate can lose acknowledged writes even though promotion succeeds. Express lag thresholds in meaningful units and tie them to the data-loss objective; a nonzero threshold is not a zero-loss promise.

Flapping health checks can provoke repeated transitions. Use deliberate failure thresholds and recovery criteria, and avoid automatic failback before resynchronization. DNS changes affect later lookups: existing connections and answers cached under the previous TTL may persist. Lowering the TTL during an incident does not retroactively expire those cached answers. Test the actual client’s retry and reconnection behavior as well as the authoritative DNS response.

Fence before promoting a writer. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

With Python 3.9 or newer, save this as failover_drill.py and run python3 failover_drill.py in a scratch directory. Run without -O, -OO, or a nonzero PYTHONOPTIMIZE setting so the assertions execute. It changes only in-memory objects. Health flags and lag values are synthetic; fence() is a set update, not a real fence command. The model keeps one designated primary while allowing zero writable nodes during isolation. It explicitly reactivates an eligible candidate, including during failback, instead of accidentally leaving the new primary in the fenced set.

"""Failover drill: single-primary switch with health and lag guards (stdlib only)."""
from dataclasses import dataclass

MAX_LAG = 128 # Synthetic units, not seconds or a zero-data-loss guarantee.

class FailoverRefused(Exception):
 pass

@dataclass
class Node:
 name: str
 healthy: bool = True
 lag: int = 0
 is_primary: bool = False

class Cluster:
 def __init__(self):
 self.nodes = {"a": Node("a", is_primary=True), "b": Node("b")}
 self.fenced: set[str] = set()

 @property
 def primary(self) -> str:
 primaries = [n for n in self.nodes.values() if n.is_primary]
 assert len(primaries) == 1, f"must keep exactly one primary, saw {[p.name for p in primaries]}"
 return primaries[0].name

 def can_write(self, name: str) -> bool:
 node = self.nodes[name]
 return node.is_primary and node.healthy and name not in self.fenced

 def health_check(self, name: str) -> bool:
 node = self.nodes[name]
 return node.healthy and node.lag <= MAX_LAG

 def fence(self, name: str) -> None:
 self.fenced.add(name)
 assert not self.can_write(name)

 def failover(self, target: str, *, fence_old: bool) -> str:
 old = self.primary
 if target == old:
 raise FailoverRefused("target is already primary")
 if target not in self.nodes:
 raise FailoverRefused(f"unknown target {target}")
 candidate = self.nodes[target]
 if not fence_old or old not in self.fenced:
 raise FailoverRefused(f"refuse: old primary {old} is not fenced; fence first to avoid split brain")
 if not candidate.healthy:
 raise FailoverRefused(f"refuse: standby {target} is unhealthy")
 if candidate.lag > MAX_LAG:
 raise FailoverRefused(f"refuse: standby {target} lags by {candidate.lag} (> {MAX_LAG})")
 self.nodes[old].is_primary = False
 # Model explicit reactivation after the old writer is fenced.
 self.fenced.discard(target)
 candidate.is_primary = True
 assert self.primary == target and self.can_write(target)
 assert sum(self.can_write(name) for name in self.nodes) <= 1
 return target

def expect_refused(fn, label: str) -> None:
 try:
 fn()
 except FailoverRefused as exc:
 print(f"refused as expected ({label}): {exc}")
 return
 raise AssertionError(f"should have refused: {label}")

def main() -> None:
 # 1. Clean drill: healthy standby, caught-up replica, old primary fenced.
 cluster = Cluster()
 assert cluster.primary == "a"
 assert cluster.health_check("b")
 cluster.fence("a")
 assert cluster.failover("b", fence_old=True) == "b"
 print("clean switch: a -> b")

 # 2. Refuse lag beyond this policy; even accepted positive lag may lose writes.
 cluster = Cluster()
 cluster.nodes["b"].lag = 900
 assert not cluster.health_check("b")
 cluster.fence("a")
 expect_refused(lambda: cluster.failover("b", fence_old=True), "stale replica")
 assert cluster.primary == "a"

 # 3. Unhealthy standby must not promote.
 cluster.nodes["b"].lag = 0
 cluster.nodes["b"].healthy = False
 expect_refused(lambda: cluster.failover("b", fence_old=True), "unhealthy standby")
 assert cluster.primary == "a"

 # 4. Unfenced old primary must block promotion (split-brain guard).
 cluster.nodes["b"].healthy = True
 unfenced = Cluster() # nobody fenced yet
 expect_refused(lambda: unfenced.failover("b", fence_old=False), "split brain")
 expect_refused(lambda: unfenced.failover("b", fence_old=True), "split brain without fence record")
 assert unfenced.primary == "a"

 # 5. Failback: resynchronize the candidate, then fence the CURRENT writer.
 cluster = Cluster()
 cluster.fence("a")
 cluster.failover("b", fence_old=True)
 cluster.nodes["a"].healthy = True
 cluster.nodes["a"].lag = 0
 cluster.fence("b")
 assert cluster.failover("a", fence_old=True) == "a"
 print("failback: b -> a")

 print("PASS: at most one writable node; 4 refusals + clean switch + failback passed")

if __name__ == "__main__":
 main()

The run prints a clean switch, four refusals in three categories, and failback, ending with PASS: at most one writable node; 4 refusals + clean switch + failback passed. Refused promotion leaves the designated primary unchanged, but a previously fenced primary remains unavailable. The clean-switch and failback fixtures use zero lag; the policy also permits positive lag up to 128 synthetic units, which carries a possible loss trade-off.

The earlier fixture recorded Python 3.14.4 on macOS. The revised fixture passed with Python 3.12.14 on Linux. These assertions test sequential decision logic, including the formerly fenced target’s writable state. They do not validate concurrent promotion, real replication, permissions, DNS propagation, storage isolation, or production recovery. Those require a separately authorized staging drill with the actual mechanisms and representative clients.

Sources

PostgreSQL 18: failover, old-primary isolation, rebuilding redundancy, and written switch procedures, checked 2026-10-15.
AWS: configuring DNS failover, health checks and routing configuration, checked 2026-10-15.
Google SRE: testing for reliability, production-test scope, automation barriers, and disaster recovery testing, checked 2026-10-15.
AWS: record TTL, resolver caching and planning TTL changes before a switch, checked 2026-10-15.

Capacity planning is a number, not a guess

Sun, 21 Feb 2027 00:00:00 +0000

Capacity planning turns a demand forecast into a provisioned operating envelope. The useful question is not how many requests a machine can accept before crashing, but what workload the service can sustain while meeting its latency and correctness requirements. That answer depends on request cost, dependencies, and failure assumptions. A spreadsheet becomes useful when each of those assumptions has a source and an owner.

Measure an acceptable operating point

Load tests should use a representative request mix, data size, cache state, concurrency, and software configuration. Increase demand while recording latency, errors, CPU, memory, queue depth, storage, and connection usage. The highest tested point meeting the chosen requirements is evidence for that workload, not proof that every intermediate point or future release behaves identically. A throughput result without outcome quality can reward a server that rejects work quickly.

Track different bottlenecks separately. More application replicas may increase CPU capacity while leaving a shared database connection limit unchanged. Expensive requests can also change the relationship between requests per second and resource consumption. A request-rate model is reasonable only while that cost distribution is comparable; monitor the underlying resources and revise the model when the workload changes. Do not project a quiet average into a peak-capacity promise.

Forecast demand and remove capacity deliberately

Start with an observed peak, then state the growth or seasonal scenario being planned. Add provisioning lead time and uncertainty explicitly. The operating reserve should cover a named reason, such as load imbalance, forecast error, or time needed to add instances, rather than being a mysterious percentage inherited from an old spreadsheet. Revisit both the measurement and the demand assumption after meaningful changes.

Next remove the capacity that the reliability design says may fail. A service spread across three zones needs a different instance count if it must serve the same peak with one zone unavailable. Account for which dependencies survive, how traffic moves, and whether enough instances can start before queues overflow. Balanced placement does not by itself guarantee balanced demand, and autoscaling cannot supply capacity instantaneously.

When it breaks

A common estimate divides forecast traffic by a single-machine maximum, rounds up, and stops. It may satisfy a healthy-cluster calculation while failing immediately after a zone loss. Counting reserved capacity twice is another error: distinguish a tested throughput point from the lower rate chosen for normal planning. State which boundary a reported headroom number uses so that spare capacity is not confused with physical maximum throughput.

Linear scaling can fail at a shared lock, database, network link, or external quota. Cold caches and synchronized retries can make a failover workload more expensive than the original benchmark. Validate the failure scenario under realistic conditions and record where the model stops applying. If a forecast cannot be served safely, admission control or a tested degraded mode may be necessary while provisioning catches up; adding an optimistic cell to the plan will not protect users.

At the example planning rate, sixteen surviving nodes provide 2,240 requests/s for a 2,000 forecast. The 240 spare is below the planning limit, under stated scaling and dependency assumptions.

Check it quickly

Save this as capacity.py and run Python 3. The inputs are synthetic load-test results for a fixed workload. This exercise verifies the sizing arithmetic and failure assumption, not the performance of your hardware. A planning rate of 70% of the accepted test point is an explicit example choice, not a universal utilization recommendation.

from math import ceil

# Synthetic load-test results for one fixed request mix and machine size.
samples = [
 {"rps": 150, "p99_seconds": 0.11, "error_ratio": 0.0001},
 {"rps": 200, "p99_seconds": 0.22, "error_ratio": 0.0003},
 {"rps": 250, "p99_seconds": 0.50, "error_ratio": 0.0020},
]
acceptable = [s["rps"] for s in samples
 if s["p99_seconds"] <= 0.30 and s["error_ratio"] <= 0.001]
tested_rps = max(acceptable)
forecast_rps = ceil(1600 * 1.25)
zones = 3
surviving_zones = zones - 1
planning_rps = tested_rps * 0.70

naive_per_zone = ceil(ceil(forecast_rps / tested_rps) / zones)
planned_per_zone = ceil(forecast_rps / (planning_rps * surviving_zones))
naive_survivor_rps = naive_per_zone * surviving_zones * tested_rps
planned_survivor_rps = planned_per_zone * surviving_zones * planning_rps

assert tested_rps == 200 and forecast_rps == 2000
assert naive_per_zone * zones == 12 and naive_survivor_rps == 1600
assert planned_per_zone * zones == 24 and planned_survivor_rps == 2240
assert naive_survivor_rps < forecast_rps <= planned_survivor_rps
print(f"tested={tested_rps} rps/node; forecast={forecast_rps} rps")
print(f"naive={naive_per_zone * zones} nodes; zone-loss capacity={naive_survivor_rps} rps")
print(f"planned={planned_per_zone * zones} nodes; zone-loss planning capacity={planned_survivor_rps:.0f} rps")
print(f"zone-loss spare={planned_survivor_rps - forecast_rps:.0f} rps")

The accepted listed point is 200 requests per second per node; the 250-request sample fails the quality requirements. Forecast demand becomes 2,000 requests per second. The naive balanced allocation uses twelve nodes but leaves only eight after a zone loss, providing 1,600 requests per second even at the accepted test rate.

The reserve-aware plan allocates eight nodes per zone, twenty-four total. Sixteen survivors at the lower planning rate provide 2,240 requests per second, leaving 240 below that operating limit. This result depends on the fixture’s fixed request mix, independent node scaling, and surviving dependencies. Replace those assumptions with measured evidence before turning the number into a purchase or a reliability commitment.

Sources

Google SRE: Handling Overload, resource capacity, request cost, and overload behavior, checked 2026-09-13.
Google SRE Workbook: Managing Load, capacity-aware routing and failure scenarios, checked 2026-09-13.
Google SRE: Monitoring Distributed Systems, saturation and limits of static load-test estimates, checked 2026-09-13.

Feature flags decouple release from exposure

Tue, 09 Feb 2027 00:00:00 +0000

Deploying code and exposing it are different acts. A feature flag can select a runtime path after the code is deployed. This supports gradual exposure and fast disablement, provided the old path remains compatible and the configuration reaches the processes that evaluate it. The flag controls a decision; it does not reverse everything the deployment or the new behavior has already done.

A flag chooses later work; it does not reverse completed work. The cohort count belongs to a fixed local fixture and does not establish request share or distributed delivery.

Ship behind a defined gate

A release flag commonly defaults to the established path while a new implementation is tested. Deploy-time behavior still changes: imports, startup hooks, schema migrations, dependencies, and resource use may run outside the guarded branch. An off flag therefore cannot promise zero user-visible effects. Test the deployed artifact with the flag off as well as on, including startup and compatibility with existing data.

Turning off the flag can stop later requests from entering the new path. It does not undo completed writes, messages sent, migrations, or work already in flight. Keep data formats compatible with both branches during the rollback window and define any repair required after partial execution. A kill switch is useful only when its propagation delay and actual stopping boundary match the incident response plan.

Choose defaults by meaning

For an unfinished release feature, an unknown configuration may safely select the old path. That is a decision about this flag’s semantics, not a universal rule that false is always safe. Disabling a protective check or reversing a flag named disable-validation can have the opposite effect. Specify fallback behavior, configuration freshness, and what happens when the service cannot evaluate the flag.

Evaluate a release decision once at the appropriate operation boundary and reuse it within that operation. Multiple services or separate requests need an explicit way to carry compatible decisions; local evaluation alone does not create global consistency. Keep authorization enforced independently. A client-side display flag must not become permission to perform a protected server-side action.

Ramp exposure with stable identities

Hashing a stable user identifier with a stable flag key gives repeatable cohorts. Changing the identifier, salt, or bucketing algorithm can reshuffle users even when the percentage stays unchanged. A percentage of users is also not necessarily the same percentage of requests: a small cohort can generate disproportionate traffic. Record the unit of assignment before interpreting rollout metrics.

A kill switch should override the cohort selection and have a tested delivery path. SDK caches and configuration outages can leave some instances on older values. Observe errors and latency by actual evaluated variant, rather than treating a control-panel change as proof that every request has switched. Choose a rollout pace from the service’s risk and observation delay, not an arbitrary universal schedule.

When it breaks

Temporary release flags can become permanent branches when nobody owns their removal. Give each one an owner and a retirement condition, and remove obsolete paths after the rollback window closes. Operational switches and entitlement controls may deliberately remain long-lived; they need ongoing tests and ownership rather than an unconditional short expiry.

Contradictory flags can create combinations nobody tested. Centralize the decision rule where practical, define precedence, and exercise both the fallback and enabled paths. Logs should record enough nonsecret evaluation context to diagnose disagreement, while avoiding unbounded retention of real user identifiers. The example below uses synthetic names only.

Check it quickly

Save the block as flag_check.py and run it with Python 3.10 or newer. The original fixture reports Python 3.14.4; the retained 2026-10-15 review reran its decision logic with Python 3.12.14. It models one percentage release flag, a kill switch, stable user bucketing, and an in-memory audit list. The percentage is assumed to be a validated value from zero through one hundred.

import hashlib


class Flag:
 """One percentage flag with kill switch, default-closed, and audit log."""

 def __init__(self, key, percent):
 self.key = key
 self.percent = percent
 self.killed = False
 self.audit = []

 def _bucket(self, user):
 digest = hashlib.sha256(f"{self.key}:{user}".encode()).hexdigest()
 return int(digest, 16) % 100

 def evaluate(self, user):
 if self.killed:
 decision, reason = False, "killed"
 elif not user:
 decision, reason = False, "default-closed"
 else:
 hit = self._bucket(user) < self.percent
 decision, reason = hit, ("in-bucket" if hit else "out-of-bucket")
 self.audit.append({"user": user, "decision": decision, "reason": reason})
 return decision


flag = Flag("new-checkout", 20)
users = [f"user-{i}" for i in range(200)]

# 1. Default closed: a missing user never gets the new path.
assert flag.evaluate("") is False
assert flag.evaluate(None) is False

# 2. Stickiness: the same user gets the same answer twice.
first = flag.evaluate("user-1024")
second = flag.evaluate("user-1024")
assert first == second

# 3. Kill switch overrides every bucket.
flag.killed = True
assert flag.evaluate("user-1024") is False
assert flag.evaluate("user-7") is False
flag.killed = False

# 4. The dial moves exposure for a fixed fixture set.
open_20 = sum(flag.evaluate(u) for u in users)
flag.percent = 100
open_100 = sum(flag.evaluate(u) for u in users)
flag.percent = 0
open_0 = sum(flag.evaluate(u) for u in users)
flag.percent = 20
assert open_0 == 0
assert 0 < open_20 < len(users)
assert open_100 == len(users)

# 5. A different bucketing key disagrees somewhere: the inconsistency to avoid.
session_flag = Flag("new-checkout-session", 20)
mismatch = [u for u in users if flag.evaluate(u) != session_flag.evaluate(u)]
assert mismatch, "expected at least one user where two flag keys disagree"

# 6. Audit log holds one entry per evaluation with its reason.
assert len(flag.audit) == 2 + 2 + 2 + 200 * 3 + 200
assert len(session_flag.audit) == 200
assert all(set(e) == {"user", "decision", "reason"} for e in flag.audit)
print(f"default-closed empty user: off; sticky user-1024: {'on' if first else 'off'} twice")
print(f"dial 0/20/100 over 200 users: {open_0}/{open_20}/{open_100}")
print(f"killed flag forces off; different flag keys disagree on {len(mismatch)} users, e.g. {mismatch[0]}")
print(f"audit entries: flag={len(flag.audit)}, session_flag={len(session_flag.audit)}")
print("PASS: default-closed, sticky bucketing, kill switch, dial, and audit log behave as asserted")

At twenty percent, missing users stay off, the kill switch overrides every named user, and repeated evaluation is stable. The fixed two-hundred-user cohort produces zero, thirty-five, and two hundred enabled users at zero, twenty, and one hundred percent. Thirty-five is a deterministic fixture result, not a guarantee of exactly twenty percent in every finite population.

The second evaluator changes the flag key while retaining the same user IDs. Its different decisions demonstrate salt/key drift, not an actual session-ID experiment. The audit counts verify one entry per evaluation. No distributed propagation, request-wide reuse, database rollback, or authorization is tested by this local model; those remain separate integration responsibilities.

Sources

Feature Toggles (Pete Hodgson), the toggle taxonomy, checked 2026-10-15.
LaunchDarkly: what are feature flags, progressive exposure, checked 2026-10-15.
12 Factor: build/release/run separation, where config and flags sit, checked 2026-10-15.

A purpose-built health check is worth keeping

Thu, 31 Dec 2026 00:00:00 +0000

A health endpoint is a contract with a consumer that will take action. Returning 200 because a handler ran does not prove that an instance can serve useful traffic. Before writing the endpoint, identify the decision it supports: allow initialization, admit requests, or restart a process that cannot recover. Different questions need different evidence.

Separate process recovery from traffic admission

Liveness should represent a local condition for which restarting is a useful remedy. A failed shared database is usually not repaired by restarting every caller. Readiness answers whether this instance should receive its intended traffic now. Include dependencies essential to that traffic, but do not make an optional recommendation service remove an otherwise useful checkout instance from rotation.

Startup protects slow initialization from premature recovery actions. With a Kubernetes startup probe configured, liveness and readiness probes wait until startup succeeds. Readiness failure affects traffic admission; repeated liveness or startup failures can cause restarts according to the probe configuration and restart policy. A single 503 response does not itself execute either action.

Document the status contract for the actual consumer. Kubernetes HTTP probes accept statuses from 200 through 399, so a login redirect can accidentally appear healthy. This example deliberately returns only 200 or 503 for its three GET probe routes. It does not depend on parsing a reassuring response body or following a redirect.

Bound the work the probe introduces

Use a cheap, read-only operation that represents the essential dependency. Acquiring a connection or making a small dependency request is evidence about that path, not proof of every business transaction. Specify a deadline shorter than the caller’s timeout, pass cancellation into downstream work, release responses, and cancel timers. Returning early while abandoned operations continue accumulating is not a bounded probe.

Estimate the load from instances, independent observers, and polling frequency. Avoid a deep dependency fan-out on every health request. If an existing background checker supplies a result, its age and failure meaning must be explicit; an indefinitely cached green value is not current health. Probe caching should not let an intermediary keep a failed instance looking ready.

When it breaks

An always-green endpoint remains successful during a broken database connection, exhausted worker pool, or incomplete initialization. The opposite mistake makes a shared-dependency outage trigger fleet-wide restart loops. Readiness checks can themselves consume the scarce connections needed for recovery, while overly aggressive thresholds remove too much capacity during a short pause.

Test startup, a negative dependency response, a stalled dependency, deliberate draining, and recovery. Liveness must remain independent of the shared dependency in this design. Draining should stop new admission before in-flight work is completed; this endpoint alone does not wait for load-balancer propagation or drain existing requests. Keep public responses small and avoid exposing dependency addresses, credentials, or stack traces. Detailed causes belong in protected operational telemetry.

The Go fixture gates readiness on startup, drain state and a bounded required-dependency check; its live handler remains responsive. Probe results alone do not prove platform actions.

Check it quickly

Save this program as health_probe.go and run go run health_probe.go. Two loopback servers model an application and its one required dependency. The dependency’s healthy contract is 204. A 100-millisecond child context bounds the dependency request; the outer test client allows two seconds. Flags model startup and draining without modifying any real service.

package main

import (
 "context"
 "fmt"
 "io"
 "net/http"
 "net/http/httptest"
 "runtime"
 "sync/atomic"
 "time"
)

func must(err error) {
 if err != nil {
 panic(err)
 }
}

func check(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func main() {
 var mode, dependencyCalls atomic.Int64
 var started, draining atomic.Bool
 canceled := make(chan struct{}, 1)
 dependency := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 dependencyCalls.Add(1)
 switch mode.Load() {
 case 1:
 w.WriteHeader(503)
 case 2:
 <-r.Context().Done()
 canceled <- struct{}{}
 default:
 w.WriteHeader(204)
 }
 }))
 defer dependency.Close()
 transport := &http.Transport{}
 defer transport.CloseIdleConnections()
 client := &http.Client{Transport: transport, Timeout: 2 * time.Second,
 CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
 }
 server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 w.Header().Set("Content-Type", "text/plain; charset=utf-8")
 w.Header().Set("Cache-Control", "no-store")
 if r.Method != "GET" {
 w.Header().Set("Allow", "GET")
 w.WriteHeader(405)
 return
 }
 ready := false
 switch r.URL.Path {
 case "/livez":
 ready = true // No shared-dependency restart policy.
 case "/startupz":
 ready = started.Load()
 case "/readyz":
 if started.Load() && !draining.Load() {
 ctx, cancel := context.WithTimeout(r.Context(), 100*time.Millisecond)
 defer cancel()
 request, err := http.NewRequestWithContext(ctx, "GET", dependency.URL, nil)
 must(err)
 response, err := client.Do(request)
 if err == nil {
 ready = response.StatusCode == 204
 must(response.Body.Close())
 }
 }
 default:
 w.WriteHeader(404)
 return
 }
 if !ready {
 w.WriteHeader(503)
 return
 }
 w.WriteHeader(200)
 }))
 defer server.Close()
 probe := func(method, route string, status int, calls int64) {
 before := dependencyCalls.Load()
 request, err := http.NewRequest(method, server.URL+route, nil)
 must(err)
 response, err := client.Do(request)
 must(err)
 _, readError := io.Copy(io.Discard, response.Body)
 closeError := response.Body.Close()
 must(readError)
 must(closeError)
 check(response.Proto == "HTTP/1.1" && response.StatusCode == status, "wrong probe status")
 check(response.Header.Get("Cache-Control") == "no-store", "probe response must not be cached")
 check(dependencyCalls.Load() == before+calls, "unexpected dependency work")
 if status == 405 {
 check(response.Header.Get("Allow") == "GET", "method contract missing")
 }
 fmt.Printf("%s: %d, dependency calls=%d\n", route, status, dependencyCalls.Load())
 }
 probe("GET", "/livez", 200, 0)
 probe("GET", "/startupz", 503, 0)
 probe("GET", "/readyz", 503, 0)
 started.Store(true)
 probe("GET", "/startupz", 200, 0)
 probe("GET", "/readyz", 200, 1)
 mode.Store(1)
 probe("GET", "/livez", 200, 0)
 probe("GET", "/readyz", 503, 1)
 mode.Store(2)
 probe("GET", "/readyz", 503, 1)
 select {
 case <-canceled:
 fmt.Println("deadline canceled the stalled dependency request")
 case <-time.After(time.Second):
 panic("dependency did not observe cancellation")
 }
 mode.Store(0)
 probe("GET", "/readyz", 200, 1)
 draining.Store(true)
 probe("GET", "/readyz", 503, 0)
 probe("GET", "/livez", 200, 0)
 draining.Store(false)
 probe("GET", "/readyz", 200, 1)
 probe("POST", "/readyz", 405, 0)
 probe("GET", "/unknown", 404, 0)
 check(dependencyCalls.Load() == 5, "only five readiness checks should query the dependency")
 fmt.Println("PASS: fourteen probes; live, starting, unready, stalled, draining, and recovered")
 fmt.Println(runtime.Version())
}

Fourteen probes produce five dependency requests. Before startup, readiness and startup return 503 while liveness stays 200. Once initialized, readiness succeeds. A dependency 503 changes readiness but not liveness. A stalled request reaches the child deadline, yields 503, and the dependency observes cancellation; the next healthy check succeeds. Draining returns 503 without contacting the dependency, while liveness remains successful. Clearing the fixture’s drain flag restores readiness. POST and an unknown route produce 405 and 404 independently.

The assertions passed with Go 1.27.0 on macOS. They verify status decisions, dependency-call counts, cancellation, and recovery, not Kubernetes restart thresholds, EndpointSlice propagation, real connection-pool exhaustion, or production load. The live endpoint here proves only that its HTTP handler responds; a real application’s unrecoverable local failure needs its own truthful liveness condition. Repeat the scenario against the configured consumer before relying on its actions.

Sources

Kubernetes: liveness, readiness and startup probes, probe actions, gating, and HTTP success statuses, checked 2026-09-09.
Go context, deadline propagation and cancellation cleanup, checked 2026-09-09.
Go net/http, request contexts, response lifecycle, and redirect policy, checked 2026-09-09.

Full versus incremental backups

Sun, 27 Dec 2026 00:00:00 +0000

A full backup provides a baseline without depending on an earlier backup from the same method. An incremental backup stores changes relative to a specified reference and depends on the material needed to reconstruct the omitted content. That distinction describes recovery dependencies, not a universal promise about speed or bytes transferred. A large changed working set can make an incremental backup substantial too.

This chain references the previous backup. Required WAL and integrity checks remain necessary; the latest increment alone is insufficient.

When it breaks

A damaged full backup can invalidate every incremental that depends on it. Likewise, removing a required middle increment can break later recovery even when the newest directory and its manifest remain intact. Retention must follow dependencies rather than independently deleting directories whose timestamps look old. A newer file is not automatically a replacement for all older recovery material.

The reference matters. A backup relative to the previous increment forms a chain; one relative to a full baseline has a different dependency shape, often described as differential. A synthetic full reconstructs a new baseline from existing backup material. These terms do not guarantee identical implementations across tools. Check the exact product’s manifest and restore rules before designing rotation around the names.

Physical database backups also need the WAL required to make their captured files consistent. A full backup is independent of earlier backups, not independent of all recovery logs, software, configuration, or external dependencies. The experiment includes the required WAL through streaming and keeps the database version and checksum configuration consistent across the chain.

Check it quickly

Run the following as a non-root Linux account with PostgreSQL 18 server and client binaries, including pg_combinebackup, in PATH. The retained article reports execution on PostgreSQL 18.6; this review did not rerun the backup or restore. The script creates private temporary primary and recovery clusters with no TCP listeners. WAL summarization is enabled before taking backups so PostgreSQL can identify changed blocks for the incremental requests.

#!/bin/sh
set -eu
umask 077
lab=$(mktemp -d /tmp/xbcatm-incremental.XXXXXX)
mkdir "$lab/socket"
primary=$lab/primary
restored=$lab/restored
cleanup() {
 for cluster in "$restored" "$primary"; do
 if test -f "$cluster/postmaster.pid"; then
 pg_ctl -D "$cluster" -m fast -w stop >/dev/null
 fi
 done
}
trap cleanup EXIT
initdb -D "$primary" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject >/dev/null
cat >> "$primary/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$lab/socket'
port = 56432
summarize_wal = on
CONF
pg_ctl -D "$primary" -l "$lab/primary.log" -w -t 15 start >/dev/null
export PGHOST=$lab/socket PGPORT=56432 PGDATABASE=postgres PGUSER=$(id -un)
export PGCONNECT_TIMEOUT=2
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
CREATE TABLE marker (id integer PRIMARY KEY,value text);
INSERT INTO marker VALUES (1,'full');
SQL
pg_basebackup -D "$lab/base" -X stream -c fast -w
psql -X -v ON_ERROR_STOP=1 -qAt -c "INSERT INTO marker VALUES (2,'first_increment')"
pg_basebackup -D "$lab/inc1" --incremental="$lab/base/backup_manifest" -X stream -c fast -w
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
UPDATE marker SET value='final' WHERE id=1;
DELETE FROM marker WHERE id=2;
INSERT INTO marker VALUES (3,'second_increment');
SQL
pg_basebackup -D "$lab/inc2" --incremental="$lab/inc1/backup_manifest" -X stream -c fast -w
for backup in "$lab/base" "$lab/inc1" "$lab/inc2"; do
 pg_verifybackup -q "$backup"
done
echo input_backups_verified
if pg_combinebackup -o "$lab/missing-middle" "$lab/base" "$lab/inc2" >"$lab/missing-middle.log" 2>&1; then
 echo 'incomplete chain unexpectedly combined' >&2
 exit 1
fi
grep -E 'starts at LSN .*but expected' "$lab/missing-middle.log"
echo missing_middle_rejected
pg_combinebackup -o "$lab/combined" "$lab/base" "$lab/inc1" "$lab/inc2"
pg_verifybackup -q "$lab/combined"
echo combined_backup_verified
mkdir "$restored"
cp -a "$lab/combined/." "$restored/"
cat >> "$restored/postgresql.conf" <<CONF
port = 56433
summarize_wal = off
CONF
pg_ctl -D "$restored" -l "$lab/restored.log" -w -t 15 start >/dev/null
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
SELECT pg_is_in_recovery();
SELECT id,value FROM marker ORDER BY id;
DO $$ BEGIN
 IF (SELECT string_agg(id||':'||value,',' ORDER BY id) FROM marker)
 IS DISTINCT FROM '1:final,3:second_increment' THEN
 RAISE EXCEPTION 'incremental restore result mismatch';
 END IF;
END $$;
SQL
printf 'lab_directory %s\n' "$lab"

The full backup contains row one with value full. The first increment, referenced to that full backup’s manifest, adds row two. The second increment references the first increment’s manifest, changes row one to final, deletes row two, and adds row three. This tests updates and deletion as well as the simple appearance of new rows.

Each input backup is verified before reconstruction. The first combine attempt intentionally supplies only the full backup and second increment. It must fail with a start-LSN mismatch, and the script checks that reason before printing missing_middle_rejected. No original backup is deleted to create the negative control. The failure demonstrates the dependency that a directory-count check would miss.

The complete combine command supplies the full backup, first increment, and second increment in order. Its synthetic full output is verified again, copied to a separate recovery directory, and started. Expected rows are 1|final and 3|second_increment; the SQL assertion checks that exact combined result. Also inspect the printed recovery flag: expect false. That flag is displayed, not asserted; the SQL assertion checks rows. The deleted row must not reappear.

pg_combinebackup validates relationships between inputs but does not replace each input’s integrity verification. The separate verification steps and real startup therefore establish different facts. The script uses ordinary copying, preserving the input backups while the recovered instance runs. All experimental servers stop on exit, and the printed directory retains logs and artifacts for inspection.

Budget the restore, not just the backup

Incrementals can reduce repeated copying of unchanged blocks, but restoration may require fetching, validating, and combining several artifacts before startup. Measure that complete path, including recovery and application checks. A faster daily backup does not establish that the incident recovery deadline is achievable.

Keep a dependency inventory alongside retention policy. PostgreSQL permits a synthetic full to become a later combine input, which can change the chain an operator needs to retrieve. Verify the new baseline before retiring its predecessors under the chosen policy. Changing checksum state or engine requirements can also require a fresh baseline. Rehearse a recent endpoint and older promised endpoints after rotation so storage savings do not silently shrink the recoverable window.

Sources

PostgreSQL 18 pg_basebackup, incremental references, WAL summaries, and required WAL collection, checked 2026-09-10.
PostgreSQL 18 pg_combinebackup, dependency order, synthetic full backups, and integrity-verification limits, checked 2026-09-12.
PostgreSQL 18 pg_verifybackup, checking backup manifests before actual restore testing, checked 2026-09-10.

Give durable state and scratch space different lifecycles

Thu, 03 Dec 2026 00:00:00 +0000

Durable application state and disposable scratch files need different retention, capacity, and recovery policies. Separating their paths makes those policies easier to express, but directories alone do not create persistence or isolate disk capacity. The storage backing each path determines what survives process restart, container replacement, host failure, or cleanup. A reliable layout connects each class of data to an explicit lifecycle.

When it breaks

A cleanup job that treats one mixed directory as disposable can remove committed records alongside temporary downloads. The reverse problem is backing up large rebuildable caches while missing the small database that actually matters. File extensions and age are weak substitutes for knowing which application owns a path and whether its contents can be reconstructed from an authoritative source.

Container restart and replacement also differ. Stopping and starting the same container normally retains its writable filesystem layer. Removing that container discards the layer, while an explicitly retained named volume can survive. A tmpfs mount has another lifecycle and loses its contents when the container stops. Calling all three locations temporary hides distinctions that affect real recovery procedures.

Two separate directories on the same full filesystem still compete for that filesystem’s free space. Moving scratch from /data/tmp to /scratch without changing storage policy does not automatically protect database writes. Capacity isolation requires an appropriate mount, quota, or other enforceable budget, together with monitoring that distinguishes scratch growth from durable-state growth.

Storage lifecycles differ across stop/start and replacement. The source also checks a bounded scratch mount and a reopened same-volume database copy; neither is independent disaster recovery.

Check it quickly

Run this script using host Python and Docker after obtaining the pinned image. The recorded environment used Docker 29.3.1 and Linux arm64 Python 3.11.16 with its SQLite module. It creates one private named volume and an eight-mebibyte tmpfs scratch mount. Only its own containers and volume are removed; the small scratch exhaustion stays inside the experiment:

import subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def execute(container,code):return docker('exec',container,'python','-c',code).stdout.strip()

name='xbcatm-state-'+uuid.uuid4().hex[:10];volume=name+'-data';containers=[name+'-one',name+'-replacement']
try:
 docker('volume','create',volume)
 def start(container):
 docker('run','-d','--name',container,'--network','none','--mount','type=volume,src='+volume+',dst=/state','--tmpfs','/scratch:rw,size=8m',BASE,'python','-c','import time;time.sleep(180)')
 start(containers[0])
 execute(containers[0],'from pathlib import Path;import sqlite3;c=sqlite3.connect("/state/data.sqlite");c.execute("create table records(value text)");c.execute("insert into records values (?)",("committed",));c.commit();c.close();Path("/ephemeral").write_text("container-layer");Path("/scratch/crumb").write_text("scratch")')
 docker('stop','--time','1',containers[0]);docker('start',containers[0])
 execute(containers[0],'from pathlib import Path;import sqlite3;assert Path("/ephemeral").exists();assert not Path("/scratch/crumb").exists();c=sqlite3.connect("/state/data.sqlite");assert c.execute("select value from records").fetchall()==[("committed",)];c.close()')
 print('stop/start retains the container writable layer and volume data, while the tmpfs scratch mount loses its file')
 docker('rm','-f',containers[0]);start(containers[1])
 execute(containers[1],'from pathlib import Path;import sqlite3;assert not Path("/ephemeral").exists();c=sqlite3.connect("/state/data.sqlite");assert c.execute("select value from records").fetchall()==[("committed",)];c.close()')
 print('replacement loses the old writable layer but retains the named-volume database')
 execute(containers[1],'''import errno,os,sqlite3
fd=os.open('/scratch/fill',os.O_CREAT|os.O_WRONLY,0o600)
try:
 try:
 while True:os.write(fd,b'x'*1048576)
 except OSError as error:assert error.errno==errno.ENOSPC
finally:os.close(fd)
source=sqlite3.connect('/state/data.sqlite');source.execute('insert into records values (?)',('after-scratch-full',));source.commit()
backup=sqlite3.connect('/state/check-copy.sqlite');source.backup(backup);backup.close();source.close()
restored=sqlite3.connect('/state/check-copy.sqlite');assert restored.execute('select value from records order by rowid').fetchall()==[('committed',),('after-scratch-full',)];restored.close()
''')
 print('bounded scratch reaches ENOSPC while state still commits; a SQLite backup copy reopens with both committed records')
finally:
 docker('rm','-f',*containers,check=False);docker('volume','rm',volume,check=False)

The first container creates a SQLite database under /state, commits one record, writes /ephemeral in its ordinary writable layer, and puts a crumb under /scratch. The three writes have distinct storage locations even though the process accesses all of them through ordinary filesystem paths. SQLite commit is used for the durable application record rather than relying on an unclosed output buffer.

After stopping and starting that same container, the database record and /ephemeral still exist, but the scratch crumb is absent. The container command does not recreate these files during startup, so the observation is about storage lifetime rather than initialization hiding the loss. This distinguishes a process restart within a retained container from destruction of its writable layer.

The original container is then removed, and a replacement mounts the same named volume. Its database still contains committed, while /ephemeral is absent. The replacement gets a new writable layer but reconnects to the retained volume. This demonstrates the volume’s independence from one container’s lifecycle, without claiming protection against losing the Docker host or deleting the volume itself.

Next, the replacement fills its bounded scratch mount until a write reports ENOSPC. It can still insert and commit after-scratch-full into the database on the volume. The successful commit demonstrates the particular capacity boundary configured here. It does not imply unlimited durable storage or protection from exhausting some other shared resource, such as total host memory.

Finally, SQLite’s backup API creates a database copy that is closed, reopened, and checked for both committed records in order. This uses the database’s copying interface instead of assuming that copying one live database filename captures every required file consistently. The copy remains on the same experimental volume, so it validates copy readability and content, not an independent disaster-recovery backup.

Make policy follow the data contract

Use a stable state directory for authoritative application data and a separately managed scratch location for reconstructable work. The Filesystem Hierarchy Standard distinguishes application state under /var/lib; container paths can differ while preserving that semantic distinction. Name the directories in application configuration and document which component owns creation, permissions, cleanup, and migration.

Set scratch budgets deliberately and handle exhaustion without deleting unrelated state. A cache is disposable only if its source and reconstruction procedure are actually available. Some apparently temporary files are required for an in-progress transaction or recovery, so cleanup needs the application’s contract rather than an indiscriminate age rule.

Keep independent backups with retention appropriate to the data, and test restoring into a fresh environment. A volume surviving replacement is persistence evidence, not backup evidence; a same-volume copy cannot survive loss of that volume. Monitor the actual backing stores, verify restore contents, and make destructive volume removal an explicit lifecycle decision.

Sources

Docker volumes, data lifecycle independent of container writable layers, checked 2026-09-12.
Docker tmpfs mounts, temporary in-memory mounts and size limits, checked 2026-09-12.
Filesystem Hierarchy Standard: /var/lib, variable state information retained for applications, checked 2026-09-12.
SQLite online backup API, consistent database copying through the database API, checked 2026-09-12.

The golden signals of health

Sun, 29 Nov 2026 00:00:00 +0000

Latency, traffic, errors, and saturation are four complementary ways to ask whether a service is serving users well. They are a starting set of questions, not four universal counters that prove health by themselves. Their usefulness depends on where they are measured, which work they describe, and how their time windows align. A fast response can still be a failed operation.

Separate experience from demand

Latency measures how long work takes at a chosen boundary. Track successful and failed operations separately so that immediate failures do not make the overall distribution appear healthier. Include useful tail information, but also preserve the observation count: a p99 calculated from a tiny population does not establish a stable view of rare behavior. Client latency and server processing time describe different portions of the experience.

Traffic describes demand or completed work in an appropriate unit, such as requests per second or bytes per second. Define whether the measurement counts arrivals, completions, or retries. A completion rate can fall while demand continues rising if a queue is stuck. Concurrent requests are a count, not a request rate; use them as additional context rather than swapping their units into an otherwise identical traffic panel.

Explain failure and resource pressure

Errors measure work that fails the promised outcome. HTTP status failures are one signal, but a nominally successful response containing incorrect data can also violate the contract. Decide which outcomes and latency violations belong in the indicator, and use a denominator with the same scope. Display counts as well as ratios so that one error in a quiet period is not mistaken for thousands of affected requests.

Saturation measures how close a constrained resource is to its usable limit. CPU, memory, connection pools, worker occupancy, and queue age may reveal different bottlenecks. A thread pool with every worker busy is not automatically unhealthy, but a growing queue and aging work show why that occupancy matters. A snapshot is not an interval average, and a low average can conceal brief overload. Capacity thresholds need workload context and measurement, not a decorative red line.

When it breaks

Mixing populations can create an impossible-looking story: latency from successful application requests, errors from an edge proxy, and traffic from another region may change independently. Name boundaries and filters explicitly before correlating them. Fast rejections can improve an all-request latency average while errors rise. Measuring only completions can also omit stalled requests, making both latency and throughput miss the work users are still waiting for.

None of the four signals proves the root cause of a deviation. CPU can be quiet while a database connection pool is exhausted, and queue growth can follow either more demand or slower downstream work. Use the signals to choose a next investigation, then inspect traces, logs, dependencies, and change evidence. Alert on actionable user impact or impending exhaustion; a momentary deviation in any one gauge need not wake a human.

Fast failures lower the combined mean while remaining failures. Compare aligned populations and units; the worker and queue gauges are a separate end-of-window snapshot.

Check it quickly

Save the following as golden.py and run Python 3. It uses one synthetic ten-second interval of completed requests and a separately labeled end-of-interval worker snapshot. The fixture verifies aggregation and interpretation of the four signals. It neither generates load nor measures a live server, and its exact nearest-rank percentile is not an estimate from histogram buckets.

from math import ceil
from statistics import mean

# Synthetic completed requests in one 10-second observation interval.
records = [(True, 0.05)] * 90 + [(True, 0.8)] * 5 + [(False, 0.005)] * 5
success = sorted(duration for ok, duration in records if ok)
failure = [duration for ok, duration in records if not ok]
p99_success = success[ceil(0.99 * len(success)) - 1]
traffic_rps = len(records) / 10
error_ratio = len(failure) / len(records)
# These are end-of-interval gauges, not interval averages.
busy_workers, total_workers = 4, 4
queued_requests, oldest_queue_seconds = 12, 1.2

assert p99_success == 0.8 and traffic_rps == 10 and error_ratio == 0.05
assert mean(failure) < mean(success)
assert busy_workers / total_workers == 1
print(f"traffic={traffic_rps:.1f} completed requests/s; errors={error_ratio:.1%}")
print(f"success p99={p99_success:.3f}s; failed mean={mean(failure):.3f}s")
print(f"success mean={mean(success):.6f}s; all-request mean={mean(d for _, d in records):.6f}s")
print(f"worker snapshot={busy_workers}/{total_workers}; queue={queued_requests}; oldest={oldest_queue_seconds}s")

The output shows ten completed requests per second, a 5% error ratio, successful-request p99 of 0.800 seconds, and failed-request mean of 0.005 seconds. The successful mean is about 0.089474 seconds, while including the fast errors lowers the all-request mean to 0.085250 seconds. A lower combined mean therefore does not imply more useful service.

All four workers are busy in the snapshot, with twelve queued requests and an oldest wait of 1.2 seconds. Those gauges provide pressure context, not proof that occupancy stayed at 100% throughout the interval. Read them with the latency, demand definition, and subsequent queue trend before deciding whether to scale, shed load, or investigate a slower dependency.

Sources

Google SRE: Monitoring Distributed Systems, the four golden signals and successful versus failed request latency, checked 2026-09-13.
Prometheus: Instrumentation, request-serving systems and thread-pool instrumentation, checked 2026-09-13.
Google SRE: Handling Overload, resource limits, queuing, and changing request cost, checked 2026-09-13.

Timestamps need explicit time semantics

Sun, 29 Nov 2026 00:00:00 +0000

A wall-clock label such as 2026-11-01 01:30 does not identify one instant by itself. Its interpretation depends on a timezone and sometimes on which occurrence was intended during a clock transition. Conversely, an explicitly defined Unix timestamp already carries epoch and unit semantics without displaying a zone name. Decide which kind of value your system is storing before choosing a format.

An offset is not a timezone rulebook

An offset-bearing timestamp identifies how its civil fields relate to UTC. The strings 12:00Z and 20:00+08:00 on the same date can represent the same instant. A named IANA zone additionally describes historical and future civil-time rules, including daylight-saving changes. An offset of minus seven hours is not a substitute for America/Los_Angeles when scheduling next month’s local appointment.

For events that already happened, retain an unambiguous instant and render it in the reader’s desired zone. A recurring instruction such as nine each business morning needs the local-time intent and applicable zone rules too. Political changes can alter future offsets, so record the relevant timezone data version when reproducibility matters. Converting today’s offset into a permanent schedule silently changes that meaning.

UTC formatting does not prove clock accuracy or causality. Two machines can both print a Z suffix while their clocks disagree. Timestamps with limited precision can also tie. Keep event identifiers and causal context where ordering matters, and use an appropriate monotonic clock for local elapsed intervals. Timezone conversion cannot repair a clock that measured the wrong instant.

Numeric interfaces must document seconds versus milliseconds and the epoch they use. A large integer is not self-describing, and interpreting its unit incorrectly cannot be corrected later by choosing another display timezone.

When it breaks

Sorting arbitrary timestamp strings can disagree with chronological order. RFC 3339 describes conditions under which text sorting works, including consistent timezone representation and fractional precision. Normalize parsed instants before sorting heterogeneous offsets. A visually later local hour can correspond to an earlier UTC instant, so comparing characters is not a replacement for interpreting the timestamp.

The autumn clock transition can repeat a local hour. Python’s ZoneInfo uses fold to distinguish its two occurrences. The spring transition can instead make a local time nonexistent. Attaching a zone object does not automatically reject such a missing time. An application accepting local appointments must define how to reject or resolve these cases rather than silently inventing the user’s intent.

Arithmetic needs equal care. Operations on datetime objects sharing a tzinfo can follow wall-time semantics and ignore the distinction implied by fold. Convert both values to UTC when you intend to compare elapsed instants. Keep that choice explicit: the interval between two appointments measured by local clock labels may differ from actual elapsed seconds across a transition.

Los Angeles 2026 examples distinguish one repeated local hour from a nonexistent spring time; compare elapsed instants in UTC.

Check it quickly

Save the following as timezone.py and run python3 timezone.py with Python 3.11 or newer and IANA timezone data available. The original report used Python 3.11.2 and Debian tzdata 2026b. Keep assertions enabled, without -O or PYTHONOPTIMIZE. The script uses fixed examples and does not change the machine’s clock or timezone.

from datetime import datetime, timezone
from zoneinfo import ZoneInfo
import json


def instant(text):
 value = datetime.fromisoformat(text)
 if value.tzinfo is None:
 raise ValueError('an instant requires an explicit offset')
 return value.astimezone(timezone.utc)


same = ['2026-09-13T12:00:00Z', '2026-09-13T20:00:00+08:00']
assert instant(same[0]) == instant(same[1])
values = ['2026-01-01T10:00:00+02:00', '2026-01-01T09:30:00+00:00']
assert sorted(values) != sorted(values, key=instant)
try:
 instant('2026-09-13T12:00:00')
except ValueError as error:
 rejection = str(error)
else:
 raise AssertionError('ambiguous instant accepted')

zone = ZoneInfo('America/Los_Angeles')
first = datetime(2026, 11, 1, 1, 30, tzinfo=zone, fold=0)
second = first.replace(fold=1)
first_utc, second_utc = first.astimezone(timezone.utc), second.astimezone(timezone.utc)
assert (second_utc - first_utc).total_seconds() == 3600
missing = datetime(2026, 3, 8, 2, 30)
round_trips = []
for fold in (0, 1):
 candidate = missing.replace(tzinfo=zone, fold=fold)
 returned = candidate.astimezone(timezone.utc).astimezone(zone)
 assert returned.replace(tzinfo=None) != missing
 round_trips.append(returned.isoformat())
print(json.dumps({'same_instant': True, 'lexical_order': sorted(values),
 'instant_order': sorted(values, key=instant), 'naive_rejected': rejection,
 'repeated_wall_time': [first.isoformat(), second.isoformat()],
 'actual_separation_seconds': 3600, 'nonexistent_time_round_trips': round_trips}))

The first assertion shows that two offset representations identify the same instant. Next, lexical sorting puts 09:30+00:00 before 10:00+02:00, while chronological sorting correctly reverses them. The helper rejects a timezone-free value instead of borrowing the host’s current zone. It is a deliberately small instant helper, not a complete RFC 3339 validator.

For Los Angeles on November 1, 2026, the two 01:30 values carry offsets of minus seven and minus eight hours. Converting them to UTC reveals 3,600 elapsed seconds. For March 8 at 02:30, neither fold survives a UTC round trip with the original local fields: the returned times are 03:30 and 01:30. Both complete language copies passed a fresh 2026-10-18 replay with Python 3.12.14 and installed IANA 2026b data; this does not reproduce the original Python 3.11.2 environment. These examples verify specific transition rules in the installed database; they do not promise that every jurisdiction retains today’s rules forever. Preserve instant meaning, local scheduling intent, and parsing policy as separate parts of the data contract.

Sources

RFC 3339: Date and Time on the Internet, offset-bearing timestamps and conditions for string sorting, checked 2026-09-13.
Python: datetime, aware values, UTC conversion and datetime arithmetic, checked 2026-09-13.
Python: zoneinfo, IANA rules, ambiguous transitions and fold, checked 2026-09-13.
Python 3.11: assert statement, optimization removes assertion checks, checked 2026-10-18.

sysctl tunes the running kernel

Sun, 22 Nov 2026 00:00:00 +0000

On Linux, sysctl exposes selected kernel parameters through /proc/sys. It can inspect the running state, and an authorized write can change that state without rebooting. A configuration file is a separate instruction to a loader. Confusing these two layers produces a familiar incident: the desired value is on disk, but the kernel is doing something else.

This article covers Linux with procps-ng and systemd. BSD and macOS also have commands named sysctl, but their interfaces and persistence mechanisms are different. A key count collected on macOS does not establish anything about Linux’s procfs tree.

Read the interface you actually have

For ordinary names, kernel.ostype corresponds to /proc/sys/kernel/ostype; procps accepts slash separators too. Names containing literal dots need the documented separator rules, rather than a blind string replacement. Available keys depend on kernel configuration, loaded modules and the execution environment. The table is not a fixed schema across machines or even across one machine’s lifetime.

Normal procps output uses key = value; -n prints only the value. A successful read establishes an observation in the caller’s environment, not a durable policy. Some values change while you inspect them. Read only the keys needed for the diagnosis: even reads are not universally passive. The manual excludes vm.stat_refresh from ordinary enumeration because reading it flushes per-CPU statistics.

Persistence belongs to a particular loader

A runtime assignment does not create a persistent configuration file. Conversely, saving a drop-in does not apply it immediately. Identify which component applies settings at boot and on reload before reasoning about precedence.

For procps --system, same-named files in higher-priority directories mask lower-priority copies. The selected drop-ins are applied in filename order, regardless of directory; /etc/sysctl.conf is then read last. Systemd’s sysctl.d loader also combines directory priority with filename ordering, but does not directly read /etc/sysctl.conf. Distribution-provided links can still make that file participate through a drop-in. Inspect the installation rather than assuming the pathname alone settles the question.

This has two distinct override rules. A local file can replace a vendor file with the same basename, or a differently named file can override individual assignments by sorting later. Neither rule stops another service from writing the parameter after the loader finishes. Successful boot-time application and the value observed hours later are different facts.

Error policies are not interchangeable

Systemd’s sysctl.d documentation treats missing variables and permission failures as debug-level, nonfatal cases. Other assignment failures make its service fail after processing the remaining settings. A leading minus on an assignment suppresses failure for that assignment. These are systemd loader rules, not a promise about every invocation of procps sysctl.

Procps -e specifically ignores unknown-key errors; it does not mean every malformed value or denied write succeeds. Its configuration format also supports a leading minus to ignore an assignment’s error. Do not infer a portable exit status by combining options and error policies from different implementations.

When it breaks

A module-specific key may not exist when the boot loader runs. Check module availability and loader logs before concluding that a later read disproves the configuration syntax. Another writer may also overwrite a successfully applied value. Record the loader, file precedence, application time and subsequent effective value separately.

Containers add another boundary: mounts, permissions and namespaces affect visibility and write scope. Determine scope from the particular parameter’s documentation; a prefix such as net.* is not sufficient proof that a proposed change is isolated. Do not grant privileges or remount procfs merely to make a troubleshooting command work.

Loader selection and ordering precede a runtime value, which a later writer can change; the source check only reads.

Check it quickly

This proposed read-only exercise requires Linux, Python 3, procps-ng 4.0.4 and readable procfs; its exact live invocation remains unrun. It compares one ordinary key through both interfaces, then checks a nonexistent key. It does not enumerate unknown handlers, load configuration, write settings or test boot persistence.

import os
import subprocess
from pathlib import Path

def require_result(result, status, stdout, stderr):
 if (type(result.returncode) is not int or type(result.stdout) is not bytes
 or type(result.stderr) is not bytes
 or (result.returncode, result.stdout, result.stderr) != (status, stdout, stderr)):
 raise RuntimeError(f"unexpected sysctl result: {result!r}")

def read_sysctl(*args):
 return subprocess.run(
 ["sysctl", *args], capture_output=True, check=False, timeout=5,
 env={**os.environ, "LC_ALL": "C"})

require_result(read_sysctl("--version"), 0,
 b"sysctl from procps-ng 4.0.4\n", b"")
require_result(read_sysctl("kernel.ostype"), 0, b"kernel.ostype = Linux\n", b"")
require_result(read_sysctl("-n", "kernel.ostype"), 0, b"Linux\n", b"")
if Path("/proc/sys/kernel/ostype").read_bytes() != b"Linux\n":
 raise RuntimeError("unexpected procfs kernel type")
require_result(read_sysctl("-n", "kernel.xbcatm_audit_missing_key"), 1, b"",
 b"sysctl: cannot stat /proc/sys/kernel/xbcatm_audit_missing_key: "
 b"No such file or directory\n")
print("matching kernel type: Linux")
print("exact missing-key diagnostic accepted")

Expect the matching-value and exact-diagnostic messages only after all checks succeed. The C-locale negative control requires status 1, empty stdout and the complete missing-path diagnostic, including its newline. Permission errors, missing tools, timeouts, partial output and unexpected success fail instead. Another procps version or diagnostic requires separate review, not a looser substring match. Pure response tests do not validate procfs or a live command. sysctl --system and sysctl -p apply settings; they are not configuration-preview commands and do not belong in this read-only check.

Sources

procps sysctl(8), runtime interface, output options, unknown-key handling and system-file precedence, checked 2026-10-15.
procps sysctl.conf(5), assignment syntax and loader distinction, checked 2026-10-15.
systemd 252 sysctl.d documentation source, separators, missing modules and error policy, checked 2026-10-15.
systemd 252 configuration precedence source, basename masking and lexicographic ordering, checked 2026-10-15.
procps-ng 4.0.4 sysctl source, exact missing-path read error and failure status, checked 2026-10-19.

Logrotate applies a rotation policy when it runs

Sat, 21 Nov 2026 00:00:00 +0000

Logrotate evaluates configuration when invoked, rotates eligible logs, and applies the selected retention and compression actions. A timer or another scheduler determines when that evaluation happens. The tool does not continuously cap file size, and renaming a log does not automatically redirect an application’s open descriptor. A working setup combines evaluation frequency, file policy, and the writer’s actual reopening behavior.

File boxes represent generations and descriptor targets, not a time or byte scale. HUP works here because the fixture explicitly implements reopening; it is not a universal application contract. All CLI outcomes are historical and were not rerun. Current article review remains pending; technical and execution checks remain blocked.

When it breaks

A size threshold can be exceeded between invocations, especially during bursts or scheduler outages. rotate limits generations, not elapsed history or temporary disk use. Threshold times generation count is no hard capacity guarantee: active logs, between-check growth, compression, and copying also consume space.

create sets replacement mode and ownership without signaling the application. Postrotate can invoke supported reopening, but HUP is not universally reload. delaycompress postpones compression one cycle; it cannot fix writers retaining old inodes.

Script failures can occur after file operations have already succeeded. A nonzero logrotate result is not a rollback of every rename or creation. Repeating forced rotation without inspecting the actual files and writer state can rotate again or discard needed generations. Diagnose the partial outcome instead of assuming that failure means nothing changed.

Check it quickly

Repair draft checked 2026-10-19: the revised source was checked statically and with offline command/file-state doubles only. No logrotate execution or signals occurred. Technical and operational verification remain blocked. The original fixture’s historical run used Linux 6.12.76, Python 3.11.2, logrotate 3.21.0 and ext4; none of those observations validate this revision.

For a future run, save as logrotate-cli.py in a separately authorized disposable Linux lab with Python 3.11+, logrotate 3.21.0 and gzip, no host mounts or unrelated workloads. Only after verifying that boundary, run python3 logrotate-cli.py --isolated-disposable-lab as root. The flag and root identity are prerequisites, not proof of isolation. Discard the lab afterward.

The private configuration and state remain separate from system policy. The known child explicitly implements HUP reopening. Startup is inside cleanup protection; nonblocking pipe reads and writes use monotonic five-second deadlines, explicit EOF/error checks and a bounded acknowledgment line. Stderr goes to a private file instead of an unread pipe. Cleanup attempts kill, bounded reap and each stream close independently, preserving the original failure. Earlier cleanup errors or an uncertain logrotate timeout skip file deletion and require disposal; a deletion failure may leave only partial diagnostics. A timed-out CLI can leave hook/compressor descendants even after the logger is reaped. These are controller deadlines, not guarantees against uninterruptible kernel operations or arbitrary descendants.

from pathlib import Path
from tempfile import mkdtemp
import gzip
import os
import select
import shutil
import stat
import subprocess
import sys
import time


worker = r'''
import os,signal,sys
path=sys.argv[1];fd=os.open(path,os.O_WRONLY|os.O_CREAT|os.O_APPEND,0o600)
def reopen(signum,frame):
 global fd
 replacement=os.open(path,os.O_WRONLY|os.O_APPEND)
 os.close(fd);fd=replacement
 print('reopened',flush=True)
signal.signal(signal.SIGHUP,reopen)
try:
 print('ready',flush=True)
 for line in sys.stdin:
 if line=='QUIT\n':break
 data=line.encode()
 if os.write(fd,data)!=len(data):raise OSError('short log write')
 print('wrote',flush=True)
finally:os.close(fd)
'''


def expect(process, expected):
 # Raw nonblocking reads: readiness alone does not mean a complete line.
 descriptor = process.stdout.fileno()
 deadline = time.monotonic() + 5
 answer = bytearray()
 while time.monotonic() < deadline:
 remaining = max(0, deadline - time.monotonic())
 if not select.select([descriptor], [], [], remaining)[0]:
 continue
 try:
 byte = os.read(descriptor, 1)
 except BlockingIOError:
 continue
 if not byte:
 raise RuntimeError(f'logger EOF waiting for {expected}')
 answer.extend(byte)
 if byte == b'\n':
 if answer != (expected + '\n').encode():
 raise RuntimeError(f'unexpected logger acknowledgment: {answer!r}')
 return
 if len(answer) > 64:
 raise RuntimeError('logger acknowledgment is too long')
 raise TimeoutError(f'logger did not acknowledge {expected}')


def send_line(process, text):
 descriptor = process.stdin.fileno()
 pending = (text + '\n').encode()
 deadline = time.monotonic() + 5
 while pending and time.monotonic() < deadline:
 remaining = max(0, deadline - time.monotonic())
 if not select.select([], [descriptor], [], remaining)[1]:
 continue
 try:
 written = os.write(descriptor, pending)
 except BlockingIOError:
 continue
 if written == 0:
 raise RuntimeError('logger input made no progress')
 pending = pending[written:]
 if pending:
 raise TimeoutError('logger input deadline expired')


def attempt(errors, name, operation):
 try:
 operation()
 except Exception as error:
 errors.append(f'{name}: {error!r}')


def main():
 if sys.argv[1:] != ['--isolated-disposable-lab']:
 raise SystemExit('requires a separately verified disposable Linux lab')
 if os.geteuid() != 0:
 raise RuntimeError('root is required, but does not prove isolation')
 root = Path(mkdtemp(prefix='xbcatm-logrotate-', dir='/tmp'))
 active = root / 'app.log'
 configuration = root / 'logrotate.conf'
 state = root / 'state'
 process = error_file = None
 rotation_uncertain = False
 try:
 # A regular error file cannot fill an unread stderr pipe.
 error_file = (root / 'logger.stderr').open('wb')
 process = subprocess.Popen([sys.executable, '-c', worker, str(active)],
 stdin=subprocess.PIPE, stdout=subprocess.PIPE,
 stderr=error_file, bufsize=0)
 os.set_blocking(process.stdin.fileno(), False)
 os.set_blocking(process.stdout.fileno(), False)
 expect(process, 'ready')
 configuration.write_text(f'''"{active}" {{
 size 64
 rotate 2
 compress
 delaycompress
 missingok
 notifempty
 create 0600 root root
 postrotate
 /bin/kill -HUP {process.pid}
 endscript
}}
''')
 def append(text):
 send_line(process, text)
 expect(process, 'wrote')
 def run_rotation(arguments):
 nonlocal rotation_uncertain
 rotation_uncertain = True
 result = subprocess.run(arguments, capture_output=True, text=True, timeout=5)
 rotation_uncertain = False
 return result
 def rotate(*options):
 return run_rotation(['logrotate', *options, '--state', str(state), str(configuration)])
 append('short')
 initial = active.read_bytes()
 result = rotate('--debug')
 assert result.returncode == 0, result.stderr
 assert active.read_bytes() == initial and not state.exists() and not (root / 'app.log.1').exists()
 result = rotate()
 assert result.returncode == 0, result.stderr
 assert state.exists() and not (root / 'app.log.1').exists()
 print('debug changes neither logs nor state; a normal below-threshold run records state without rotating')
 first = initial + b'first-' + b'a' * 80 + b'\n'
 second = b'second-' + b'b' * 80 + b'\n'
 third = b'third-' + b'c' * 80 + b'\n'
 for number, payload in enumerate([first, second, third], 1):
 append((payload[len(initial):] if number == 1 else payload).decode().rstrip('\n'))
 result = rotate()
 assert result.returncode == 0, result.stderr
 expect(process, 'reopened')
 assert active.read_bytes() == b'' and stat.S_IMODE(active.stat().st_mode) == 0o600
 assert (root / 'app.log.1').read_bytes() == payload
 if number >= 2:
 with gzip.open(root / 'app.log.2.gz', 'rb') as handle:
 assert handle.read() == (first if number == 2 else second)
 assert not (root / 'app.log.3.gz').exists()
 append('new-active')
 assert active.read_bytes() == b'new-active\n'
 print('three actual rotations reopen the writer, delay compression one generation, retain two archives, and preserve the expected bytes')
 failed = root / 'failed.log'
 failed.write_bytes(b'failure-fixture')
 bad = root / 'failed.conf'
 bad.write_text(f'''"{failed}" {{
 size 1
 rotate 1
 create 0600 root root
 postrotate
 /bin/false
 endscript
}}
''')
 result = run_rotation(['logrotate', '--state', str(root / 'failed.state'), str(bad)])
 assert result.returncode != 0
 assert (root / 'failed.log.1').read_bytes() == b'failure-fixture' and failed.read_bytes() == b''
 print('a failed postrotate command returns failure after rename/create side effects have already happened')
 send_line(process, 'QUIT')
 assert process.wait(timeout=5) == 0, (root / 'logger.stderr').read_text()
 finally:
 original = sys.exc_info()[1]
 errors = []
 if rotation_uncertain:
 errors.append('logrotate did not return; hook/compressor descendants are unverified')
 if process is not None:
 # Popen.kill is a no-op when this owned child has already completed.
 attempt(errors, 'kill owned logger', process.kill)
 attempt(errors, 'reap logger', lambda: process.wait(timeout=5))
 for name in ('stdin', 'stdout'):
 stream = getattr(process, name)
 if stream is not None:
 attempt(errors, 'close ' + name, stream.close)
 if error_file is not None:
 attempt(errors, 'close logger error file', error_file.close)
 if not errors:
 attempt(errors, 'remove private files', lambda: shutil.rmtree(root))
 if errors:
 message = 'cleanup incomplete; discard lab: ' + '; '.join(errors)
 if original is not None:
 original.add_note(message)
 else:
 raise RuntimeError(message)


if __name__ == '__main__':
 main()

The configuration selects a 64-byte threshold, two generations, one-cycle delayed compression, and root-owned mode-0600 replacements. Its hook signals only the controlled child’s known PID. Write/reopen acknowledgments verify handoff rather than assuming signal-command success proves it.

For a short log, debug mode changes neither logs nor private state. Normal invocation creates state without rotating below threshold: successful evaluation can legitimately leave logs unchanged, and inspection differs from mutation.

Three invocations each follow a known above-threshold payload. Every rotation creates an empty mode-0600 active file; the child acknowledges reopening. Generation one stays uncompressed; its predecessor becomes gzip-compressed generation two.

The fixture decompresses and compares exact bytes. After rotation three, generation two holds payload two and generation one holds payload three; no third compressed generation remains. A final record reaches the current active file, checking retention, compression, and writer placement beyond filenames.

A second private configuration deliberately fails postrotate: logrotate returns failure after moving old content into generation one and creating an empty active file. Bounded logger cleanup precedes private-file removal, as described above. These assertions require a future real run; they are not current execution evidence.

Operate the policy as a complete workflow

Use a state file consistent with the intended rotation job, and account for its locking when scheduling concurrent invocations. Separate state files are not a shared history. Debug mode is useful for reviewing decisions, while force mode changes eligibility and should not be mistaken for proof that the ordinary schedule will rotate at the intended time.

Choose directives according to the desired trigger. In the cited version, size and time criteria are mutually exclusive, with the last specified taking precedence. minsize requires both size and the time interval; maxsize permits earlier size-triggered rotation. All still require an invocation. Verify the effective policy against representative small, large, empty, and missing files, and check the application identity can write the newly created file.

Monitor scheduled evaluations, reopening, filesystem capacity, and retained history. If copytruncate is necessary, accept and test its copy-to-truncate loss window. Add independent shipping or backups when retention requires them; verify old records remain readable after compression, rotation, and recovery.

Sources

logrotate 3.21.0 manual, size, rotate, create, compression, state, debugging, and script failures, checked 2026-09-12.
rename(2), why renaming a pathname does not redirect a live writer, checked 2026-09-12.
signal(7), application-defined HUP handling versus default termination, checked 2026-09-12.
Python 3.11 subprocess, timeout and child cleanup semantics, checked 2026-10-19.
Python 3.11 os, nonblocking descriptor I/O, checked 2026-10-19.

cgroup v2 enforces different budgets with different failure modes

Tue, 10 Nov 2026 00:00:00 +0000

cgroup v2 organizes resource control in a unified hierarchy. A workload’s processes can share aggregate budgets, with limits inherited through the hierarchy rather than applied only to one particular PID. CPU bandwidth, memory usage, and task count are separate controllers with different enforcement behavior. Reading the actual cgroup files and failure counters is more useful than treating every configured limit as the same kind of hard stop.

When it breaks

cpu.max limits CPU bandwidth over a period. It can throttle runnable work after the group consumes its quota, but it does not reserve a dedicated processor or guarantee a particular request latency. Multiple threads can consume a shared quota quickly. CPU weights instead influence relative sharing under contention; confusing weights, quotas, and CPU affinity leads to incorrect capacity expectations.

Memory limits do not universally turn a large allocation into a clean application-level error. At memory.max, the kernel can reclaim memory and may invoke the cgroup OOM mechanism if usage cannot be reduced. Some allocation paths have other failure behavior. memory.high applies reclaim pressure and throttling rather than itself invoking OOM, so the two settings should not be described interchangeably.

Task limits also cover threads, not just the number of application processes a dashboard happens to display. Reaching pids.max can make fork or clone fail even when CPU and memory still look available. Hierarchical restrictions mean a generous child value cannot simply override a tighter ancestor budget. The relevant path and controller delegation matter when interpreting visible files.

CPU bandwidth, memory pressure and task counts have different limits; exit 137 alone does not establish OOM.

Check it quickly

Run the script with Python and a Docker daemon using cgroup v2, after obtaining the pinned image. The recorded environment used Docker 29.3.1, Linux 6.12.76, and Python 3.11.16. Every limit applies only to a disposable experiment container. The script performs a bounded memory exhaustion test, removes its containers, and never changes host-wide cgroup settings:

import json,subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(name):return json.loads(docker('inspect',name).stdout)[0]

name='xbcatm-budgets-'+uuid.uuid4().hex[:10];containers=[name+'-limits',name+'-oom',name+'-killed']
limits_code='''from pathlib import Path
import errno,json,subprocess,time
root=Path('/sys/fs/cgroup')
assert (root/'cgroup.controllers').exists()
quota,period=(root/'cpu.max').read_text().split();assert int(quota)*2==int(period)
assert int((root/'memory.max').read_text())==64*1024*1024
assert int((root/'pids.max').read_text())==12
def stats():return dict(line.split() for line in (root/'cpu.stat').read_text().splitlines())
before=stats();end=time.monotonic()+2
while time.monotonic()<end:pass
after=stats();assert int(after['nr_throttled'])>int(before['nr_throttled'])
children=[]
try:
 while True:
 try:children.append(subprocess.Popen(['sleep','30']))
 except OSError as error:
 assert error.errno==errno.EAGAIN;break
 assert children and int((root/'pids.current').read_text())==12
 assert int(dict(line.split() for line in (root/'pids.events').read_text().splitlines())['max'])>0
 print(json.dumps({'cpu.max':[quota,period],'memory.max':(root/'memory.max').read_text().strip(),'pids.max':12,'children':len(children),'throttled_periods':int(after['nr_throttled'])-int(before['nr_throttled'])}),flush=True)
finally:
 for child in children:child.terminate()
 for child in children:child.wait()
'''
try:
 result=docker('run','--name',containers[0],'--network','none','--cpus','.5','--memory','64m','--memory-swap','64m','--pids-limit','12',BASE,'python','-c',limits_code)
 print(result.stdout.strip())
 oom=docker('run','--name',containers[1],'--network','none','--memory','64m','--memory-swap','64m',BASE,'python','-c','print("allocating",flush=True);data=bytearray(256*1024*1024)',check=False)
 state=inspect(containers[1])['State'];assert oom.returncode==137 and state['OOMKilled'] and state['ExitCode']==137
 docker('run','-d','--name',containers[2],'--network','none',BASE,'python','-c','import time;time.sleep(60)')
 docker('kill','--signal','KILL',containers[2]);docker('wait',containers[2])
 state=inspect(containers[2])['State'];assert state['ExitCode']==137 and not state['OOMKilled']
 print('a 64 MiB memory budget caused actual OOM termination; an explicit SIGKILL also returned 137 but OOMKilled was false')
finally:docker('rm','-f',*containers,check=False)

The first container reads cpu.max as 50000 100000, memory.max as 67,108,864 bytes, and pids.max as twelve. The CPU values represent fifty milliseconds of bandwidth per hundred-millisecond period. Setting Docker’s memory-plus-swap limit equal to its memory limit prevents this fixture from extending its budget through swap; the memory test remains deliberately small.

A two-second busy loop increases cpu.stat’s nr_throttled counter. The recorded increase was twenty periods, while the assertion requires only a positive increase. This confirms actual enforcement rather than merely checking that a Docker flag was accepted. It is not a benchmark proving an exact wall-clock slowdown for every workload or scheduler environment.

The process then starts sleeping children until task creation raises EAGAIN. In the recorded run, eleven children plus the controlling Python process reach the limit of twelve, and the pids controller’s rejection counter increases. The children are terminated and reaped. The refusal comes from the group’s task budget even though the children are mostly idle.

Another container receives a 64 MiB memory limit and attempts to initialize a 256 MiB bytearray. It exits with status 137, and Docker records OOMKilled as true. This validates an actual memory-limit failure for the tested allocation, rather than asserting that every possible allocation must produce the same exception or termination pattern.

A control container is explicitly sent SIGKILL. It also exits with status 137, but OOMKilled is false. That comparison is essential: the exit status alone cannot establish an OOM diagnosis. Correlate runtime state with memory events and system evidence when investigating production incidents, especially if several supervisors can terminate a workload.

Size and observe the whole workload

Budget for the application’s workers and relevant charged memory, not just a single process’s heap estimate. Page cache and other accounted memory can contribute to pressure. Leave room for startup, dependency changes, and recovery operations, then measure the workload under representative concurrency. A tight limit can change latency or failure rates before it produces an obvious crash.

Monitor current usage, pressure signals, and controller events alongside application behavior. Pressure indicators can reveal sustained contention, but a sudden allocation spike may outrun a periodic alert. Monitoring should explain failures and guide sizing without promising that every limit breach will be predicted in advance.

Change resource policy through the manager that owns the workload, such as Docker or systemd, and verify the effective controller values afterward. Container-local cgroup paths may reflect a namespace view rather than the host’s full hierarchy. Correct diagnosis connects that view, inherited constraints, and the specific enforcement mechanism that actually acted.

Sources

Linux 6.12 cgroup v2 documentation, hierarchical CPU, memory, and task controllers and their counters, checked 2026-09-12.
Docker resource constraints, Docker CPU, memory, and swap configuration, checked 2026-09-12.
cgroups(7), control-group membership and resource-accounting semantics, checked 2026-09-12.

Operations · XB Field Notes

Startup probes delay other probes until initialization succeeds

Give each stage an observable condition

When it breaks

Check it quickly

Sources

Change windows fit people, not servers

Traffic is only part of the math

Watch the human clock

Let the window be a habit

When it breaks

Check it quickly

Further reading

Sources

A rota needs continuous coverage and a load review

A job that is shared

Handoff is half the craft

Rehearse before the bell rings

When it breaks

Check it quickly

Further reading

Sources

Backups are untested hope without proof

When it breaks

Check it quickly

Record a recovery result people can use

Sources

Bucket versioning and lifecycles manage cost

Distinguish versioning states

A delete marker changes visibility

Lifecycle expresses a retention tradeoff

When it breaks

Check it quickly

Sources

A pinned base fixes one input; the release needs the rest

When it breaks

Check it quickly

Record a release, then replace its instances

Further reading

Sources

Idle timeouts recycle edge connections

Idle is defined at a particular boundary

When it breaks

Check it quickly

Sources

Error budgets are a real budget

Give the account a measurement rule

Decide the response before the argument

When it breaks

Check it quickly

Further reading

Sources

Incident response needs a timeline

Write as it happens

Separate communication from execution

Learning requires the record

When it breaks

Check it quickly

Further reading

Sources

Failover is a practiced switch

Design the decision in advance

Rehearse the actual path

When it breaks

Check it quickly

Further reading

Sources

Capacity planning is a number, not a guess

Measure an acceptable operating point

Forecast demand and remove capacity deliberately

When it breaks

Check it quickly

Further reading

Sources

Feature flags decouple release from exposure

Ship behind a defined gate

Choose defaults by meaning

Ramp exposure with stable identities

When it breaks

Check it quickly