Incident · XB Field Notes

Postmortems are for learning, not blame

Fri, 05 Mar 2027 00:00:00 +0000

After an incident, a postmortem should explain what happened, what people knew at the time, and which changes could reduce future harm. Blamelessness makes candid reporting safer. It does not guarantee that an incident cannot recur, remove ownership, or turn every explanation into an established fact. Learning requires both an honest account and evidence that useful follow-up actually happened.

Reconstruct the decision context

Hindsight makes an earlier decision look obvious once its consequences are known. Ask what signals, documentation, time pressure, and alternatives were available when the decision was made. An operator’s action belongs in the timeline when relevant, but stopping at that person’s name leaves the conditions that made the action reasonable or likely unexplored.

A timeline is an evidence-backed reconstruction, not an immutable transcript. Preserve logs and messages, distinguish observation from inference, and mark uncertain timestamps or conflicting accounts. Correct the account when stronger evidence appears. A short timeline is a reason to investigate missing context, not proof that somebody deliberately hid information. Accusing authors of concealment can reproduce the blame the review is intended to avoid.

Connect consequences, conditions, and actions

Begin with customer impact and how it was measured. Record detection, mitigation, and recovery separately: a rollback command finishing does not establish that customer traffic recovered at that exact moment. Describe the triggering change alongside contributing conditions, such as a missing limit or a dependency that made a local error spread.

Choose a small, prioritized set of actions tied to those conditions. Useful actions may prevent recurrence, improve detection, limit impact, or make recovery safer. Not every contributing factor needs its own production change; investigate uncertainty and document accepted tradeoffs. A runbook correction or training exercise can be valuable when it addresses a concrete failure, rather than serving as a generic instruction to be more careful.

Give each accepted action one accountable owner, a target date, a tracking reference, and a verifiable outcome. The owner coordinates completion; naming them does not blame them for the incident. A verification statement should describe an observable result, such as a controlled test rejecting an unsafe release, rather than simply stating that a document was written.

Follow through without inventing certainty

Schedule follow-up according to risk and delivery time. A review a week later can be useful, but no universal seven-day deadline fits every incident. Check whether each change landed, whether its validation passed, and whether it produced new problems. If the proposed fix fails, keep the work open with an updated plan rather than declaring the loop closed because another meeting occurred.

Recurring incidents deserve investigation even when previous actions were completed. The original explanation may have been incomplete, a mitigation may only reduce impact, or conditions may have changed. Compare consequences and contributing factors as well as counts. Fewer reports alone cannot establish improvement, particularly if reporting incentives have changed.

When it breaks

Vague tasks can survive every meeting because nobody can tell whether they are finished. Overly ambitious actions can remain open indefinitely because their priority and staffing never matched the commitment. Review feasibility when accepting work and record explicit deferrals instead of silently losing them.

A checklist can also become a substitute for learning. Filled fields do not establish a sound causal explanation, a safe proposed change, or a trustworthy verification method. Keep sensitive customer data and credentials out of broadly shared reports; preserve necessary evidence with appropriate access rather than copying everything into the narrative.

Complete fields can still hide a weak action. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

Save the following as postmortem_check.py and run python3 postmortem_check.py. Python 3.7 or newer checks three synthetic review records and additional date controls using only its standard library. Run without -O, -OO, or a nonzero PYTHONOPTIMIZE setting so the assertions execute. The accepted input shape is a dictionary with an action list of dictionaries; this is a local metadata exercise, not a general untrusted-document validator.

from copy import deepcopy
from datetime import date
import re


def check_review(doc):
 errors = []
 for key in ("timeline", "causes", "actions"):
 if not doc.get(key):
 errors.append(f"missing {key}")
 for i, action in enumerate(doc.get("actions", [])):
 for field in ("task", "owner", "proof"):
 value = action.get(field)
 if not isinstance(value, str) or not value.strip():
 errors.append(f"action {i} missing {field}")
 due = action.get("due")
 try:
 if not isinstance(due, str) or not re.fullmatch(r"[0-9]{4}-[0-9]{2}-[0-9]{2}", due):
 raise ValueError("date format")
 date.fromisoformat(due)
 except ValueError:
 errors.append(f"action {i} invalid due date")
 return errors


GOOD = {
 "timeline": ["00:01 alert fired", "00:07 rollback completed"],
 "causes": {"trigger": "deployment", "conditions": ["no canary gate"]},
 "actions": [{"task": "add a canary gate", "owner": "ana",
 "due": "2026-10-22", "proof": "staging test blocks a bad release"}],
}
BAD = deepcopy(GOOD)
BAD["actions"][0].update(owner=None, due="soon", proof="")
SHALLOW = deepcopy(GOOD)
SHALLOW["actions"][0].update(task="be more careful", proof="manager agrees")

assert check_review(GOOD) == []
assert check_review(BAD) == ["action 0 missing owner", "action 0 missing proof",
 "action 0 invalid due date"]
assert check_review(SHALLOW) == [] # Metadata completeness cannot judge quality.
for invalid in ("2026-02-30", "20261022", None):
 candidate = deepcopy(GOOD)
 candidate["actions"][0]["due"] = invalid
 assert check_review(candidate) == ["action 0 invalid due date"]
print("PASS: complete metadata accepted; missing owner/proof and invalid dates rejected")
print("LIMIT: a vague action still passes; causal review requires human judgment")

The complete record passes. A null owner, empty proof, informal deadline, impossible calendar date, compact date, and null date are rejected. Crucially, the vague action still passes because its fields are populated. The final LIMIT line exposes that deliberate boundary: metadata validation supports review, but cannot certify learning or prevent another incident.

Sources

Google SRE: postmortem culture, blameless investigation, review and appropriate sharing, checked 2026-10-15.
Google SRE Workbook: postmortem culture, action ownership, priorities, verification and follow-up, checked 2026-10-15.

Incident response needs a timeline

Mon, 01 Mar 2027 00:00:00 +0000

During an incident, memory is unreliable, and clocks can disagree too. A timeline — who was paged, what was observed, what was tried, when the change landed — is what makes a response recordable and later reviewable.

Times and actors are transcribed from the inert source fixture. The shared date and UTC offset are display shorthand; underlying entries require full timestamps. Vertical spacing is schematic, not elapsed-time scale. A planned effect is not shown as a confirmed fix; no validator or incident system was run.

Write as it happens

Memory is the least reliable instrument in an incident, so record while the alert is fresh. Capture the timestamp of the first alert, the initial symptoms, each action tried and its observable result, and the moment the fix actually took effect. That running record is what keeps a team from repeating a failed fix during the response, and what later makes an honest postmortem possible instead of a plausible story. Make the timeline a live document from the minute the page fires: note the tool that alerted, the metric, the suspected and the confirmed cause, and record ‘same as before, did not help’ rather than silently re-trying.

Separate communication from execution

As an incident grows, debugging and keeping everyone informed compete for attention. Split operational and communication responsibilities: one person acts, another maintains the timeline and shares status, scope, and open questions. That second seat prevents the classic failure where everyone is heads-down troubleshooting and nobody is recording what was tried. A shared timeline also shortens onboarding: a teammate who arrives late reads the record and catches up, instead of interrupting the lead with questions or trusting a scattershot retelling. Update it as events land, with links to supporting logs and decisions.

Learning requires the record

A postmortem written from memory is a plausible story; one written from a timeline is a reconstruction of facts you can cross-check. Because the timeline pins actions to timestamps and outcomes, the review can focus on systemic gaps rather than individual blame: which alert did not exist, which runbook was wrong, which decision had no owner. That is the real payoff of recording from the first minute. Start at the first alert when possible, and label reconstructed entries if recording begins later. A partial, sourced record is still useful; do not fill gaps with assumed certainty.

When it breaks

A live timeline fails while the incident is still open, not after it. The first failure is a clock without a zone: one responder writes 10:02 from a laptop on local time while the alert pipeline logs in UTC, so identical-looking marks can represent different instants. Record date, offset, clock source, and any known skew. Even synchronized timestamps alone cannot establish causation.

The second failure is the next-morning rewrite. The team reconstructs the record from memory, keeps the fix, drops the two retries that did not help, and reorders events into a tidy story. That version hides exactly what the next responder needs: what was already tried and what observably changed.

The third failure is concurrent work without coordination. Two responders can both claim the rollback slot even if the editor preserves every line. Assign change owners and stable entry identifiers; preserve the original recording order and mark late additions. A chronological view should not silently rewrite the underlying evidence.

The fourth failure is the chat burial. The failover decision is approved deep in a thread with two hundred side messages, but nobody copies the decision, owner, and expected effect into the timeline. An hour later nobody can answer who decided what. Unlike the companion postmortem piece about blame-free learning after the fact, these are recording failures during the response itself.

Check it quickly

Save the following as timeline_check.py and run python3 timeline_check.py. It uses only the Python standard library, builds one good and one bad in-memory timeline, writes no files, and checks the three fields every live entry needs: t, actor, and action.

"""Live incident timeline validator: each event needs t/actor/action, ordered by time."""
from datetime import datetime


def parse_time(raw):
 text = str(raw or "").strip().replace("Z", "+00:00")
 # Require a full timestamp with date and timezone, not a bare wall-clock time.
 if "T" not in text or not text:
 raise ValueError(f"bad timestamp {raw!r}: use ISO-8601 with date and timezone")
 try:
 stamp = datetime.fromisoformat(text)
 except ValueError:
 raise ValueError(f"bad timestamp {raw!r}: use ISO-8601 with date and timezone") from None
 if stamp.tzinfo is None:
 raise ValueError(f"bad timestamp {raw!r}: missing timezone")
 return stamp


def check_timeline(events):
 errors = []
 seen = []
 for i, ev in enumerate(events):
 for field in ("t", "actor", "action"):
 if not isinstance(ev.get(field), str) or not ev[field].strip():
 errors.append(f"event {i} missing {field} or not a nonempty string")
 try:
 seen.append(parse_time(ev.get("t", "")))
 except ValueError as exc:
 errors.append(f"event {i} {exc}")
 seen.append(None)
 stamps = [s for s in seen if s is not None]
 # Only judge ordering over entries that parsed, so one bad clock
 # does not hide a second sequencing mistake.
 ordered = sorted(stamps)
 if stamps != ordered:
 errors.append("events out of order: review clock sources and late entries")
 return errors


GOOD = [
 {"t": "2027-03-01T10:01:00+00:00", "actor": "pager", "action": "alert fired: error-rate burn"},
 {"t": "2027-03-01T10:04:00+00:00", "actor": "ana", "action": "took timeline keeper seat, opened shared doc"},
 {"t": "2027-03-01T10:09:00+00:00", "actor": "lee", "action": "rolled back deploy 1a2b3c, error-rate still 8%"},
 {"t": "2027-03-01T10:14:00+00:00", "actor": "ana", "action": "decision: fail over to region-b, owner lee, effect visible by 10:20"},
]

BAD = [
 {"t": "10:09", "actor": "lee", "action": "rollback done?"},
 {"t": "2027-03-01T10:04:00+00:00", "actor": None, "action": "someone joined late, backfilled from memory"},
 {"t": "2027-03-01T10:02:00+00:00", "actor": "ana", "action": []},
]


def main():
 good_errors = check_timeline(GOOD)
 bad_errors = check_timeline(BAD)
 assert good_errors == [], f"good sample should pass, got: {good_errors}"
 assert any("missing actor" in e for e in bad_errors), f"bad sample should flag actor, got: {bad_errors}"
 assert any("missing action" in e for e in bad_errors), f"bad sample should flag action, got: {bad_errors}"
 assert any("bad timestamp" in e or "timezone" in e for e in bad_errors), f"bad sample should flag clock, got: {bad_errors}"
 assert any("out of order" in e for e in bad_errors), f"bad sample should flag order, got: {bad_errors}"
 print("PASS: good timeline has t/actor/action on every event in time order")
 print(f"FAIL as expected: bad timeline -> {'; '.join(bad_errors)}")
 print("summary: 1 passed check, 1 failed check (expected 1 and 1)")


if __name__ == "__main__":
 main()

The good sample passes because every event carries a full ISO-8601 timestamp with timezone, a named actor, and an observable action in time order. The bad sample fails together on a bare wall-clock time, a missing actor, a missing action, and backward stamps. Expect one PASS line, one FAIL line naming each defect, and a one-passed-one-failed summary. The revised script was run with Python 3.12.14 on Linux. Equal timestamps are allowed; this checks nondecreasing recorded times, not unique sequencing or truthful observations. Python accepts several ISO-8601 forms here; this is not a strict RFC 3339 validator. It checks recording discipline during the response, not the post-incident review in the companion postmortem article.

Sources

Google SRE: managing incidents, incident roles, living documents, and explicit handoffs, checked 2026-10-15.
Google SRE: postmortem culture, evidence-based and blameless learning, checked 2026-10-15.
Python datetime, aware timestamp parsing and comparisons, checked 2026-10-15.
RFC 3339 timestamps, offsets and limitations of textual ordering, checked 2026-10-15.

A postmortem should leave a fact record

Tue, 07 Jul 2026 00:00:00 +0000

A postmortem earns trust when its claims can be re-checked. Record each observation’s source, collection time and timezone, command or query, and exit status where relevant. Logs, metrics, and clearly attributed human observations also count; distinguish confirmed facts from hypotheses and preserve redacted evidence rather than secrets. The four phases below keep a write-up reviewable at any later time.

Tie every phase to captured evidence. A post-fix sample is a bounded observation; causes, uncertainty and owned prevention still need explicit treatment.

Detect: the symptom, with evidence

Start with the smallest verifiable symptom and the command that showed it. Save the actual capture. The following is an illustrative excerpt, not evidence of a fresh probe:

$ check-http-headers.sh https://www.xbcatm.com/
HTTP/2 200
content-security-policy: ... script-src 'none' ...
last-modified: Mon, 24 Aug 2026 11:20:54 GMT

If the symptom is that a header changed, the output showing the previous policy and the new one belongs side by side. If it is a failure, include the exit status and the error line.

Scope: how wide, and how you know

State the blast radius with evidence, not adjectives. Define the boundary you can inspect:

Which hosts and paths show the symptom? Record each probed URL and observation time; one successful request cannot establish the state of every edge or visitor.
Does the cache differ from origin? Compare actual responses, cache metadata, and validators from both paths using the same request headers. Vary describes selection inputs, not proof that the response came from cache.
What delays recovery? For a DNS change, record the old TTL and resolver observations. For HTTP caches, record freshness and revalidation policy separately; a DNS TTL is not a universal rollback deadline.

Fix: the change and its intended effect

Record the actual configuration diff and deployment reference so the corrective action can be understood and, where safe, replayed. This header fragment is an expected observation, not a shell command or a complete policy. Prefer the smallest change that restores the contract, and name what the post-fix run should show:

Content-Security-Policy: script-src 'none'

Verify: the post-incident run

Repeat the detection command after the change and show that the observed output now matches the intended contract. A postmortem without this final section is a story; with it, it is a record that approves its own future re-check.

Close the record with contributing causes, what remains uncertain, and preventive actions with an owner, due date, and acceptance check. A successful probe confirms that sampled path at that time; it does not establish the cause or prevent recurrence by itself.

Sources

RFC 9110 — HTTP Semantics, response metadata and conditional requests, checked 2026-07-07.
artifact HTTP header check, read-only inspection used throughout this record.
Google SRE: Postmortem Culture, impact, contributing causes, evidence review, and preventive actions, checked 2026-10-15.
RFC 9111 — HTTP Caching, cache selection and freshness are distinct from DNS cache lifetimes, checked 2026-10-15.

Incident · XB Field Notes

Postmortems are for learning, not blame

Reconstruct the decision context

Connect consequences, conditions, and actions

Follow through without inventing certainty

When it breaks

Check it quickly

Further reading

Sources

Incident response needs a timeline

Write as it happens

Separate communication from execution

Learning requires the record

When it breaks

Check it quickly

Further reading

Sources

A postmortem should leave a fact record

Detect: the symptom, with evidence

Scope: how wide, and how you know

Fix: the change and its intended effect

Verify: the post-incident run

Sources