During an incident, memory is unreliable, and clocks can disagree too. A timeline — who was paged, what was observed, what was tried, when the change landed — is what makes a response recordable and later reviewable.
Write as it happens
Memory is the least reliable instrument in an incident, so record while the alert is fresh. Capture the timestamp of the first alert, the initial symptoms, each action tried and its observable result, and the moment the fix actually took effect. That running record is what keeps a team from repeating a failed fix during the response, and what later makes an honest postmortem possible instead of a plausible story. Make the timeline a live document from the minute the page fires: note the tool that alerted, the metric, the suspected and the confirmed cause, and record ‘same as before, did not help’ rather than silently re-trying.
Separate communication from execution
As an incident grows, debugging and keeping everyone informed compete for attention. Split operational and communication responsibilities: one person acts, another maintains the timeline and shares status, scope, and open questions. That second seat prevents the classic failure where everyone is heads-down troubleshooting and nobody is recording what was tried. A shared timeline also shortens onboarding: a teammate who arrives late reads the record and catches up, instead of interrupting the lead with questions or trusting a scattershot retelling. Update it as events land, with links to supporting logs and decisions.
Learning requires the record
A postmortem written from memory is a plausible story; one written from a timeline is a reconstruction of facts you can cross-check. Because the timeline pins actions to timestamps and outcomes, the review can focus on systemic gaps rather than individual blame: which alert did not exist, which runbook was wrong, which decision had no owner. That is the real payoff of recording from the first minute. Start at the first alert when possible, and label reconstructed entries if recording begins later. A partial, sourced record is still useful; do not fill gaps with assumed certainty.
When it breaks
A live timeline fails while the incident is still open, not after it. The first failure is a clock without a zone: one responder writes 10:02 from a laptop on local time while the alert pipeline logs in UTC, so identical-looking marks can represent different instants. Record date, offset, clock source, and any known skew. Even synchronized timestamps alone cannot establish causation.
The second failure is the next-morning rewrite. The team reconstructs the record from memory, keeps the fix, drops the two retries that did not help, and reorders events into a tidy story. That version hides exactly what the next responder needs: what was already tried and what observably changed.
The third failure is concurrent work without coordination. Two responders can both claim the rollback slot even if the editor preserves every line. Assign change owners and stable entry identifiers; preserve the original recording order and mark late additions. A chronological view should not silently rewrite the underlying evidence.
The fourth failure is the chat burial. The failover decision is approved deep in a thread with two hundred side messages, but nobody copies the decision, owner, and expected effect into the timeline. An hour later nobody can answer who decided what. Unlike the companion postmortem piece about blame-free learning after the fact, these are recording failures during the response itself.
Check it quickly
Save the following as timeline_check.py and run python3 timeline_check.py. It uses only the Python standard library, builds one good and one bad in-memory timeline, writes no files, and checks the three fields every live entry needs: t, actor, and action.
"""Live incident timeline validator: each event needs t/actor/action, ordered by time."""
from datetime import datetime
def parse_time(raw):
text = str(raw or "").strip().replace("Z", "+00:00")
# Require a full timestamp with date and timezone, not a bare wall-clock time.
if "T" not in text or not text:
raise ValueError(f"bad timestamp {raw!r}: use ISO-8601 with date and timezone")
try:
stamp = datetime.fromisoformat(text)
except ValueError:
raise ValueError(f"bad timestamp {raw!r}: use ISO-8601 with date and timezone") from None
if stamp.tzinfo is None:
raise ValueError(f"bad timestamp {raw!r}: missing timezone")
return stamp
def check_timeline(events):
errors = []
seen = []
for i, ev in enumerate(events):
for field in ("t", "actor", "action"):
if not isinstance(ev.get(field), str) or not ev[field].strip():
errors.append(f"event {i} missing {field} or not a nonempty string")
try:
seen.append(parse_time(ev.get("t", "")))
except ValueError as exc:
errors.append(f"event {i} {exc}")
seen.append(None)
stamps = [s for s in seen if s is not None]
# Only judge ordering over entries that parsed, so one bad clock
# does not hide a second sequencing mistake.
ordered = sorted(stamps)
if stamps != ordered:
errors.append("events out of order: review clock sources and late entries")
return errors
GOOD = [
{"t": "2027-03-01T10:01:00+00:00", "actor": "pager", "action": "alert fired: error-rate burn"},
{"t": "2027-03-01T10:04:00+00:00", "actor": "ana", "action": "took timeline keeper seat, opened shared doc"},
{"t": "2027-03-01T10:09:00+00:00", "actor": "lee", "action": "rolled back deploy 1a2b3c, error-rate still 8%"},
{"t": "2027-03-01T10:14:00+00:00", "actor": "ana", "action": "decision: fail over to region-b, owner lee, effect visible by 10:20"},
]
BAD = [
{"t": "10:09", "actor": "lee", "action": "rollback done?"},
{"t": "2027-03-01T10:04:00+00:00", "actor": None, "action": "someone joined late, backfilled from memory"},
{"t": "2027-03-01T10:02:00+00:00", "actor": "ana", "action": []},
]
def main():
good_errors = check_timeline(GOOD)
bad_errors = check_timeline(BAD)
assert good_errors == [], f"good sample should pass, got: {good_errors}"
assert any("missing actor" in e for e in bad_errors), f"bad sample should flag actor, got: {bad_errors}"
assert any("missing action" in e for e in bad_errors), f"bad sample should flag action, got: {bad_errors}"
assert any("bad timestamp" in e or "timezone" in e for e in bad_errors), f"bad sample should flag clock, got: {bad_errors}"
assert any("out of order" in e for e in bad_errors), f"bad sample should flag order, got: {bad_errors}"
print("PASS: good timeline has t/actor/action on every event in time order")
print(f"FAIL as expected: bad timeline -> {'; '.join(bad_errors)}")
print("summary: 1 passed check, 1 failed check (expected 1 and 1)")
if __name__ == "__main__":
main()
The good sample passes because every event carries a full ISO-8601 timestamp with timezone, a named actor, and an observable action in time order. The bad sample fails together on a bare wall-clock time, a missing actor, a missing action, and backward stamps. Expect one PASS line, one FAIL line naming each defect, and a one-passed-one-failed summary. The revised script was run with Python 3.12.14 on Linux. Equal timestamps are allowed; this checks nondecreasing recorded times, not unique sequencing or truthful observations. Python accepts several ISO-8601 forms here; this is not a strict RFC 3339 validator. It checks recording discipline during the response, not the post-incident review in the companion postmortem article.
Further reading
Sources
- Google SRE: managing incidents, incident roles, living documents, and explicit handoffs, checked 2026-10-15.
- Google SRE: postmortem culture, evidence-based and blameless learning, checked 2026-10-15.
- Python datetime, aware timestamp parsing and comparisons, checked 2026-10-15.
- RFC 3339 timestamps, offsets and limitations of textual ordering, checked 2026-10-15.
