Reliability · XB Field Notes

Backpressure works when bounded admission changes the producer’s behavior

Thu, 12 Aug 2027 00:00:00 +0000

Backpressure connects a consumer’s limited capacity to the behavior of its producers. A full channel may make a producer wait, reject new work, or apply a deliberately chosen shedding policy. The important property is that the limit changes what upstream components do. A growing queue or a dashboard showing lag is evidence of imbalance, but neither is automatically a mechanism that controls admission.

When it breaks

An unbounded queue can absorb a temporary burst while hiding sustained overload. If work arrives faster than it finishes, outstanding work and waiting time continue growing. Moving the same backlog into pending threads, retry tasks, or another unbounded buffer does not remove that pressure. The capacity boundary must cover the place where work is actually retained, not merely one convenient queue object.

A bounded queue also does not define a complete overload policy by itself. Blocking forever can consume all producer threads or leave shutdown stuck. Immediate rejection can lose work if callers treat it as success. Retrying every rejection immediately can increase load. Decide whether work should wait within a deadline, be rejected for later retry, or be discarded under an explicit application contract.

Queue capacity and processing completion are separate. Removing an item can free a queue slot while the consumer is still working on that item. A capacity of two waiting items may therefore coexist with additional in-progress work. Likewise, a producer’s successful enqueue says only that the queue accepted the item; it does not establish execution, durable storage, or completion of its downstream side effects.

Removing a frees admission capacity before processing is acknowledged. A successful put, an empty queue and a completed join represent different boundaries; queue accounting is not durable delivery.

Check it quickly

Save this as check-backpressure.py and run python3 check-backpressure.py. It uses only Python’s standard library and was executed with Python 3.14.4. Threads coordinate through explicit events and bounded waits, so the example does not depend on a remote broker, database, or an artificially slow external service. All queues and work items exist only inside the process:

from queue import Queue,Full
from threading import Event,Thread
import json

unbounded=Queue()
for item in range(1000):unbounded.put_nowait(item)
assert unbounded.qsize()==1000
channel=Queue(maxsize=2)
channel.put('a');channel.put('b')
try:
 channel.put('deadline-rejected',timeout=.02)
 raise AssertionError('a full bounded queue accepted another item')
except Full:rejected=True
assert channel.qsize()==2
entered,delivered=Event(),Event()
def produce():
 entered.set();channel.put('c',timeout=2);delivered.set()
producer=Thread(target=produce,daemon=True);producer.start();assert entered.wait(1)
assert not delivered.wait(.05) and channel.qsize()==2
first=channel.get(timeout=1);assert first=='a'
assert delivered.wait(1);producer.join(timeout=1);assert not producer.is_alive()
assert channel.qsize()==2
rest=[channel.get(timeout=1),channel.get(timeout=1)];assert rest==['b','c']
assert channel.empty()
completed=Event()
def join():channel.join();completed.set()
waiter=Thread(target=join,daemon=True);waiter.start();assert not completed.wait(.05)
for item in [first,*rest]:channel.task_done()
assert completed.wait(1);waiter.join(timeout=1);assert not waiter.is_alive()
print(json.dumps({'unbounded_backlog':unbounded.qsize(),'bounded_capacity':2,'deadline_rejected':rejected,
 'blocked_producer_resumed_after_get':delivered.is_set(),'delivered_items':[first,*rest],
 'empty_queue_did_not_mean_processing_complete':True,'join_finished_after_task_done':completed.is_set()}))

The first queue has the default unbounded capacity. With no consumer running, a thousand put_nowait calls all succeed and leave a backlog of a thousand items. This is a small behavioral demonstration, not a memory-exhaustion benchmark. It shows that successful admission alone cannot establish downstream progress or that the arrival rate is sustainable.

The second queue has capacity two and starts with items a and b. Another put uses a short timeout and raises Full, leaving the queued items unchanged. The failed item is not secretly enqueued for later execution. This gives the caller a definite admission outcome on which to base retry or rejection handling instead of presenting an unlimited wait as successful delivery.

A producer thread then tries to put c. The test confirms that it remains blocked while both slots are occupied. Removing a opens one slot, the producer completes its put, and b and c remain queued. The program verifies that all three accepted items are retrieved in order. This demonstrates pressure reaching the producer and being released by a concrete capacity change.

After retrieving the remaining items, the queue is empty, but a separate thread waiting in join still has not completed. Only after task_done is called for all three accepted items does join return. Dequeueing freed capacity earlier; task acknowledgment marks the tracked work complete later. The test separates these events so an empty queue cannot be mistaken for completed business processing.

Preserve the admission contract across the pipeline

Use the queue operation’s actual result rather than a check-then-act decision based on qsize or full. Those observations are approximate when other threads can change the queue. The fixture’s events and paused consumers make its assertions controlled observations; a production caller should handle Full or a successful put atomically through the provided operation.

Choose a bound appropriate to retained resources. An item-count limit does not cap memory when items can be arbitrarily large, and a waiting-queue limit does not include every active worker or external request. Include processing concurrency and upstream retry behavior when evaluating overload. A bounded component surrounded by unlimited pending work can still participate in an unbounded system.

Finally, define what completion means. Python task_done updates queue accounting; it is not a durable transaction or proof that an external side effect succeeded. A worker failure, process crash, or retry after an ambiguous response needs the application’s corresponding delivery and idempotency contract. Backpressure controls admission and pace; correctness still depends on handling accepted, rejected, in-progress, and completed work explicitly.

Sources

Python queue documentation, bounded capacity, Full, blocking puts and task completion, checked 2026-09-12.
Google SRE: Handling Overload, client throttling, load shedding, and resource-based overload limits, checked 2026-09-12.

Cache invalidation is a push, not a prayer

Thu, 05 Aug 2027 00:00:00 +0000

Changing an origin does not notify every cache that already holds its response. Invalidation is about controlling which stored copies remain eligible for reuse, not merely updating a file. A reliable release distinguishes expiration, validation, targeted removal, and choosing a new URL. These mechanisms have different scopes, and none remotely erases every copy a reader has downloaded.

Two independent private caches make removal scope visible: changing one entry leaves the other usable. Publishing and requesting a new URL creates a separate entry; it does not erase the old URL or update every reader automatically.

Expiration is not a broadcast

A fresh response can remain usable after its source changes. Freshness is calculated from lifetime and current age, accounting for cache metadata and elapsed time; it is not simply a countdown from one origin Date header. Expiration makes direct reuse conditional on the applicable rules. It neither guarantees physical deletion nor automatically causes an idle cache to contact the origin.

Validators are consulted when a request requires validation, not pushed to clients when content changes. A correct ETag lets the origin confirm an unchanged representation or send its replacement. It cannot force a browser holding a still-fresh response to ask. Changing the origin to no-cache or no-store does not retroactively rewrite previously received caching instructions.

Identify the scope before invalidating

RFC 9111 requires a cache receiving a successful or redirect response to an unsafe method to invalidate the target URI. This affects caches on that request’s path; it is not global notification. Application database writes made through another route do not automatically invalidate every derived page, query variant, translated response, or search listing.

A CDN purge similarly controls the provider’s managed copies, not independent browser caches. Identify the affected host, paths, variants, and cache tags using the provider’s actual contract. Publish the replacement before invalidating, track completion, then compare response bodies as well as cache status. An accepted purge request alone proves neither propagation nor that the origin now serves the desired bytes.

When it breaks

Purging before the origin is updated can immediately repopulate an edge with the old response. An in-flight refill can also race with invalidation; its handling depends on the cache implementation. Repeating a broad purge is not evidence that either race is resolved. Prefer a narrowly identified object set, and account for the additional origin load when those objects become cold.

For static assets, publish a new content-fingerprinted filename before switching HTML or a manifest to reference it. Keep the old asset available while previously cached documents can still reference it. The new URL avoids confusion with the old cache entry, but readers must receive the updated document first. Updating only the asset filename without updating its referring document changes nothing for those readers.

Deletion for confidentiality is a different problem. Removing an origin object or purging a CDN cannot revoke bytes already downloaded, and a cache policy is not authorization. Prevent sensitive responses from entering inappropriate caches in the first place. For an urgent repair, define what “visible” means: a fresh visit, an existing tab, or every previously downloaded copy cannot share one guarantee.

Check it quickly

Use Python 3.10 or newer and a POSIX shell. The tested versions are Python 3.14.4, CacheControl 0.14.4, and Requests 2.34.2. Run the Python block with "$invalidation_lab/venv/bin/python" after installing the pinned dependencies. It starts one loopback HTTP server on an allocated port, uses two separate in-memory private caches, disables environment proxies, and closes its resources. Installation alone needs external network access.

invalidation_lab=$(mktemp -d)
python3 -m venv "$invalidation_lab/venv"
"$invalidation_lab/venv/bin/python" -m pip install CacheControl==0.14.4 requests==2.34.2

from collections import Counter
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread

import requests
from cachecontrol import CacheControl
from cachecontrol.cache import DictCache

assets = {"/manual.txt": b"A"}
counts = Counter()


class Origin(BaseHTTPRequestHandler):
 def log_message(self, *_):
 pass

 def do_GET(self):
 counts[self.path] += 1
 body = assets.get(self.path, b"missing")
 status = 200 if self.path in assets else 404
 self.send_response(status)
 self.send_header("Cache-Control", "max-age=600" if status == 200 else "no-store")
 self.send_header("Content-Length", str(len(body)))
 self.end_headers()
 self.wfile.write(body)


server = HTTPServer(("127.0.0.1", 0), Origin)
worker = Thread(target=server.serve_forever)
worker.start()
base = f"http://127.0.0.1:{server.server_port}"
caches = [DictCache(), DictCache()]
clients = []
try:
 for cache in caches:
 client = CacheControl(requests.Session(), cache=cache)
 client.trust_env = False
 clients.append(client)

 def check(index, path, status, body, cached):
 response = clients[index].get(base + path, timeout=3)
 assert (response.status_code, response.content, response.from_cache) == (
 status, body, cached
 )

 for index in range(2):
 check(index, "/manual.txt", 200, b"A", False)
 check(index, "/manual.txt", 200, b"A", True)
 assert counts["/manual.txt"] == 2
 print("warm: two independent caches hold A; origin requests=2")

 assets["/manual.txt"] = b"B"
 caches[0].delete(base + "/manual.txt")
 check(0, "/manual.txt", 200, b"B", False)
 check(1, "/manual.txt", 200, b"A", True)
 assert counts["/manual.txt"] == 3
 print("targeted invalidation: cache 0=B; cache 1=A")

 check(1, "/manual.v2.txt", 404, b"missing", False)
 assets["/manual.v2.txt"] = b"B"
 for index in range(2):
 check(index, "/manual.v2.txt", 200, b"B", False)
 check(index, "/manual.v2.txt", 200, b"B", True)
 assert counts["/manual.v2.txt"] == 3
 check(1, "/manual.txt", 200, b"A", True)
 print("new URL: 404 before publication, then both caches=B")
 print("old URL: cache 1 still holds A")
finally:
 for client in clients:
 client.close()
 server.shutdown()
 worker.join()
 server.server_close()

Both clients first receive A from the origin, then hit their own cache. After the origin changes to B, deleting only cache zero’s exact URL makes that client retrieve B; cache one still returns A without another origin request. The old URL has reached the origin three times, demonstrating scoped removal rather than global convergence.

The new filename initially returns 404 with no-store, exposing incorrect publication order. CacheControl 0.14.4 also excludes 404 from its default cacheable statuses, so this fixture does not isolate no-store’s effect on negative caching. After publication, both clients obtain and cache B under that new URL; cache one still retains A at the original URL. This is a real HTTP and cache-storage experiment, not a CDN purge implementation or browser test. CacheControl is a private-cache library and must not be repurposed as a multi-user shared edge.

Sources

RFC 9111: invalidating stored responses, invalidation scope and unsafe methods, checked 2026-09-07.
CloudFront: invalidation and versioned filenames, edge removal does not clear independent caches, checked 2026-09-07.
CacheControl usage, sessions and separate cache stores, checked 2026-09-07.
CacheControl security model, private-cache restrictions, checked 2026-09-07.
CacheControl 0.14.4 controller, default cacheable statuses and storage checks, checked 2026-10-18.

Connection timeouts prevent silent hangs

Sun, 01 Aug 2027 00:00:00 +0000

A timeout names how long a particular operation may wait. It is not automatically a deadline for the whole request, and it does not prove that the remote operation stopped. Separate connection establishment, individual reads, and the caller’s overall budget before choosing numbers or interpreting a timeout error.

Conceptual timing, not to scale. The numeric-loopback TCP fixture does not bound DNS or TLS. A local deadline does not cancel remote work; complete data still needs validation.

Successful connection does not mean a useful reply

A TCP connection can complete while the application behind it never produces a response. That failure belongs to the read stage, not to an unfinished TCP handshake. Name the stage in diagnostics; otherwise a message such as “database connection timed out” can hide whether the caller was acquiring a pool lease, opening a socket, or waiting for query results.

Python’s socket timeout applies to blocking socket operations. Repeated successful recv() calls can each consume a fresh allowance, so a peer that keeps sending small pieces can extend the whole read loop well beyond that number. In the exercise, each read asks for one byte to make this distinction visible.

For an overall receive budget, create one deadline using time.monotonic(). Before the next operation, compute its remaining duration and choose the smaller of that duration and the operation limit. Do not start another wait after the budget is exhausted. The monotonic clock avoids wall-clock corrections changing a local elapsed-time budget; it does not make thread scheduling instantaneous.

Read the actual client contract before extending this pattern. Name resolution, address retries, TLS negotiation, request writing, and connection-pool acquisition may have separate behavior. This example uses one numeric loopback address and plain TCP. It does not demonstrate bounded DNS resolution or a complete HTTPS client deadline.

When it breaks

A silent accepted peer holds a reader until its read timeout. A trickling peer can keep individual reads successful while making the full response too slow. Setting only the connection timeout handles neither problem after the connection has already succeeded. Conversely, a timeout that is too small can reject useful work during normal scheduling or network variation.

The failure path must close owned resources and preserve the difference between partial and complete data. An empty socket read means EOF, not proof that the application message is complete; this fixture knows the expected six-byte frame and rejects premature closure. A partial response must not silently become a successful result.

Also separate abandoning a wait from canceling remote work. The peer might already have performed a side effect when the caller times out. Retrying needs an appropriate replay contract and a remaining budget, not a fresh unlimited sequence of waits. The local deadline remains a cooperative boundary between operations, with scheduling and cleanup overhead, rather than a hard real-time guarantee.

Check it quickly

Save this as connection_probe.py and run python3 connection_probe.py. It uses only the standard library and was checked with Python 3.14.4 on macOS. Each case creates a task-owned loopback listener and peer thread, then closes the sockets and joins that thread. No external host, firewall rule, or production service is involved.

import socket
import threading
import time


PAYLOAD = b"abcdef"
PER_OPERATION = 0.5


def exercise(trickle, total=None, peer_payload=PAYLOAD):
 stop = threading.Event()
 failures = []
 with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener:
 listener.bind(("127.0.0.1", 0))
 listener.listen(1)
 listener.settimeout(2.0)

 def peer():
 try:
 connection, _ = listener.accept()
 with connection:
 connection.settimeout(0.5)
 if not trickle:
 stop.wait(2.0)
 return
 for byte in peer_payload:
 if stop.wait(0.1):
 return
 connection.sendall(bytes([byte]))
 except (BrokenPipeError, ConnectionResetError):
 pass # The deadline case deliberately closes the client.
 except Exception as error:
 failures.append(error)

 worker = threading.Thread(target=peer)
 worker.start()
 started = time.monotonic()
 deadline = None if total is None else started + total
 received = bytearray()
 phase, outcome = "connect", "complete"

 def allowance():
 if deadline is None:
 return PER_OPERATION
 remaining = deadline - time.monotonic()
 if remaining <= 0:
 raise TimeoutError("total budget expired")
 return min(PER_OPERATION, remaining)

 try:
 with socket.create_connection(listener.getsockname(), timeout=allowance()) as client:
 phase = "read"
 while len(received) < len(PAYLOAD):
 client.settimeout(allowance())
 chunk = client.recv(1)
 if not chunk:
 raise EOFError(f"short frame: {len(received)}/{len(PAYLOAD)} bytes")
 received.extend(chunk)
 except TimeoutError:
 outcome = "total" if deadline is not None and time.monotonic() >= deadline else phase
 finally:
 elapsed = time.monotonic() - started
 stop.set()
 worker.join(2.0)
 assert not worker.is_alive(), "fixture peer failed to stop"
 assert not failures, failures
 return outcome, bytes(received), elapsed


quiet = exercise(trickle=False)
assert quiet[0] == "read" and quiet[1] == b""
trickle = exercise(trickle=True)
assert trickle[0] == "complete" and trickle[1] == PAYLOAD
assert trickle[2] >= 0.5 # Each successful recv got a fresh operation timeout.
bounded = exercise(trickle=True, total=0.25)
assert bounded[0] == "total" and bounded[1] != PAYLOAD
try:
 exercise(trickle=True, peer_payload=b"ab")
except EOFError as error:
 assert str(error) == "short frame: 2/6 bytes"
else:
 raise AssertionError("a truncated frame must not become success")
print("early EOF rejects a two-byte fragment of the six-byte frame")
for name, result in (("quiet", quiet), ("trickle", trickle), ("bounded", bounded)):
 outcome, data, elapsed = result
 print(name, outcome, repr(data), "elapsed", round(elapsed, 3))

The quiet case connects successfully but receives nothing, then reports a read timeout around half a second. The trickle case sends six bytes with a pause before each one. All reads finish within their individual allowance, yet the complete frame takes longer than the configured half-second operation timeout. “Complete” here means all six expected bytes arrived, not merely that one read succeeded.

With the quarter-second overall budget, the same trickle cannot complete the frame. The output reports total expiration and partial bytes; the exact byte count and elapsed values depend on scheduling. Assertions deliberately check the outcome and completeness instead of promising exact millisecond timing. A fourth peer sends two bytes and closes; the reader raises EOFError instead of accepting that fragment as a complete frame.

The stop event is explicit test coordination that releases the local peer during cleanup. It is not cancellation transmitted by TCP to an arbitrary remote operation. These results validate the difference between per-operation waits and a shared receive budget, not connection blackholes, TLS failures, retries, or production tail latency.

Sources

Python socket, blocking-operation timeouts, connection establishment and socket reads, checked 2026-09-09.
Python time, monotonic elapsed-time measurement, checked 2026-09-09.

A rota needs continuous coverage and a load review

Fri, 16 Jul 2027 00:00:00 +0000

A rota can distribute responsibility, but equal shift counts do not guarantee equal burden. Review nighttime pages, incident effort, recovery time, and reliable backup alongside the schedule.

A job that is shared

On-call work is a job that belongs to a team, not to one hero. A rotation turns ‘whoever was last awake’ into a defined roster and a schedule, so responsibility is explicit and no one is assumed to carry the whole queue indefinitely. The schedule itself — a weekly handoff, a backup/secondary on-call, a follow-the-sun structure across sites — is how you keep the obligation sustainable without letting it turn into a possession. Don’t let the urgency of tonight’s page decide who owns tomorrow; the rotation decides on purpose, ahead of time.

Handoff is half the craft

Handoff is where a rotation most often breaks. When the incoming person inherits a blank slate, what should be a handoff becomes a re-investigation. Two habits close that gap: a written handoff (what is broken, what has been tried, what the next person should own) and an overlap window so the incoming engineer is shadowed by the outgoing one before they leave. A rotation that swaps names but not context turns the same incident into a fresh one every week. The craft is not the schedule; it is the continuity the schedule protects.

Rehearse before the bell rings

The time to rehearse is before the pager rings, not during. Run the runbook in quiet hours, drill the restore path (not just the diagnosis), and map each alert to a decision tree so a cold Sunday-morning page is a checklist rather than a riddle. Rehearsal improves the team’s options under pressure: the cheap, repeatable steps taken calmly in daylight are what keep the team from improvising during the real incident. That rehearsal is the difference between ‘we will figure it out’ and a boring, repeatable process.

When it breaks

Single-person coverage looks simple until it breaks the person. One engineer holding the pager for months stops sleeping well, dreads every notification, and eventually burns out or quits; Google’s SRE guidance emphasizes sustainable workload and staffing. The fix is a bounded rotation with a maximum tour length and a real off-call week where the previous primary is not the silent backup.

A broken escalation chain fails differently. If the secondary never acknowledges, contacts are stale, or paging the manager reaches a mailbox nobody reads, a 3 a.m. incident waits while the primary debugs alone. Rehearse the chain like a restore: page the secondary in business hours and confirm a human answers within minutes.

Alert fatigue is the third failure. Hundreds of noisy, unactionable alerts teach the team the pager lies, so the one real page is swiped away with the rest. Every alert needs an owner, a runbook link, and a threshold tuned after the last false positive.

Missing handoffs complete the set. The name changes on Monday with no written state, so Tuesday repeats Monday’s investigation from scratch. Require a short note — symptoms, tried steps, next move — plus a fifteen-minute overlap before the old primary goes dark.

Keep both roles covered through every week. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

Use this paper-model exercise before publishing a schedule. It assigns fictional primary and secondary responders for twenty consecutive weeks. Two invented special-coverage weeks remain staffed; they are not assertions about any country’s holidays. A real continuously operated service still needs coverage when an engineer is on leave.

Save the snippet as rota_probe.py in an empty directory.
Run python3 rota_probe.py with Python 3.9 or newer; it touches no network or files. Run without -O, -OO, or a nonzero PYTHONOPTIMIZE setting so the assertions execute.
Check the weekly assignments, role counts, and final PASS line.

from collections import Counter
from datetime import date, timedelta

TEAM = ["ana", "ben", "cara", "dev", "elin"] # fictional names
START = date(2027, 8, 2)
WEEKS = 20
SPECIAL_WEEKS = {START + timedelta(weeks=1), START + timedelta(weeks=4)}
rota = [(START + timedelta(weeks=i), TEAM[i % 5], TEAM[(i + 1) % 5])
 for i in range(WEEKS)]
assert len(rota) == WEEKS
assert len({week for week, _, _ in rota}) == WEEKS
assert all(primary != secondary for _, primary, secondary in rota)
assert all(primary in TEAM and secondary in TEAM for _, primary, secondary in rota)
assert all(b[0] - a[0] == timedelta(days=7) for a, b in zip(rota, rota[1:]))
assert all(a[1] != b[1] for a, b in zip(rota, rota[1:]))
primary_counts = Counter(primary for _, primary, _ in rota)
secondary_counts = Counter(secondary for _, _, secondary in rota)
assert set(primary_counts.values()) == {4}
assert set(secondary_counts.values()) == {4}
for week, primary, secondary in rota:
 marker = " special-coverage" if week in SPECIAL_WEEKS else ""
 print(f"{week}: primary={primary} secondary={secondary}{marker}")
assert sum(week in SPECIAL_WEEKS for week, _, _ in rota) == 2
print("primary:", dict(sorted(primary_counts.items())))
print("secondary:", dict(sorted(secondary_counts.items())))
print("PASS: 20 covered weeks, distinct backup, four turns per role; special weeks remain staffed")

Every week has distinct primary and secondary names, including both marked weeks. Each engineer receives four turns in each role, and adjacent weeks have different primaries. These assertions passed in Python 3.12.14 during this review. They establish the stated roster properties only; equal counts do not measure equal overnight interruptions, overlapping incidents, availability, time zones, compensation, or recovery time. Obtain team agreement on those constraints and test the actual escalation contacts before activating a real rota.

Sources

SRE Workbook: on-call, structuring fair on-call rotations, checked 2026-10-15.
SRE book: being on-call, on-call practice and limits, checked 2026-10-15.

Probes tell the platform when to act

Fri, 18 Jun 2027 00:00:00 +0000

A probe result is evidence consumed by a control loop. Kubernetes does not infer your application’s recovery strategy from a route named healthz. The probe type, result, configured thresholds, and container lifecycle determine what happens next. Treating every failure as “restart the Pod” erases distinctions that directly affect availability.

Three questions, different actions

A startup probe gives initialization its own detection policy. When configured, it prevents liveness and readiness probes from running until it succeeds. It stops checking after success for that container lifetime; a newly started container needs to pass startup again.

Liveness is intended to identify conditions for which restarting the affected container can help. After the failure threshold, the kubelet kills that container and applies its restart policy. This is not a promise to replace the whole Pod, change its identity, or repair an unavailable external database.

Readiness controls whether the container is considered ready. A failed readiness check leaves the container running and continues checking for recovery. It affects ordinary readiness-aware Service routing, but is not a firewall and does not cancel requests already using a connection.

Thresholds operate on consecutive observations

A failureThreshold of three is not a three-failures-ever counter. A success breaks the run. Readiness can require multiple consecutive successes to recover; successThreshold must be one for startup and liveness. Know both directions of a transition before interpreting a short trace.

Timeouts, initial delays, periods, and termination grace also matter. Threshold multiplied by period is useful for planning, not an exact end-to-end failover stopwatch. Scheduling, probe execution, and routing propagation intervene. Readiness may be checked more frequently than its configured period while a container is not ready.

The experiment uses four startup failures, three liveness failures, and two readiness failures, with two readiness successes required for recovery. These settings make counters observable; they are not production recommendations.

Match the mechanism to the evidence

An HTTP check uses the documented 200–399 success range. A 200 response containing an error message is not a failed HTTP probe merely because a human dislikes its body. Verify redirects and authentication on the actual probe route rather than assuming the same behavior as an interactive browser.

A TCP check shows that a connection can be opened, not that a query or transaction succeeds. An exec check needs a command present in the container and a meaningful exit status; frequent process creation also costs resources. Select the cheapest mechanism that answers the required question.

Readiness is not a prerequisite for liveness. Without startup gating, a long initialization period can repeatedly fail liveness before the application has time to become ready. Conversely, an excessively generous startup allowance can postpone detection of a genuinely broken launch.

Probe failure and an Unknown result are distinct. Kubernetes documents Unknown as leaving container state unchanged while further checks occur. This binary teaching model intentionally has no third outcome, so it cannot predict transport-error classification. Use kubelet events and the deployed probe handler to establish how a real error was classified.

When it breaks

Using external dependency availability as liveness can restart healthy processes throughout a shared outage. An always-green handler on an unrelated thread can miss a deadlocked business worker. Restarts are not automatically cheap: they can lose in-flight work and repeat expensive initialization.

A readiness gate covering every optional dependency can withdraw useful capacity. Rapid state flapping can transfer load between replicas and worsen saturation. Check false positives and recovery behavior, not just whether a deliberately broken endpoint turns red.

Compare probe outcomes with container restart counts, events, readiness transitions, and actual client traffic. Confirm the deployed configuration and routing consumer. A successful local policy model cannot establish that a real kubelet, network path, or load balancer applies it correctly.

Three probe responsibilities: mechanism and evidence boundaries.

Check it quickly

Save this block as probe_policy_check.py and run python3 probe_policy_check.py with Python 3.10 or newer. Retained records report Python 3.12.14; this review did not rerun it. Keep assertions enabled, without -O or PYTHONOPTIMIZE, because some sample calls occur inside assert. This deterministic model consumes binary observations; it does not deploy a Pod, call Kubernetes, schedule probes, or restart an operating-system process.

class ProbePolicy:
 # (consecutive successes, consecutive failures), chosen for this experiment.
 thresholds = {"startup": (1, 4), "liveness": (1, 3), "readiness": (2, 2)}

 def __init__(self):
 self.restart_requests = 0
 self.start_container()

 def start_container(self):
 self.started = self.ready = self.restart_requested = False
 self.successes = dict.fromkeys(self.thresholds, 0)
 self.failures = dict.fromkeys(self.thresholds, 0)

 def sample(self, kind, passed):
 if self.restart_requested:
 return "awaiting-restart"
 if kind != "startup" and not self.started:
 return "suppressed"
 if kind == "startup" and self.started:
 return "disabled"
 success_limit, failure_limit = self.thresholds[kind]
 self.successes[kind] = self.successes[kind] + 1 if passed else 0
 self.failures[kind] = 0 if passed else self.failures[kind] + 1
 if passed and self.successes[kind] >= success_limit:
 if kind == "startup":
 self.started = True
 elif kind == "readiness":
 self.ready = True
 if not passed and self.failures[kind] >= failure_limit:
 self.ready = False
 if kind != "readiness":
 self.restart_requested = True
 self.restart_requests += 1
 return "restart-requested" # A model output, not a kubelet or OS action.
 return "sampled"


policy = ProbePolicy()
for _ in range(3):
 assert policy.sample("startup", False) == "sampled"
 assert policy.sample("liveness", False) == "suppressed"
 assert policy.sample("readiness", True) == "suppressed"
assert policy.restart_requests == 0 and not policy.ready
assert policy.failures["liveness"] == 0 and policy.successes["readiness"] == 0
policy.sample("startup", True)
assert policy.started and not policy.ready
assert policy.sample("startup", False) == "disabled"
print("three startup failures do not trigger liveness; startup success enables the other probes")

policy.sample("readiness", True)
assert not policy.ready
policy.sample("readiness", True)
assert policy.ready
policy.sample("readiness", False)
assert policy.ready
policy.sample("readiness", True) # Breaks the consecutive-failure run.
policy.sample("readiness", False)
assert policy.ready
policy.sample("readiness", False)
assert not policy.ready and policy.restart_requests == 0
for result in (True, False, True):
 policy.sample("readiness", result)
assert not policy.ready
policy.sample("readiness", True)
assert policy.ready
print("two readiness failures remove readiness without restart; two consecutive successes restore it")

for result in (False, False, True, False, False):
 policy.sample("liveness", result)
assert policy.restart_requests == 0
assert policy.sample("liveness", False) == "restart-requested"
assert policy.restart_requests == 1 and not policy.ready
assert policy.sample("liveness", False) == "awaiting-restart"
assert policy.restart_requests == 1
policy.start_container() # Explicitly model the beginning of another container lifetime.
assert not policy.started and not policy.ready
for _ in range(4):
 policy.sample("startup", False)
assert policy.restart_requests == 2
print("three consecutive liveness failures request a restart; a fresh lifetime runs startup checks again")
print("four startup failures request another restart; no real container, clock, or traffic route was changed")

Three startup failures remain below the fixture’s threshold, and attempted liveness/readiness observations are suppressed. Startup success enables those checks without automatically declaring readiness.

Two consecutive readiness failures remove readiness without requesting a restart; recovery needs two consecutive successes. Intervening opposite outcomes reset the relevant streak. Three consecutive liveness failures produce one restart request, not repeated requests while awaiting restart.

An explicitly modeled new container lifetime clears startup state. Four startup failures then request another restart. Unknown results, transport error classification, real timing, restart policy execution, termination grace, and endpoint propagation remain untested; validate those on the intended cluster before relying on the full operational behavior.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
Kubernetes: Liveness, readiness, and startup probes, probe actions, mechanisms, thresholds, and startup gating, checked 2026-10-15.
Kubernetes: Configure probes, executable probe configuration and observed container restarts, checked 2026-10-15.
Kubernetes: Pod lifecycle, container restart policy and termination versus traffic lifecycle, checked 2026-10-15.

Resource limits constrain execution; requests guide placement

Mon, 14 Jun 2027 00:00:00 +0000

A Kubernetes resource request and a resource limit belong to different decisions. Requests participate in scheduling and resource sharing; limits configure execution constraints supported by the runtime and operating system. Treating a limit as the scheduler’s reservation obscures both mechanisms. A useful resource policy identifies placement needs, acceptable contention and the failure behavior when a running workload reaches its boundary.

The diagram separates placement, accounting, termination, restart policy and Pod eviction. The idle neighbor was Ready at one final historical observation; no neighbor-latency or full-isolation guarantee follows. No example was rerun.

CPU and memory fail differently

On a typical Linux setup, a CPU limit becomes a bandwidth constraint: a busy group can be throttled while its processes remain alive. A memory limit uses memory accounting and reclaim, with cgroup OOM handling when the constrained allocation cannot be satisfied. The kernel does not gently slow every memory allocation in the same way that CPU bandwidth can be delayed.

Memory enforcement is reactive. Do not describe it as an application callback fired at an exact byte boundary or as a guarantee that usage can never transiently exceed a configured number. The application might receive an allocation failure, or OOM handling can terminate a process. Observe the actual termination reason, container state and available memory events before naming the cause.

The memory limit covers more than the application’s intended payload. Runtime overhead, additional processes and charged cache or other memory can consume the group’s allowance. A program with a sixty-four-MiB limit cannot assume it can safely allocate a sixty-four-MiB data structure on top of everything else. Memory-backed volumes can contribute to memory consumption too; their storage appearance does not make that cost disappear.

When it breaks

An exit code of 137 indicates a SIGKILL-style termination convention; it is not sufficient evidence of OOM by itself. Correlate it with the container’s OOMKilled reason and relevant runtime or kernel evidence. A node-pressure eviction is another mechanism, with Pod-level status and eviction decisions that should not be conflated with one container exceeding its cgroup budget.

Restart policy determines what follows termination. A repeatedly leaking application under Always policy can enter a restart cycle while the underlying defect remains. With Never policy, the terminated container stays terminated and the Pod can fail. The limit contains one resource dimension; it neither repairs the application nor guarantees that neighboring workloads experience no contention before the failure.

Defaulting also matters. If a resource has a limit but no request and no admission mechanism supplies a different default, Kubernetes copies that limit into the request. Review the admitted object instead of assuming omitted requests mean zero. Matching memory requests and limits alone is not enough to establish the conventional Guaranteed QoS requirements; CPU settings and all relevant containers matter as well.

Use explicit units such as Mi and Gi for memory. Lowercase m denotes a milli-unit, not megabytes. Base limits on measured working sets and bursts, accounting for startup, concurrency and runtime overhead. A copied ratio or fixed percentile cannot choose acceptable OOM behavior for every application.

Check it quickly

The unchanged cluster experiment was not rerun in this review; the Kubernetes 1.34.11 observations below remain historical.

Use a disposable Linux Kubernetes cluster with cgroup v2, kubectl access and python:3.11-slim-bookworm on its nodes. Save this as memory-limit.py and run python3 memory-limit.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11. The fixture bounds its attempted payload to 128 MiB and places a 64 MiB limit on the allocating container.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-memory-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 allocator = '''from pathlib import Path
import json,time
group=Path('/sys/fs/cgroup')
print(json.dumps({'memory_max':(group/'memory.max').read_text().strip(),
 'initial_memory_current':int((group/'memory.current').read_text())}),flush=True)
while not Path('/tmp/go').exists(): time.sleep(0.1)
chunks=[]
for i in range(16):
 chunks.append(bytearray(8*1024*1024))
 print('allocated_chunks='+str(len(chunks)),flush=True)
 time.sleep(0.05)
raise RuntimeError('bounded allocation unexpectedly escaped the expected limit')
'''
 constrained = spec(allocator)
 constrained['restartPolicy'] = 'Never'
 constrained['containers'][0]['resources'] = {
 'requests': {'cpu': '10m', 'memory': '32Mi'}, 'limits': {'cpu': '200m', 'memory': '64Mi'}}
 for name, podspec in [('limited', constrained), ('neighbor', spec())]:
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': name}, 'spec': podspec})
 wait_for(lambda name=name: obj(namespace, 'pod', name), pod_ready)
 first_log = wait_for(lambda: k('logs', '-n', namespace, 'limited').stdout.strip(), bool)
 before = json.loads(first_log)
 assert before['memory_max'] == str(64 * 1024 * 1024)
 k('exec', '-n', namespace, 'limited', '--', 'python3', '-c',
 'from pathlib import Path; Path("/tmp/go").touch()')
 failed = wait_for(lambda: obj(namespace, 'pod', 'limited'), lambda p: p.get('status', {}).get('phase') == 'Failed')
 status = failed['status']['containerStatuses'][0]
 terminated = status['state']['terminated']
 assert terminated['reason'] == 'OOMKilled' and terminated['exitCode'] == 137
 assert status['restartCount'] == 0
 assert failed['status'].get('reason') != 'Evicted'
 neighbor = obj(namespace, 'pod', 'neighbor')
 assert pod_ready(neighbor) and neighbor['status']['containerStatuses'][0]['restartCount'] == 0
 output = k('logs', '-n', namespace, 'limited').stdout.splitlines()
 chunks = [int(line.split('=')[1]) for line in output if line.startswith('allocated_chunks=')]
 assert chunks and max(chunks) < 16
 print(json.dumps({'before': before, 'last_reported_allocated_chunks': max(chunks),
 'termination_reason': terminated['reason'], 'exit_code': terminated['exitCode'],
 'container_restarts_with_never_policy': 0,
 'pod_was_not_evicted': True, 'neighbor_still_ready': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The first log records memory.max as 67108864 bytes. After a release file appears, the original container process allocates eight-MiB chunks. The recorded run reported six completed chunks before termination. Its status shows OOMKilled and exit 137, while the Pod is Failed rather than Evicted. Never policy leaves its restart count at zero.

A separate idle test Pod remains Ready with no restart at the final observation. All assertions passed and the namespace was removed. This verifies one constrained OOM event and its lifecycle result; it does not benchmark neighbor latency or establish full node isolation. Use the failure evidence to size and diagnose the workload, rather than treating a configured limit as a complete availability guarantee.

Sources

Kubernetes: Resource Management, requests, limits, defaulting and Linux enforcement, checked 2026-10-15.
Kubernetes: Assign Memory Resources, memory limits and observed OOM termination, checked 2026-10-15.
Linux: Control Group v2, memory.max, reclaim and cgroup OOM behavior, checked 2026-10-15.

Load shedding protects the system

Sun, 16 May 2027 00:00:00 +0000

Load shedding deliberately refuses some work when accepting everything would threaten useful service. The aim is to preserve capacity for work that can still complete, not to promise that every accepted request will be fast or important. Rejection is itself work, so the admission point and its cost matter as much as the threshold.

Protect useful capacity

An overloaded queue consumes memory and adds waiting time. Clients can abandon requests before workers start them, leaving the server to spend resources on answers nobody needs. Longer queues can absorb some bursts, but cannot create processing capacity under sustained excess demand.

Google SRE describes queue limits, resource-pressure signals, selective rejection, and reduced-quality responses. Shedding and degradation are related but different: refusing a request is not the same as completing a cheaper version of it. The latter needs an explicit product contract about the result that remains useful.

Protection can also target different components. Envoy’s overload manager protects the proxy’s own resources, while its circuit-breaking mechanisms primarily protect upstream services. A dependency breaker does not automatically prevent local memory, connection, or file-descriptor exhaustion.

Choose a justified policy

Priority-aware shedding can protect critical business paths, but priority is not determined by HTTP method or retryability alone. A read can be essential, while a write can be expendable telemetry. Use trusted application classification rather than letting every caller declare its own request critical.

Uniform admission limits or proportion-based rejection can also be legitimate policies. They are not inherently incorrect because they lack business priority. State the fairness and capacity tradeoffs, including whether lower-priority work can starve and whether different tenants share the same limit.

Count, bytes, execution time, and CPU cost are different budgets. A cap on three tasks does not bound memory when each task can retain an arbitrarily large payload. Select signals that correspond to the resource being protected and measure their behavior under realistic request mixes.

Bound admission, not only workers

A thread-pool worker limit bounds simultaneous execution, not necessarily accepted queued work. A semaphore acquired only inside a worker leaves the queue in front of it unbounded. Make a bounded admission decision before submitting expensive work, and return that capacity when its lifecycle actually ends.

Even similarly named proxy limits need precise interpretation. NGINX limit_conn counts requests under its documented connection rules; with HTTP/2 and HTTP/3, each concurrent request counts separately. Do not assume a configured connection cap always means a count of TCP sockets.

The fixture below bounds running plus queued tasks. Refused submissions create no future. A completion callback returns permits after success, failure, or cancellation before execution; a rejected submission to a closed executor also returns its unused permit.

When it breaks

Rejecting after parsing huge bodies, opening database connections, or performing a business write may be too late to protect the intended resource. Expensive logging or response generation can itself overload the rejection path. Observe rejection cost alongside accepted throughput, queue age, active work, and resource pressure.

HTTP 503 can signal temporary service unavailability, while 429 concerns excessive request rates under a caller policy. Retry-After is a wait signal, not proof that replaying a write is safe or that capacity will exist later. Immediate retries can restore the very load that was shed.

Do not reclaim a running task’s permit merely because its caller stopped waiting. A future-result timeout does not terminate the worker. Premature release allows replacement work to overlap the supposedly removed work and breaks the capacity bound.

Three permits bound queued plus running work before submission; nine of twelve tasks are refused. Permits track actual lifecycle. The uniform gate does not implement priority or deadline scheduling.

Check it quickly

Save the block as shedding_check.py and run python3 shedding_check.py with Python 3.10 or newer. It ran with Python 3.14.4 using only the standard library. Real worker threads wait on test coordination events; no external service is called, and no production load is generated. The fixture compares identical twelve-task bursts.

from concurrent.futures import ThreadPoolExecutor
from threading import Barrier, BoundedSemaphore, Event, Lock


def exercise(gated):
 slots = BoundedSemaphore(3) # Bounds queued plus running work, not just threads.
 started, release, lock = Barrier(3), Event(), Lock()
 running, began = set(), []

 def held(job):
 with lock:
 running.add(job)
 began.append(job)
 try:
 if job < 2:
 started.wait(timeout=5)
 assert release.wait(timeout=5), "test coordinator did not release workers"
 return job
 finally:
 with lock:
 running.remove(job)

 with ThreadPoolExecutor(max_workers=2) as pool:
 def submit(function, *args):
 if not gated:
 return pool.submit(function, *args)
 if not slots.acquire(blocking=False):
 return None # Explicit local admission refusal; no future was enqueued.
 try:
 future = pool.submit(function, *args)
 except RuntimeError:
 slots.release()
 raise
 # Completion, failure, and cancellation before execution all return the slot.
 future.add_done_callback(lambda _: slots.release())
 return future

 futures = [submit(held, job) for job in range(12)]
 accepted = [future for future in futures if future is not None]
 replacement = None
 try:
 started.wait(timeout=5)
 with lock:
 assert running == {0, 1} and set(began) == {0, 1}
 assert all(not future.done() for future in accepted)
 if gated:
 assert len(accepted) == 3 and futures[3:] == [None] * 9
 assert futures[2].cancel() # Both workers are held, so job 2 has not started.
 replacement = submit(held, 12)
 assert replacement is not None
 assert submit(held, 13) is None
 print("gated burst: 2 running + 1 queued, 9 initial refusals; cancellation returns a slot")
 else:
 assert len(accepted) == 12
 print("ungated burst: 2 workers accept 12 unfinished tasks; the other 10 still wait")
 finally:
 release.set()
 completed = [future.result(timeout=5) for future in accepted if not future.cancelled()]
 if replacement is not None:
 completed.append(replacement.result(timeout=5))
 assert sorted(completed) == ([0, 1, 12] if gated else list(range(12)))

 if gated:
 def fail():
 raise ValueError("injected task failure")

 failed = submit(fail)
 assert failed is not None
 try:
 failed.result(timeout=5)
 except ValueError as error:
 assert str(error) == "injected task failure"
 else:
 raise AssertionError("the failed task must preserve its failure")

 if gated:
 # The executor is now closed; a failed submission must not consume a slot.
 try:
 submit(lambda: None)
 except RuntimeError:
 pass
 else:
 raise AssertionError("submission to a closed executor must fail")
 acquired = [slots.acquire(blocking=False) for _ in range(3)]
 try:
 assert acquired == [True, True, True]
 assert not slots.acquire(blocking=False)
 finally:
 for success in acquired:
 if success:
 slots.release()
 print("completion, task failure, cancellation, and closed-executor submission leave all 3 slots reusable")


exercise(False)
exercise(True)

With two workers but no admission gate, twelve tasks remain unfinished: two running and ten waiting. With three permits, only three are accepted and nine are refused before submission. The admitted set contains two running tasks and one queued task, not three simultaneous workers.

Canceling the queued task returns a permit for one replacement, while another submission is refused. After coordination releases the workers, the expected tasks complete. An injected task failure and a submission after executor shutdown both preserve their errors. Public semaphore operations finally confirm that all three permits are reusable.

This tests admission accounting and task lifecycle, not HTTP responses, byte limits, priority scheduling, production latency, or an automatic resource-pressure controller. NGINX and Envoy are cited for their contracts, not claimed to be deployed or exercised here.

Sources

Google SRE: Handling overload, criticality, resource signals, rejection costs, and client behavior, checked 2026-09-08.
Google SRE: Addressing cascading failures, queue limits, load shedding, deadlines, and degradation, checked 2026-09-08.
Envoy overload manager, local resource protection versus upstream circuit breaking, checked 2026-09-08.
NGINX connection limiting, counted requests, HTTP/2–HTTP/3 semantics, and rejection status, checked 2026-09-08.
Python futures, executor submission, result timeouts, cancellation, and completion callbacks, checked 2026-09-08.
Python threading, nonblocking bounded-semaphore admission and coordination primitives, checked 2026-09-08.
RFC 9110, service-unavailability, Retry-After, and retry safety semantics, checked 2026-09-08.
RFC 6585, section 4’s separate definition of 429 and server-selected counting policy, checked 2026-09-08.

Retry-After tells when to try again

Tue, 04 May 2027 00:00:00 +0000

Retry-After tells a client how long it ought to wait before a follow-up request. It is a scheduling signal, not proof that the next attempt will succeed or that replaying the original operation is safe. A client still needs a deadline, an attempt policy, and a decision about which operations it can retry.

Interpret the value and its context

RFC 9110 permits either a nonnegative decimal integer number of seconds or an HTTP date. Delay-seconds are measured after receiving the response, not after sending the original request. An HTTP date names an absolute UTC time, so interpreting it depends on the client’s wall clock being reasonably accurate.

Senders generate the preferred IMF-fixdate form. Recipients parsing HTTP dates must accept the two obsolete forms as well; the two-digit year in the old RFC 850 form has a fifty-year interpretation rule. Negative numbers, fractional seconds, and arbitrary words are not valid delay-seconds. Zero and a date already in the past do not require additional server-directed waiting.

With 503, the field estimates how long the service is expected to be unavailable. With a 3xx response, it asks for a minimum wait before the redirected request. RFC 6585 separately permits Retry-After on 429 responses. It is optional there, not a header every throttling response is required to contain.

Respect the wait without waiting forever

A valid wait can exceed the caller’s remaining deadline. The client can stop retrying and report failure or arrange a separate authorized workflow. It should not shorten a server-requested wait and retry early merely to fit a convenient local cap. A long value is not automatically malformed.

After converting an absolute date into a delay, schedule and account for elapsed time with a monotonic clock. Recheck the deadline before the actual attempt, reserving time for that attempt to finish. Clock skew, response transit time, and scheduling delays are reasons to distinguish a plan from observed retry timing.

When the field is absent or unusable, a bounded client policy can use its own backoff. That is a client decision, not a server-supplied recovery estimate. Keep maximum attempts and total elapsed time explicit so that an optional field does not turn transient failure into unlimited retries.

The same delay can synchronize callers

If thousands of clients receive the same delay at about the same time, they can return together. Retry-After alone does not spread them across a recovery window. Add nonnegative jitter after respecting the requested delay, or otherwise schedule no earlier than that boundary.

Combining local backoff and server guidance requires preserving both constraints. The example takes the larger of its local backoff and parsed server delay, then adds positive jitter. Its injected offsets demonstrate the scheduling relationship; they are not measurements of real distributed traffic or a recommendation for a particular production jitter range.

When it breaks

Blindly replaying a write can duplicate side effects when the original outcome was uncertain. A 429 or 503 response and a Retry-After field do not grant an idempotency guarantee. Use the application’s retry contract, including any operation identifiers or deduplication rules, before repeating nonidempotent work.

Libraries also differ in automatic retries and redirect handling. Verify which statuses, methods, and fields the actual client honors. The fixture’s policy retries only GET and HEAD after 429 or 503; it deliberately does not follow redirects or replay POST. That is a bounded example policy, not the full set of methods HTTP considers idempotent.

Malformed values, huge waits, stale dates, exhausted deadlines, and attempt limits belong in tests. Observe actual attempts and delay rather than assuming that configuring a header caused clients to obey it. The service can remain unavailable after its estimate, so retries still need a terminal outcome.

Wait at least as long as both server guidance and local backoff require, then add nonnegative jitter. These are injected scheduling plans, not observed timed retries or a replay-safety guarantee.

Check it quickly

Save the block as retry_check.py and run python3 retry_check.py with Python 3.10 or newer. Only the standard library is required. The script makes real requests to its own loopback HTTP server, uses a fixed fixture date and injected clocks, then stops that server. It does not sleep until a retry or contact an external service.

Validation (2026-10-17): The unchanged exercise passed on Python 3.14.4 (macOS arm64), with assertions enabled and loopback-only guards. The scope limits below still apply.

from datetime import datetime, timedelta, timezone
from email.utils import format_datetime, parsedate_tz
from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, HTTPServer
import re
from threading import Thread


UTC = timezone.utc
REFERENCE = datetime(2026, 9, 8, 12, 0, 0, tzinfo=UTC)


def delay_seconds(value, received_wall):
 if value is None:
 return None
 value = value.strip(" \t")
 if re.fullmatch(r"[0-9]+", value):
 # Huge decimal values become infinity and fail the bounded retry budget.
 return float(value)
 parts = parsedate_tz(value)
 if parts is None:
 return None
 year, month, day, hour, minute, second = parts[:6]
 old_date = re.fullmatch(r"[A-Za-z]+, [0-9]{2}-[A-Za-z]{3}-([0-9]{2}) [0-9:]{8} GMT", value)
 if old_date:
 now = datetime.fromtimestamp(received_wall, UTC)
 year = now.year // 100 * 100 + int(old_date[1])
 if (year, month, day, hour, minute, second) > (
 now.year + 50, now.month, now.day, now.hour, now.minute, now.second
 ):
 year -= 100
 if not 0 <= second <= 60:
 return None
 try:
 stamp = datetime(year, month, day, hour, minute, min(second, 59), tzinfo=UTC)
 stamp += timedelta(seconds=(second == 60) - parts[9])
 except (ValueError, OverflowError):
 return None
 return max(0.0, stamp.timestamp() - received_wall)


def retry_at(status, method, value, received_wall, monotonic_now, remaining, attempt, jitter=0.25):
 assert attempt >= 1 and jitter >= 0
 if status not in (429, 503) or method not in ("GET", "HEAD") or attempt >= 3:
 return None
 server_delay = delay_seconds(value, received_wall)
 wait = max(server_delay or 0.0, 2 ** (attempt - 1)) + jitter
 if wait >= remaining:
 return None
 return monotonic_now + wait


wall = REFERENCE.timestamp()
for value in ("Tue, 08 Sep 2026 12:00:07 GMT", "Tuesday, 08-Sep-26 12:00:07 GMT",
 "Tue Sep 8 12:00:07 2026"):
 assert delay_seconds(value, wall) == 7
assert delay_seconds("Sunday, 06-Nov-94 08:49:37 GMT", wall) == 0
assert delay_seconds("Monday, 08-Sep-70 12:00:00 GMT", wall) == (
 datetime(2070, 9, 8, 12, tzinfo=UTC).timestamp() - wall
)
assert delay_seconds("Sat, 31 Dec 2016 23:59:60 GMT",
 datetime(2016, 12, 31, 23, 59, 59, tzinfo=UTC).timestamp()) == 1
for invalid in ("-1", "1.5", "later", "", "٣", "Tue, 31 Feb 2026 12:00:00 GMT"):
 assert delay_seconds(invalid, wall) is None
assert delay_seconds(" 0003\t", wall) == 3
assert retry_at(503, "GET", "9" * 5000, wall, 100, 10, 1) is None
assert retry_at(503, "GET", "3", wall, 100, 2, 1) is None
assert retry_at(503, "POST", "3", wall, 100, 10, 1) is None
assert retry_at(503, "GET", "3", wall, 100, 10, 3) is None
print("both delay forms, obsolete dates, year interpretation, and invalid values are checked")
print("an oversized wait, exhausted deadline, POST, or attempt cap stops this retry policy")

cases = {
 "/seconds": (429, "3"),
 "/date": (503, format_datetime(REFERENCE + timedelta(seconds=7), usegmt=True)),
 "/invalid": (503, "later"),
 "/missing": (503, None),
 "/redirect": (307, "3"),
 "/ready": (200, None),
}
visited = []


class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 visited.append(self.path)
 status, value = cases.get(self.path, (404, None))
 self.send_response_only(status)
 self.send_header("Date", format_datetime(REFERENCE, usegmt=True))
 if value is not None:
 self.send_header("Retry-After", value)
 if status == 307:
 self.send_header("Location", "/ready")
 self.send_header("Content-Length", "0")
 self.end_headers()

 def log_message(self, *_):
 pass


with HTTPServer(("127.0.0.1", 0), Handler) as server:
 thread = Thread(target=server.serve_forever, kwargs={"poll_interval": 0.05})
 thread.start()
 try:
 for route, expected in (("/seconds", 103.25), ("/date", 107.25),
 ("/invalid", 101.25), ("/missing", 101.25), ("/redirect", None)):
 client = HTTPConnection(*server.server_address, timeout=2)
 try:
 client.request("GET", route)
 response = client.getresponse()
 response.read()
 planned = retry_at(response.status, "GET", response.getheader("Retry-After"),
 wall, 100, 10, 1)
 assert planned == expected
 print(route, "status", response.status, "planned monotonic time", planned)
 finally:
 client.close()
 assert "/ready" not in visited
 finally:
 server.shutdown()
 thread.join(timeout=2)
 assert not thread.is_alive()

same_delay = [retry_at(429, "GET", "3", wall, 100, 10, 1, jitter=0) for _ in range(4)]
spread = [retry_at(429, "GET", "3", wall, 100, 10, 1, jitter=j) for j in (0.1, 0.2, 0.3, 0.4)]
assert same_delay == [103] * 4
assert len(set(spread)) == 4 and min(spread) > 103
print("identical server delays synchronize plans; injected positive jitter spreads them:", spread)

Parser checks cover the listed dates and invalid values. email.utils also accepts non-HTTP date syntax: this is a permissive scheduling demonstration, not a strict RFC grammar validator. The bounded policy abandons a wait exceeding its deadline, POST replay, and attempts beyond its cap.

The local 429 with three seconds plans time 103.25 from injected monotonic time 100. The dated 503 plans 107.25. Missing or invalid fields use the stated local backoff, while the redirect is not followed. Identical delays produce four plans at 103; supplied positive jitter produces 103.1 through 103.4.

These are response and scheduling-decision tests. Actual timed retries, clock corrections, network partitions, proxy behavior, production jitter distributions, and business-operation deduplication remain untested.

Sources

RFC 9110, sections 5.6.7 and 10.2.3 define HTTP dates and Retry-After, checked 2026-10-15.
RFC 6585, optional Retry-After with 429 responses, checked 2026-10-15.
Python email utilities, date parsing, UTC offsets, and formatted timestamps, checked 2026-10-15.

Graceful degradation keeps partial value

Thu, 22 Apr 2027 00:00:00 +0000

Graceful degradation preserves a deliberately supported subset of a service when another part fails. An article page might omit recommendations while still showing the requested article. The important word is supported: a partial answer must remain useful and truthful under its product contract. Returning something is not automatically safer than returning an explicit failure.

Choose what may be missing

Identify optional regions and required outcomes before writing fallback code. A missing sidebar can be acceptable; silently bypassing authorization, presenting an uncertain payment as successful, or substituting an unverified price can violate the core promise. Those cases require an explicit failure or a separate approved workflow rather than a generic “keep the page green” rule.

Keep the optional dependency outside the core response’s critical path where possible. Give its call a bounded budget, but remember that a caller timeout does not necessarily stop underlying work. Bound concurrency and queued work as well. A placeholder should name the missing capability without exposing internal errors, and should not imply that fresh data was obtained when it was not.

Make stale-data acceptance explicit

Last-known-good data is useful only within a declared scope and age policy. Record when the data was obtained, preserve its identity and authorization scope, and decide what happens when its age is unknown, excessive, or the cache is empty. A timestamp alone does not establish correctness: some data is unsafe to reuse even briefly after a relevant state change.

Serving stale after an upstream failure is not identical to stale-while-revalidate, which can serve an eligible stale response while validation proceeds. Either approach needs bounded freshness and resource behavior. Successful refresh should actually replace the stored value and its age. Otherwise a service can repeatedly claim to use its latest good result while returning an older snapshot forever.

When it breaks

Fallback may amplify an outage. AWS’s account of avoiding fallback describes a cache failure that redirected traffic into a database unable to absorb it. That source is a warning about difficult, rarely exercised alternate paths, not a blanket endorsement of partial answers. Prefer predictable primary behavior and regularly exercised recovery paths over a complicated fallback whose resource demands are unknown.

Synchronized cache expiry and simultaneous recovery probes can create bursts. Spread refresh work, limit retries and concurrency, and measure whether the fallback competes with the same exhausted resource. A circuit breaker decides whether to call a dependency; the degradation policy decides what acceptable answer remains. An open breaker does not itself prove that a cached response is valid or that a static answer meets the contract.

A failed recovery probe must leave normal admission blocked. Closing a breaker before testing recovery can release traffic too early. Production implementations also need synchronization, probe scheduling, and bounded recovery concurrency; a serial example cannot establish those properties. Degraded responses should have their own count and age observations so that an HTTP 200 does not hide persistent loss of functionality.

Only the optional recommendation contract permits this fallback. Ages 0–120 seconds may reuse the latest cache; otherwise the model returns a labeled static simplification, and a failed probe keeps normal calls blocked.

Check it quickly

Save this as degrade_probe.py and run python3 degrade_probe.py. The standard-library fixture models one optional recommendation region. It performs no network or filesystem operations. Cache age and dependency failure are injected inputs, and the 120-second limit is an illustrative product decision. Probe calls are explicit and synchronous; no timer or concurrent request admission is implemented.

class DependencyDown(Exception):
 pass

class PageService:
 def __init__(self):
 self.cached_recs = ["cached-pick"]
 self.cached_age_s = 90 # Injected age, not a running clock.
 self.max_age_s = 120
 self.failures, self.threshold = 0, 3
 self.open = False
 self.live_calls = 0

 def live_recs(self, fail):
 self.live_calls += 1
 if fail:
 raise DependencyDown("recs down")
 return ["fresh-pick"]

 def fresh(self, data):
 self.cached_recs, self.cached_age_s = list(data), 0
 self.failures, self.open = 0, False
 return {"mode": "fresh", "data": data}

 def fallback(self, reason):
 if self.cached_recs and 0 <= self.cached_age_s <= self.max_age_s:
 return {"mode": "cached", "data": list(self.cached_recs),
 "age_s": self.cached_age_s, "reason": reason}
 return {"mode": "simplified", "data": ["popular-picks"], "reason": reason}

 def get_recs(self, fail=False):
 if self.open:
 return self.fallback("breaker-open")
 try:
 return self.fresh(self.live_recs(fail))
 except DependencyDown:
 self.failures += 1
 self.open = self.failures >= self.threshold
 return self.fallback("dependency-failed")

 def half_open_probe(self, fail=False):
 # Explicit synchronous trial; not a concurrent admission controller.
 assert self.open, "probe requires an open breaker"
 try:
 data = self.live_recs(fail)
 except DependencyDown:
 return self.fallback("probe-failed") # Leave the breaker open.
 return self.fresh(data) # Close only after success.

svc = PageService()
assert svc.get_recs() == {"mode": "fresh", "data": ["fresh-pick"]}
assert svc.cached_recs == ["fresh-pick"] and svc.cached_age_s == 0
svc.cached_age_s = 120
cached = svc.get_recs(fail=True)
assert cached["mode"] == "cached" and cached["age_s"] == 120
assert cached["data"] == ["fresh-pick"]
svc.cached_age_s = 121
assert svc.get_recs(fail=True)["mode"] == "simplified"
svc.get_recs(fail=True)
assert svc.open
calls = svc.live_calls
assert svc.get_recs()["reason"] == "breaker-open" and svc.live_calls == calls
assert svc.half_open_probe(fail=True)["reason"] == "probe-failed"
assert svc.open and svc.live_calls == calls + 1
calls = svc.live_calls
svc.get_recs()
assert svc.live_calls == calls # A failed probe must not reopen normal admission.
assert svc.half_open_probe()["mode"] == "fresh" and not svc.open
assert svc.cached_age_s == 0
svc.cached_recs = []
assert svc.get_recs(fail=True)["mode"] == "simplified"
svc.cached_recs, svc.cached_age_s = ["unknown-age"], -1
assert svc.get_recs(fail=True)["mode"] == "simplified"
print("PASS: fresh refresh, bounded cache, empty/invalid cache, open gate, failed/successful probe")

A successful call refreshes the cache. The next failure serves that actual latest value when its injected age is exactly 120 seconds. At 121 seconds it uses a clearly simplified answer instead. Three consecutive failures open the call gate, and another ordinary request returns fallback without incrementing the live-call count. These checks distinguish acceptable stale data from merely available stale data.

The failed probe performs one live attempt but leaves the breaker open; a subsequent ordinary request still makes no live call. A successful probe then closes the gate and refreshes the cache. Separate empty-cache and invalid-negative-age controls also choose the simplified response. The final PASS line covers these deterministic decisions, not production availability.

Before deployment, exercise the chosen contract against real dependency failures and recovery under controlled load. Verify deadlines, cancellation, authorization scope, cache aging, and visible degradation metrics independently. The example’s static recommendation fallback is intentionally cheap; it does not demonstrate that an alternate remote service would remain healthy during a correlated outage.

Sources

AWS: Avoiding fallback in distributed systems, risks of infrequent fallback paths and amplified outages, checked 2026-10-15.
Google SRE: graceful degradation, shedding to keep the useful parts, checked 2026-10-15.
Martin Fowler: CircuitBreaker, open-state rejection and controlled recovery probes, checked 2026-10-15.

Retries with jitter and backoff

Sat, 10 Apr 2027 00:00:00 +0000

A retry is another unit of work, not an automatic improvement in reliability. It can recover from a transient failure, but can also repeat an unsafe operation or add demand to an overloaded dependency. Decide whether the operation is eligible, how much total effort is allowed, and when that effort should happen before choosing a delay formula.

Dots illustrate samples, not measured arrivals. Jitter guarantees neither capacity nor increasing waits. Check eligibility, attempts, shared deadline, wider budgets, and Retry-After; abandon a required wait that cannot fit.

Bound attempts before spreading them

Distinguish attempts from retries: three total attempts means one initial call and at most two retries. Also choose which layer owns recovery. If three nested layers each allow three attempts, one outer action can cause twenty-seven calls to a persistently failing leaf. Adding jitter changes their timing, not that multiplication.

Google SRE describes both per-request and wider retry budgets. A per-request cap alone still allows a large population of failing clients to generate substantial extra work. Measure retry traffic separately from original demand and coordinate with existing SDK, proxy, and application policies.

Exponential backoff sets a ceiling

A capped exponential schedule increases its ceiling until reaching a maximum. Without jitter, clients that fail together can calculate the same next time, including the same capped interval. Waiting longer does not necessarily break their synchronization.

Full jitter samples between zero and the current ceiling. Because the draw is random, the next realized wait can be shorter than the previous one even while the ceiling grows. It is inaccurate to say every individual wait must increase.

AWS’s comparison distinguishes full, equal, and decorrelated jitter and reports results from a particular contention simulation. Those results are useful evidence, not a promise that one policy minimizes every workload’s latency or prevents all bursts. Correlated random seeds, timer granularity, and many clients can still create clusters.

Use one end-to-end deadline

The budget must include execution, waiting, and subsequent attempts. Creating a fresh full timeout each time silently expands the original caller’s allowed lifetime. Propagate the remaining budget to downstream operations and recheck it before doing more work.

Use a monotonic clock for elapsed-time accounting within a process. A client timeout is not proof that the server stopped or never committed. Cancellation and remote deadlines need real transport support, and the application still needs its retry or idempotency contract.

A server-directed wait such as Retry-After adds another constraint. Do not sample an earlier retry than that boundary merely because a local jitter formula permits it. If the required wait cannot fit, abandon that retry under the current deadline instead of shortening the requirement.

When it breaks

Permanent validation errors do not become valid through repetition. A nonidempotent operation with an uncertain result can duplicate its effect. Select retriable failure classes deliberately; a blanket catch-and-retry loop hides programming errors and may convert a small incident into an overload cycle.

A backoff cap without an attempt or elapsed-time bound still permits endless work. Conversely, a deadline that is too short can reject every expensive legitimate operation. Observe completed work, retry counts, total latency, and abandonment reasons together; a falling error rate alone can hide rising retry cost.

Check it quickly

Save the block as retry_check.py and run python3 retry_check.py with Python 3.10 or newer. The complete unchanged program passed on 2026-10-19 with Python 3.12.14 using only the standard library. The clock and downstream calls are deterministic models; the script does not sleep, start servers, or contact a dependency. Its retry policy handles timeouts and 503 only for GET or HEAD.

from collections import Counter
import random


class Clock:
 def __init__(self):
 self.now = 0.0

 def advance(self, seconds):
 self.now += seconds


class ScriptedCall:
 def __init__(self, clock, steps):
 self.clock, self.steps, self.timeouts = clock, iter(steps), []

 def __call__(self, timeout):
 self.timeouts.append(timeout)
 status, duration = next(self.steps)
 self.clock.advance(min(duration, timeout))
 if duration >= timeout:
 raise TimeoutError("the cooperative fixture used the remaining budget")
 return status


def retry(call, clock, budget, sample, method="GET", max_attempts=3):
 deadline = clock.now + budget
 for attempt in range(1, max_attempts + 1):
 remaining = deadline - clock.now
 if remaining <= 0:
 return "deadline", attempt - 1
 try:
 status = call(remaining)
 except TimeoutError:
 status = None
 if clock.now >= deadline:
 return "deadline", attempt
 if status == 200:
 return "success", attempt
 if status not in (None, 503) or method not in ("GET", "HEAD"):
 return "not-retryable", attempt
 if attempt == max_attempts:
 return "attempt-limit", attempt
 ceiling = 2.0 ** min(attempt - 1, 3)
 delay = ceiling * sample()
 assert 0 <= delay <= ceiling
 if delay >= deadline - clock.now:
 return "wait-exceeds-budget", attempt
 clock.advance(delay)
 raise AssertionError("positive attempt limit required")


clock = Clock()
call = ScriptedCall(clock, [(503, 0.25), (503, 0.25), (200, 0.25)])
assert retry(call, clock, 5, lambda: 0.5) == ("success", 3)
assert clock.now == 2.25 and call.timeouts == [5, 4.25, 3]
print("three attempts finish at 2.25; propagated timeouts shrink from 5 to 4.25 to 3")

class EarlyTimeout:
 def __init__(self, clock):
 self.clock, self.timeouts = clock, []

 def __call__(self, timeout):
 self.timeouts.append(timeout)
 self.clock.advance(0.25)
 if len(self.timeouts) == 1:
 raise TimeoutError("a transient failure before the deadline")
 return 200


clock = Clock()
call = EarlyTimeout(clock)
assert retry(call, clock, 5, lambda: 0.5) == ("success", 2)
assert clock.now == 1.0 and call.timeouts == [5, 4.25]
print("an early timeout can recover without resetting the shared deadline")

for method, status in (("POST", 503), ("GET", 400)):
 clock = Clock()
 call = ScriptedCall(clock, [(status, 0.25)])
 assert retry(call, clock, 5, lambda: 0.5, method) == ("not-retryable", 1)
clock = Clock()
call = ScriptedCall(clock, [(503, 0.25)] * 4)
assert retry(call, clock, 5, lambda: 0.5) == ("attempt-limit", 3)
assert len(call.timeouts) == 3
print("POST, permanent failure, and the total-attempt cap stop further work")

clock = Clock()
call = ScriptedCall(clock, [(503, 0.25)] * 7)
assert retry(call, clock, 100, lambda: 1.0, max_attempts=7) == ("attempt-limit", 7)
assert clock.now == 32.75 # Seven 0.25 calls plus waits 1, 2, 4, 8, 8, 8.
print("the exponential ceiling stops at 8 rather than growing without bound")

clock = Clock()
call = ScriptedCall(clock, [(503, 0.1)])
assert retry(call, clock, 0.5, lambda: 1.0) == ("wait-exceeds-budget", 1)
assert clock.now == 0.1 and len(call.timeouts) == 1
clock = Clock()
call = ScriptedCall(clock, [(503, 99)])
assert retry(call, clock, 0.5, lambda: 0.5) == ("deadline", 1)
assert clock.now == 0.5
clock = Clock()
call = ScriptedCall(clock, [])
assert retry(call, clock, 0, lambda: 0.5) == ("deadline", 0) and call.timeouts == []
print("a wait that cannot fit is abandoned; cooperative execution cannot consume a fresh budget per attempt")

rng = random.Random(7) # Repeatable samples for this experiment, not a production seed.
delays = [rng.random() for _ in range(100)]
bins = Counter(int(delay * 10) for delay in delays)
fixed_bins = Counter(int(1.0 * 10) for _ in range(100))
assert all(0 <= delay < 1 for delay in delays) and sum(bins.values()) == 100
assert max(bins.values()) < 100
assert max(fixed_bins.values()) == 100
print("first-retry 0.1-second-bin peak: fixed delay", max(fixed_bins.values()), "; full-jitter sample", max(bins.values()))

leaf_calls = 0


class Unavailable(Exception):
 pass


def nested(depth):
 global leaf_calls
 if depth == 0:
 leaf_calls += 1
 raise Unavailable
 for _ in range(3):
 try:
 return nested(depth - 1)
 except Unavailable:
 pass
 raise Unavailable


try:
 nested(3)
except Unavailable:
 pass
assert leaf_calls == 27
print("three retrying layers, each allowing three total attempts, invoke the failing leaf 27 times")

Two failures followed by success finish at modeled time 2.25. The propagated timeouts shrink from five to 4.25 to three seconds; execution does not receive a fresh budget. An early timeout also recovers within that shared budget. The exponent is capped before evaluation. Separate assertions cover the total-attempt limit, an eight-second ceiling, permanent errors, POST exclusion, a wait that cannot fit, and an already exhausted deadline.

One hundred fixed first-retry delays land in one 0.1-second bin. The reproducible full-jitter sample has a peak bin count of thirteen. This compares calculated schedules, not actual network arrival times or successful throughput. The fixed random seed exists only to reproduce the experiment, not to synchronize production clients.

Finally, three nested retry loops produce twenty-seven leaf invocations before failure. The example deliberately shows that amplification rather than recommending it. Wider retry quotas, server-directed delay parsing, real timers, transport cancellation, distributed clocks, and production latency remain outside this model’s tests.

Sources

AWS: Exponential backoff and jitter, capped backoff, full-jitter formula, and workload-specific comparisons, checked 2026-10-15.
Google SRE: Handling overload, request/client retry budgets and retry ownership, checked 2026-10-15.
Google SRE: Addressing cascading failures, retry multiplication, deadline propagation, and cancellation, checked 2026-10-15.
RFC 9110, idempotent-method retries and Retry-After semantics, checked 2026-10-15.

Circuit breakers stop storming a down service

Tue, 06 Apr 2027 00:00:00 +0000

A circuit breaker decides whether a new call should reach a dependency that appears unhealthy. It can reject promptly instead of spending resources on another likely failure. It does not repair that dependency, cancel calls already running, or prove that rejected work would have failed. Its value depends on the failure signals and recovery policy it actually implements.

Name the policy, not just the states

Closed means calls may proceed, not that every call will succeed. Open means new calls are refused under the breaker’s policy. Half-open admits limited trial work to gather evidence before deciding whether ordinary traffic can resume.

The detection rule is not universally a sliding-window failure percentage. Fowler’s example counts failures and resets on success; other implementations use time windows, failure ratios, slow-call thresholds, or minimum sample counts. Specify the actual rule rather than borrowing a diagram and assuming its thresholds.

The fixture uses two consecutive classified failures in completion order, a five-second cooldown, and one half-open probe. One success resets the consecutive counter. These numbers make transitions easy to verify; they are not suggested production settings.

Classify and scope failures

A malformed caller request does not necessarily mean a dependency is down. Decide which exceptions, statuses, and slow outcomes should influence the breaker. A locally refused call is also different from a request that reached the dependency and failed. Mixing those observations can make the breaker reinforce its own refusal state.

Scope state to the failure domain you intend to protect. One unavailable shard or operation need not justify blocking a healthy independent provider. Conversely, thousands of independent process-local breakers can each allow a probe, so “one half-open request” may mean one per instance, not one globally.

Timeouts remain necessary. A call that never finishes might hold a connection or worker without ever producing the failure observation needed to open the breaker. The admission decision does not replace a deadline, a concurrency limit, or resource cleanup.

Recovery is a controlled experiment

After cooldown, reserve probe admission atomically. If several threads all observe that trial work is allowed before any claims the permit, the supposed single probe becomes a burst. The example protects its small state transition with a lock; it does not hold that lock while a dependency operation runs.

An old in-flight result also needs an explicit interpretation. Here, opening creates a new local generation, and results from older permits cannot close that newer open state. The generation is bookkeeping inside this model, not a distributed fencing token or a way to undo remote work.

A failed probe starts another cooldown. A successful probe closes this fixture’s breaker, but one successful trial does not prove capacity for all returning traffic. If gradual recovery is required, validate the actual traffic-admission policy instead of assuming the word half-open guarantees a ramp.

When it breaks

Aggressive thresholds can deny healthy requests; long cooldowns can keep refusing after recovery. Overly broad scopes hide partial health. A retry loop that repeatedly catches an open-circuit refusal can waste resources even though it sends no new dependency requests.

Observe forwarded calls, classified outcomes, local refusals, state transitions, and probe results separately. Test sustained failures, occasional successes, concurrent trial admission, and late completions. A green unit test for a counter is not evidence that a live dependency recovers safely.

The model uses two consecutive classified failures, a five-second cooldown and one atomic trial permit. Old-generation results cannot close a newer opening; no production recovery throughput is established.

Check it quickly

Save the block as breaker_check.py and run python3 breaker_check.py with Python 3.10 or newer. It ran with Python 3.14.4 using only the standard library. The script models outcomes and injects monotonic-time values without sleeping or contacting a service. Eight real threads contend for a half-open permit.

from concurrent.futures import ThreadPoolExecutor
from threading import Barrier, Lock


class OpenCircuit(Exception):
 pass


class Breaker:
 # Explicit fixture policy: two consecutive classified failures, five-second cooldown.
 def __init__(self):
 self.state, self.failures, self.until = "closed", 0, 0
 self.epoch, self.probe_busy = 0, False
 self.lock = Lock()

 def acquire(self, now):
 with self.lock:
 if self.state == "open":
 if now < self.until:
 raise OpenCircuit
 self.state = "half-open"
 if self.state == "half-open":
 if self.probe_busy:
 raise OpenCircuit
 self.probe_busy = True
 return self.epoch, True
 return self.epoch, False

 def complete(self, permit, succeeded, now):
 epoch, probe = permit
 with self.lock:
 if epoch != self.epoch:
 return # This outcome belongs to an earlier breaker generation.
 if probe:
 self.probe_busy = False
 self.epoch += 1
 self.failures = 0
 self.state = "closed" if succeeded else "open"
 if not succeeded:
 self.until = now + 5
 elif self.state == "closed":
 self.failures = 0 if succeeded else self.failures + 1
 if self.failures >= 2:
 self.state, self.until = "open", now + 5
 self.epoch += 1


breaker = Breaker()
other_dependency = Breaker()
forwarded = []


def invoke(succeeded, now):
 permit = breaker.acquire(now)
 forwarded.append(now) # A modeled dependency invocation, not a network call.
 breaker.complete(permit, succeeded, now)


invoke(False, 0)
assert breaker.failures == 1
invoke(True, 1)
assert breaker.failures == 0
late = breaker.acquire(1)
invoke(False, 2)
invoke(False, 3)
assert breaker.state == "open" and breaker.until == 8
other_permit = other_dependency.acquire(4)
other_dependency.complete(other_permit, True, 4)
assert other_dependency.state == "closed"
before = len(forwarded)
try:
 invoke(True, 7.99)
except OpenCircuit:
 pass
else:
 raise AssertionError("open circuit must reject before dispatch")
assert len(forwarded) == before
breaker.complete(late, True, 7.99)
assert breaker.state == "open"
print("success resets consecutive failures; two later failures open until time 8")
print("open calls are not dispatched; an older in-flight success does not close a newer open generation")

gate = Barrier(8)


def probe_attempt(_):
 gate.wait(timeout=5)
 try:
 return breaker.acquire(8)
 except OpenCircuit:
 return None


with ThreadPoolExecutor(max_workers=8) as pool:
 permits = list(pool.map(probe_attempt, range(8)))
admitted = [permit for permit in permits if permit is not None]
assert len(admitted) == 1 and admitted[0][1] is True
assert breaker.state == "half-open" and breaker.probe_busy
breaker.complete(admitted[0], False, 8)
assert breaker.state == "open" and breaker.until == 13
print("eight simultaneous half-open candidates receive one probe permit and seven refusals")
print("failed probe opens a new cooldown ending at time 13")

try:
 breaker.acquire(12.99)
except OpenCircuit:
 pass
else:
 raise AssertionError("second cooldown must still apply")
probe = breaker.acquire(13)
breaker.complete(probe, True, 13)
assert breaker.state == "closed" and not breaker.probe_busy
invoke(True, 14)
assert other_dependency.state == "closed" and other_dependency.failures == 0
print("a successful later probe closes this breaker; another dependency has separate state")

An initial success resets the failure count. Two later failures open the breaker until time eight, and an attempted call before that boundary adds no modeled dispatch. A previously admitted success cannot close the newer generation. A second dependency’s already existing breaker remains usable.

At time eight, eight simultaneous candidates obtain one permit and seven refusals. Failure of that probe opens another interval ending at thirteen. A later successful probe closes the breaker and permits ordinary work again.

Each granted permit in this model receives one completion with an already classified outcome. Real transport timeouts, cancellation, automatic failure classification, distributed probe coordination, library integration, and recovery throughput are not tested. Use an established implementation where appropriate and verify its precise contracts against the real workload.

Sources

Martin Fowler: Circuit breaker, failure counting, success reset, and trial recovery, checked 2026-09-08.
Azure architecture: Circuit breaker, state semantics, exception classification, concurrency, and failure-domain scope, checked 2026-09-08.
Python threading, locks and barriers used by the concurrent permit experiment, checked 2026-09-08.

Error budgets are a real budget

Thu, 25 Mar 2027 00:00:00 +0000

An error budget makes a reliability agreement usable during planning. Product and operations choose an objective, measure failures against it, and agree what changes when the allowance is exhausted. The number supports that agreement; it does not predict whether tomorrow’s deployment will cause an incident. Shipping a release creates risk, while observed bad events consume the measured budget.

Request counts and policy cases come from the article’s synthetic fixture. The bars share a request-count scale. This drawing does not execute the code or grant deployment approval; the positive balance only makes a closed-incident case eligible for review.

Give the account a measurement rule

For a request-based objective, allowed bad events equal eligible requests multiplied by one minus the target. A 99.9% objective over three million requests permits three thousand bad requests. Specify which requests count, which outcomes are bad, the measurement location, and the window before attaching a release rule. A traffic-weighted allowance is not automatically a fixed number of downtime minutes.

A rolling thirty-day account drops old observations as new ones arrive. Its denominator changes when traffic changes, so the allowance can change even when the target stays fixed. A calendar-month account resets at a calendar boundary instead. Those approaches answer different questions; switching between them during an incident makes an apparently precise balance difficult to audit. Keep the definition versioned with the policy and show the window beside the balance.

Decide the response before the argument

An example policy might hold nonessential feature releases when the budget is overspent, prioritize reliability work, and allow urgent security or incident fixes through review. Name who owns the decision, what evidence permits an exception, and when ordinary release review can resume. Google’s published example is a policy for its example service, not a universal requirement to copy every threshold or escalation rule.

Remaining budget alone does not establish release readiness. An open incident, missing monitoring, a failed regression, or unsafe rollback can still prevent a change. Conversely, a necessary repair should not be blocked merely because the incident it repairs exhausted the budget. Record these decisions explicitly. The useful result is a shared response to customer impact, not an automated mechanism for blaming the team whose dashboard turned red.

When it breaks

A team can appear to recover simply because yesterday’s large incident leaves the rolling window. The arithmetic is correct, but it says nothing about whether the defect was repaired. Changing exclusions, silently replacing unavailable samples with successes, or averaging differently sized service percentages can also create a misleading surplus. Display measurement coverage and investigate definition changes before interpreting an improved balance.

Short-window burn rate supplies another view: the observed bad-event fraction divided by the allowed fraction. A high burn can warn before the longer account is exhausted. Estimating time to exhaustion from that rate assumes comparable traffic and continued behavior; it is not a promised deadline. Pair actionable alerts with the long-window account and account for sparse traffic, rather than paging on every isolated failure.

Check it quickly

Save the following as budget-policy.py and run it with Python 3. It uses synthetic daily request counts to verify rolling-window accounting and a deliberately small example policy. It does not change a deployment system or estimate the risk of a real release.

from fractions import Fraction

target = Fraction(999, 1000)
# Every synthetic daily bin has 100,000 eligible requests.
days = [(100000, 1300)] + [(100000, 70)] * 29

def account(window):
 eligible = sum(total for total, _ in window)
 bad = sum(errors for _, errors in window)
 allowed = eligible * (1 - target)
 return bad, allowed, allowed - bad

def feature_policy(remaining, incident_open):
 return "hold" if remaining < 0 or incident_open else "eligible for review"

before = account(days)
after = account(days[1:] + [(100000, 50)])
assert before == (3330, 3000, -330)
assert after == (2080, 3000, 920)
assert feature_policy(before[2], False) == "hold"
assert feature_policy(after[2], True) == "hold"
assert feature_policy(after[2], False) == "eligible for review"
for label, (bad, allowed, remaining) in [("before", before), ("after", after)]:
 print(f"{label}: bad={bad}, allowed={float(allowed):.0f}, remaining={float(remaining):.0f}, spent={float(bad / allowed):.2%}")
print("after rotation, incident still open:", feature_policy(after[2], True))
print("after rotation, incident closed:", feature_policy(after[2], False))

The first account has 3,330 bad requests against an allowance of 3,000: 111% spent and a balance of minus 330. Removing the oldest day and adding the new day leaves 2,080 bad requests, 69.33% spent, and 920 remaining. Both windows contain the same three million eligible requests in this fixture; real traffic need not stay constant.

The policy still prints hold while an incident is open, despite the positive new balance. Only the final case becomes eligible for review, which intentionally stops short of granting deployment approval. This separates a verified calculation from the operational judgment that uses it. Review the definition, coverage, incident evidence, and release checks together before acting on the account.

Sources

Google SRE Workbook: Example Error Budget Policy, release restrictions, exceptions, and ownership, checked 2026-09-13.
Google SRE: Service Level Objectives, measurement boundaries and reliability targets, checked 2026-09-13.
Google SRE Workbook: Alerting on SLOs, burn rate and alerting windows, checked 2026-09-13.

Failover is a practiced switch

Thu, 25 Feb 2027 00:00:00 +0000

A standby helps only if the promotion path works and the old writer cannot corrupt the new history. Failover is a decision, a procedure, and a rehearsal. A healthy-looking replica is not proof that clients can reach it or that it contains every acknowledged write. Define the required recovery time and acceptable data loss before deciding when to switch.

Design the decision in advance

Choose who can authorize promotion, what evidence demonstrates failure, and whether automation or an operator acts. A lost heartbeat can mean a network partition rather than a dead primary. Failure detection alone cannot authorize two independent writers. The system needs an enforced ownership mechanism, such as verified node isolation or resource-side fencing, with guarantees appropriate to its replication protocol.

For a single-writer service, the safety goal is at most one accepted writer, not uninterrupted availability of exactly one. Fencing may deliberately leave zero writers during a transition. The runbook should identify the candidate, its replication position, the acceptable loss boundary, how clients reconnect, and who stops the procedure if prerequisites fail. Automatic failover can shorten an outage but cannot manufacture missing replicated data.

Rehearse the actual path

Practice at a cadence matched to risk and after relevant changes. A drill validates the paths it actually exercises: a database promotion that bypasses DNS says little about cached addresses, and a test client using a different identity says little about production permissions. Include realistic client reconnection and checks of acknowledged writes where safe. Record scope, timings, lag, refusals, and recovery evidence, rather than treating the date of the last successful drill as a complete readiness score.

Plan the safe stopping point and recovery procedure before starting. Once the promoted node has accepted new writes, switching back is not necessarily a reversible configuration edit. The former primary may need reconciliation or rebuilding as a replica. Backups provide another recovery path, but their existence alone does not prove a usable restore.

When it breaks

Split brain appears when the old primary remains writable after promotion. Merely recording “fenced” in a controller is insufficient; the storage or isolation mechanism must enforce it. A stale candidate can lose acknowledged writes even though promotion succeeds. Express lag thresholds in meaningful units and tie them to the data-loss objective; a nonzero threshold is not a zero-loss promise.

Flapping health checks can provoke repeated transitions. Use deliberate failure thresholds and recovery criteria, and avoid automatic failback before resynchronization. DNS changes affect later lookups: existing connections and answers cached under the previous TTL may persist. Lowering the TTL during an incident does not retroactively expire those cached answers. Test the actual client’s retry and reconnection behavior as well as the authoritative DNS response.

Fence before promoting a writer. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

With Python 3.9 or newer, save this as failover_drill.py and run python3 failover_drill.py in a scratch directory. Run without -O, -OO, or a nonzero PYTHONOPTIMIZE setting so the assertions execute. It changes only in-memory objects. Health flags and lag values are synthetic; fence() is a set update, not a real fence command. The model keeps one designated primary while allowing zero writable nodes during isolation. It explicitly reactivates an eligible candidate, including during failback, instead of accidentally leaving the new primary in the fenced set.

"""Failover drill: single-primary switch with health and lag guards (stdlib only)."""
from dataclasses import dataclass

MAX_LAG = 128 # Synthetic units, not seconds or a zero-data-loss guarantee.

class FailoverRefused(Exception):
 pass

@dataclass
class Node:
 name: str
 healthy: bool = True
 lag: int = 0
 is_primary: bool = False

class Cluster:
 def __init__(self):
 self.nodes = {"a": Node("a", is_primary=True), "b": Node("b")}
 self.fenced: set[str] = set()

 @property
 def primary(self) -> str:
 primaries = [n for n in self.nodes.values() if n.is_primary]
 assert len(primaries) == 1, f"must keep exactly one primary, saw {[p.name for p in primaries]}"
 return primaries[0].name

 def can_write(self, name: str) -> bool:
 node = self.nodes[name]
 return node.is_primary and node.healthy and name not in self.fenced

 def health_check(self, name: str) -> bool:
 node = self.nodes[name]
 return node.healthy and node.lag <= MAX_LAG

 def fence(self, name: str) -> None:
 self.fenced.add(name)
 assert not self.can_write(name)

 def failover(self, target: str, *, fence_old: bool) -> str:
 old = self.primary
 if target == old:
 raise FailoverRefused("target is already primary")
 if target not in self.nodes:
 raise FailoverRefused(f"unknown target {target}")
 candidate = self.nodes[target]
 if not fence_old or old not in self.fenced:
 raise FailoverRefused(f"refuse: old primary {old} is not fenced; fence first to avoid split brain")
 if not candidate.healthy:
 raise FailoverRefused(f"refuse: standby {target} is unhealthy")
 if candidate.lag > MAX_LAG:
 raise FailoverRefused(f"refuse: standby {target} lags by {candidate.lag} (> {MAX_LAG})")
 self.nodes[old].is_primary = False
 # Model explicit reactivation after the old writer is fenced.
 self.fenced.discard(target)
 candidate.is_primary = True
 assert self.primary == target and self.can_write(target)
 assert sum(self.can_write(name) for name in self.nodes) <= 1
 return target

def expect_refused(fn, label: str) -> None:
 try:
 fn()
 except FailoverRefused as exc:
 print(f"refused as expected ({label}): {exc}")
 return
 raise AssertionError(f"should have refused: {label}")

def main() -> None:
 # 1. Clean drill: healthy standby, caught-up replica, old primary fenced.
 cluster = Cluster()
 assert cluster.primary == "a"
 assert cluster.health_check("b")
 cluster.fence("a")
 assert cluster.failover("b", fence_old=True) == "b"
 print("clean switch: a -> b")

 # 2. Refuse lag beyond this policy; even accepted positive lag may lose writes.
 cluster = Cluster()
 cluster.nodes["b"].lag = 900
 assert not cluster.health_check("b")
 cluster.fence("a")
 expect_refused(lambda: cluster.failover("b", fence_old=True), "stale replica")
 assert cluster.primary == "a"

 # 3. Unhealthy standby must not promote.
 cluster.nodes["b"].lag = 0
 cluster.nodes["b"].healthy = False
 expect_refused(lambda: cluster.failover("b", fence_old=True), "unhealthy standby")
 assert cluster.primary == "a"

 # 4. Unfenced old primary must block promotion (split-brain guard).
 cluster.nodes["b"].healthy = True
 unfenced = Cluster() # nobody fenced yet
 expect_refused(lambda: unfenced.failover("b", fence_old=False), "split brain")
 expect_refused(lambda: unfenced.failover("b", fence_old=True), "split brain without fence record")
 assert unfenced.primary == "a"

 # 5. Failback: resynchronize the candidate, then fence the CURRENT writer.
 cluster = Cluster()
 cluster.fence("a")
 cluster.failover("b", fence_old=True)
 cluster.nodes["a"].healthy = True
 cluster.nodes["a"].lag = 0
 cluster.fence("b")
 assert cluster.failover("a", fence_old=True) == "a"
 print("failback: b -> a")

 print("PASS: at most one writable node; 4 refusals + clean switch + failback passed")

if __name__ == "__main__":
 main()

The run prints a clean switch, four refusals in three categories, and failback, ending with PASS: at most one writable node; 4 refusals + clean switch + failback passed. Refused promotion leaves the designated primary unchanged, but a previously fenced primary remains unavailable. The clean-switch and failback fixtures use zero lag; the policy also permits positive lag up to 128 synthetic units, which carries a possible loss trade-off.

The earlier fixture recorded Python 3.14.4 on macOS. The revised fixture passed with Python 3.12.14 on Linux. These assertions test sequential decision logic, including the formerly fenced target’s writable state. They do not validate concurrent promotion, real replication, permissions, DNS propagation, storage isolation, or production recovery. Those require a separately authorized staging drill with the actual mechanisms and representative clients.

Sources

PostgreSQL 18: failover, old-primary isolation, rebuilding redundancy, and written switch procedures, checked 2026-10-15.
AWS: configuring DNS failover, health checks and routing configuration, checked 2026-10-15.
Google SRE: testing for reliability, production-test scope, automation barriers, and disaster recovery testing, checked 2026-10-15.
AWS: record TTL, resolver caching and planning TTL changes before a switch, checked 2026-10-15.

Queues define delivery; consumers must protect effects

Fri, 05 Feb 2027 00:00:00 +0000

A queue’s delivery promise describes a bounded part of a workflow. Getting a message to a consumer is different from committing its business effect. At-least-once delivery allows redelivery; at-most-once delivery permits loss rather than retrying uncertain delivery. Neither label alone guarantees that an order, notification, or database update completes exactly once under every failure.

Locate publication, acknowledgment, and commitment

First establish when the producer knows that the broker accepted a message, how long the broker retains it, and which failures its durability policy covers. A message never successfully published, expired by retention, or deliberately deleted is not rescued by calling the consumer at-least-once. Recovery assumptions belong beside the delivery label.

On the consumer side, acknowledging before the effect commits creates a loss window: the broker can stop retrying work the application has not finished. Committing first leaves another window: the effect succeeds, but acknowledgment is lost and the work returns. Redelivery is then compatible with the delivery contract. It does not mean the original operation failed.

Visibility timeouts and acknowledgment deadlines govern availability for another attempt. They are not proof that the first worker stopped. A slow worker and a redelivery can overlap, so extending a deadline and protecting the effect from duplicates solve different problems. Record both delivery attempts and committed business outcomes.

Make repeated effects harmless within a real boundary

For effects inside one transactional database, store a unique operation identity and its effect in the same transaction. The uniqueness check must arbitrate concurrent attempts; a separate read followed by an unprotected write is insufficient. Persist enough payload information to reject reuse of the same identity for a different operation rather than silently accepting conflicting instructions.

A database transaction cannot automatically include an external email, payment API, or another independent store. Those destinations need their own idempotency or coordination contract. A local marker written before the remote call can lose the effect; written afterward, it can leave a duplicate window. Do not present a local deduplication table as an end-to-end exactly-once guarantee.

Some services provide explicitly scoped exactly-once delivery features. Google Pub/Sub, for example, documents regional and subscription-type restrictions and distinguishes delivery duplicates from publish-side duplicates. Read that contract instead of claiming the feature is impossible or universally solves downstream effects. Stable business identities remain useful when multiple published messages represent the same intended operation.

When it breaks

A repeatedly failing message can block ordered progress, while a short deadline can cause overlapping attempts for otherwise valid work. Redelivery counts suggest where to investigate; they do not uniquely distinguish poison data, slow dependencies, lost acknowledgments, or stalled workers. Correlate identity, payload, exception, processing duration, and acknowledgment evidence before changing retry policy.

Bound retries and define how isolated failures are reviewed or replayed. Dead-lettering moves unresolved work; it does not complete it. Deduplication retention must cover the replay horizon you intend to tolerate. Removing the identity record can allow an old request to take effect again, while retaining identifiers indefinitely has storage and privacy costs.

Historical exact SQLite sink evidence covers rollback, reopen and duplicate handling. The total is a synthetic fixture value; broker behavior and external effects remain separate.

Check it quickly

Save as delivery_guarantee_probe.py and use Python 3 with assertions enabled, without -O or PYTHONOPTIMIZE: assertions contain required calls. Retained exact-fence records report Python 3.12.14 and SQLite 3.53.1 passing; this review did not rerun them. A temporary SQLite database stores identities and a harmless numeric total. This exercises a transactional sink; no account is charged or external message sent.

from contextlib import closing
from pathlib import Path
from tempfile import TemporaryDirectory
import sqlite3


def apply(db, message_id, amount, interrupt=False):
 db.execute('BEGIN IMMEDIATE')
 try:
 prior = db.execute('SELECT amount FROM inbox WHERE id=?', (message_id,)).fetchone()
 if prior is not None:
 if prior[0] != amount:
 raise ValueError('same identity, different payload')
 db.commit()
 return False
 db.execute('INSERT INTO inbox VALUES (?, ?)', (message_id, amount))
 db.execute('UPDATE totals SET amount=amount+? WHERE id=1', (amount,))
 if interrupt:
 raise RuntimeError('controlled failure before commit')
 db.commit()
 return True
 except BaseException:
 db.rollback()
 raise


with TemporaryDirectory(prefix='queue-sink-') as directory:
 path = Path(directory) / 'sink.db'
 with closing(sqlite3.connect(path, isolation_level=None)) as db:
 db.execute('CREATE TABLE inbox(id TEXT PRIMARY KEY, amount INTEGER NOT NULL)')
 db.execute('CREATE TABLE totals(id INTEGER PRIMARY KEY, amount INTEGER NOT NULL)')
 db.execute('INSERT INTO totals VALUES (1, 0)')
 try:
 apply(db, 'work-1', 100, interrupt=True)
 except RuntimeError:
 pass
 else:
 raise AssertionError('controlled failure did not occur')
 assert db.execute('SELECT count(*) FROM inbox').fetchone() == (0,)
 assert db.execute('SELECT amount FROM totals').fetchone() == (0,)
 assert apply(db, 'work-1', 100)
 assert apply(db, 'work-2', 50)
 # Close and reopen the real database rather than retaining an in-memory set.
 with closing(sqlite3.connect(path, isolation_level=None)) as db:
 assert not apply(db, 'work-1', 100)
 try:
 apply(db, 'work-1', 999)
 except ValueError:
 pass
 else:
 raise AssertionError('conflicting payload reused the same identity')
 assert db.execute('SELECT count(*) FROM inbox').fetchone() == (2,)
 assert db.execute('SELECT amount FROM totals').fetchone() == (150,)
 assert not db.in_transaction
 print('rollback kept marker and effect together; reopened retry changed nothing; total=150')

A controlled exception after both writes rolls back both the marker and total. Retrying then commits two operations. After closing and reopening the database, replaying the first identity changes nothing, while reusing it with a different amount raises an error. The final state contains two identities and a total of 150, outside any open transaction.

The experiment verifies rollback, committed-state persistence across connection reopening, and duplicate handling for this SQLite sink. It does not kill the process, simulate power failure, test concurrent consumers, or exercise broker publication and acknowledgment. Test those boundaries separately with the intended service before making a broader delivery claim.

Sources

Amazon SQS standard queues, at-least-once delivery and duplicate handling, checked 2026-10-15.
Amazon SQS visibility timeout, retry eligibility and duplicate limits, checked 2026-10-15.
Google Pub/Sub exactly-once delivery, scoped guarantees and publish-side duplicates, checked 2026-10-15.
SQLite transactions, explicit transactions, commit and rollback, checked 2026-10-15.
Python assert, optimization removes assertion evaluation, checked 2026-10-18.

Idempotency keys are client promises

Wed, 20 Jan 2027 00:00:00 +0000

An idempotency key names an intended action, not a transmission attempt. When a service supports it, the client sends the same key when retrying that action after an uncertain result. Generating a new key for every retry tells the server something different: this is another action, which may legitimately produce another business effect.

Create once, retain across attempts

Generate the key before the first attempt and keep it with the operation’s parameters and target. If a job can resume after a client restart, its operation record needs to retain that identity too. An identifier held only in a temporary network connection cannot reliably connect later attempts to the original intent.

Use a new key for a genuinely new action, even when the parameters happen to match a previous purchase. A body hash alone cannot distinguish “buy the same item again” from “retry the first purchase.” Keys and request fingerprints have different jobs: one identifies the action, while the other detects incompatible reuse.

Follow the provider’s format and scope. Stripe suggests random identifiers such as UUIDv4 and warns against sensitive data in keys. A key is not authentication, and possession of it must not replace authorization to access a stored result. Do not put bearer credentials or personal details into an identifier intended for correlation.

The receiver must implement the promise

Merely adding an Idempotency-Key request header does not implement deduplication. The service must recognize the identifier under its documented contract. That contract should explain which parameters are compared, which results are retained, what happens while execution is in progress, and how long records remain available.

Stripe API v1 documents saving the first executed request’s status and body, including a 500 response. It does not save results for validation failures or conflicts with concurrently executing requests when endpoint execution has not begun. Therefore, “every duplicate waits for the first result” is not a universal rule.

Stripe API v2 instead can retry failed or partially failed work and return updated results. Do not transfer v1’s saved-response rule to v2. Nor does every provider replay identical response bytes. EC2 documents that a successful retry avoids additional action while its result can contain updated information, such as creation status. RFC 9110’s definition concerns intended effect; a provider’s result-replay policy is an additional contract.

Keep meaning stable

Reusing one key with incompatible parameters should follow the service’s explicit error behavior, not silently create a different operation or overwrite the saved result. JSON field ordering, defaults, numerical representation, and omitted fields complicate comparisons. Do not claim that arbitrary serialized bodies are semantically equivalent merely because one small example canonicalizes them.

The example accepts exactly a product string and a positive integer quantity. Sorting these two fields defines its comparison rule; it is not a general JSON equivalence algorithm. Its 409 conflict is a fixture policy, not a status guaranteed by every idempotent API.

When it breaks

A timeout does not prove that execution failed. Retrying with a fresh key can turn a lost successful reply into a duplicate purchase. Similarly, switching keys to escape a remembered error can create new work while the original outcome remains uncertain. Resolve the original action using the provider’s documented recovery path.

Retention also bounds the promise. Stripe API v1 permits pruning keys after they are at least twenty-four hours old and treats reuse after pruning as a new request. Do not read that as permanent identity storage or an exact universal expiration timer. Align client recovery windows with the actual service contract.

Preserve one action key across uncertain retries; use the provider’s parameter, result and retention contract. The source’s sequential simulation preserves identity through serialization but does not test a real restart or durable concurrent service.

Check it quickly

Save the block as intent_check.py and run python3 intent_check.py with Python 3.10 or newer. It ran with Python 3.14.4 and requires only the standard library. It writes one intent record inside a temporary directory, reads it back, and removes the directory on exit. The API and response loss are explicit sequential simulations, not network requests.

import json
from pathlib import Path
from tempfile import TemporaryDirectory
from uuid import uuid4


def new_intent(parameters):
 return {"key": str(uuid4()), "parameters": dict(parameters)}


class FixtureAPI:
 # One already authenticated scope, sequential execution, no network service.
 def __init__(self):
 self.saved = {}
 self.orders = []
 self.attempts = []

 def execute(self, intent):
 key, parameters = intent["key"], intent["parameters"]
 self.attempts.append(key)
 if (set(parameters) != {"product", "quantity"}
 or not isinstance(parameters["product"], str) or not parameters["product"]
 or type(parameters["quantity"]) is not int or parameters["quantity"] < 1):
 return 400, '{"error":"invalid parameters"}'
 fingerprint = json.dumps(parameters, sort_keys=True, separators=(",", ":"))
 if key in self.saved:
 original, response = self.saved[key]
 return response if original == fingerprint else (409, '{"error":"key reused with other parameters"}')
 self.orders.append(dict(parameters))
 response = (201, json.dumps({"order_id": len(self.orders)}, separators=(",", ":")))
 self.saved[key] = (fingerprint, response)
 return response


def deliver(api, intent, lose_response=False):
 response = api.execute(intent)
 if lose_response:
 raise TimeoutError("simulated response loss after the fixture saved its result")
 return response


api = FixtureAPI()
parameters = {"product": "notebook", "quantity": 2}
intent = new_intent(parameters)

with TemporaryDirectory(prefix="idempotency-intent-") as directory:
 record = Path(directory) / "intent.json"
 with record.open("x", encoding="utf-8") as stream:
 json.dump(intent, stream)
 try:
 deliver(api, intent, lose_response=True)
 except TimeoutError:
 pass
 else:
 raise AssertionError("expected simulated response loss")
 assert len(api.orders) == 1
 with record.open(encoding="utf-8") as stream:
 recovered = json.load(stream)
 assert recovered == intent
 first_known_result = deliver(api, recovered)
 assert first_known_result == (201, '{"order_id":1}')
 assert deliver(api, recovered) == first_known_result
 assert len(api.orders) == 1 and api.attempts[:3] == [intent["key"]] * 3
 print("one persisted intent survives serialization and three attempts produce one order")

 reordered = {"key": intent["key"], "parameters": {"quantity": 2, "product": "notebook"}}
 assert deliver(api, reordered) == first_known_result
 changed = {"key": intent["key"], "parameters": {**parameters, "quantity": 3}}
 assert deliver(api, changed)[0] == 409 and len(api.orders) == 1
 assert deliver(api, recovered) == first_known_result
 print("equivalent field ordering replays; changed parameters conflict without replacing the saved result")

 mistaken_retry = new_intent(parameters)
 assert mistaken_retry["key"] != intent["key"]
 assert deliver(api, mistaken_retry) == (201, '{"order_id":2}')
 assert len(api.orders) == 2
 print("generating a new key for an uncertain retry creates a second order")

 invalid = new_intent({"product": "notebook", "quantity": 0})
 assert deliver(api, invalid)[0] == 400
 assert invalid["key"] not in api.saved and len(api.orders) == 2
 corrected = {**invalid, "parameters": {"product": "notebook", "quantity": 1}}
 assert deliver(api, corrected) == (201, '{"order_id":3}')
 assert len(api.orders) == 3
 print("this fixture does not save pre-execution validation failures; the corrected request can execute")

The fixture saves the first result before simulating a timeout. Reading the saved intent and reusing its key gives three attempts but one order. Reordered fields replay the same result; changed quantity conflicts without replacing it. A fresh key for the same uncertain operation creates order two.

A validation failure is not retained in this fixture, so correcting its invalid quantity can execute with that still-unused key. That illustrates this stated policy, not permission to change an already executed request.

The file round trip verifies identity preservation through serialization. It does not test an actual process restart, crash-safe client storage, durable server receipts, concurrent execution, real transport loss, or provider APIs. Those boundaries must be validated before treating an application deployment as retry-safe.

Sources

Stripe: Idempotent requests, stable keys, API v1 retention and pre-execution failures, and API v2 differences, checked 2026-10-15.
EC2 API idempotency, client-token semantics and results that can reflect updated status, checked 2026-09-08.
RFC 9110, section 9.2.2 defines intended effects and conditions for nonidempotent-method retries, checked 2026-09-08.

Graceful shutdown stops work, not life

Sat, 19 Dec 2026 00:00:00 +0000

Graceful shutdown is a transition in how a service accepts and finishes work. Receiving a signal is only the trigger. The useful guarantee is narrower than “nothing is lost”: stop admitting new business work, account for what was already accepted, allow bounded completion, and report honestly when that bound expires.

A signal begins coordination

SIGTERM can be handled so an application starts its shutdown sequence. SIGKILL cannot be caught, blocked, or ignored. Verify which process receives the stop signal: a handler inside a worker does not help if a wrapper receives the signal and never forwards it.

Keep the signal handler small. Python executes its handlers in the main interpreter thread and warns against synchronization locks inside them. The fixture only sets a flag; ordinary control flow changes readiness and admission under a lock. It does not perform network I/O or wait for workers inside the handler.

Readiness should be withdrawn early enough for routing changes to begin. Admission still needs its own local gate, because existing connections and delayed routing updates can deliver requests afterward. Changing the gate and registering accepted work must use a consistent boundary.

The deadline covers a lifecycle

Track work until its real completion point. A caller timeout is not proof that a worker stopped, and a sent response is not proof that a business transaction is durable. Select the completion boundary appropriate to the operation instead of treating every closed socket as success.

Use a monotonic deadline for local elapsed time. Leave room for cleanup within the supervisor’s actual allowance. When the deadline expires, remaining work needs an explicit cancellation, abandonment, or durable handoff policy. Merely breaking out of a wait loop does not cancel a running thread.

Successful draining and incomplete shutdown should remain distinguishable in records. A particular nonzero exit code is a service contract, not a universal rule that guarantees correct recovery. A client losing a response may still need idempotency or reconciliation before retrying.

Coordinate with the supervisor

In systemd, TimeoutStopSec governs configured stop-command and service-stop waits. KillMode determines the processes targeted, not how much drain time they receive. With mixed mode, the initial termination signal targets the main process and final escalation targets remaining control-group processes; early main-process exit can also trigger that escalation. Review stop hooks and notification-based extensions in the real unit.

Kubernetes begins the termination grace period before running a configured preStop hook. The hook does not receive an independent extra drain budget. EndpointSlice termination updates and local shutdown proceed alongside each other; they are not an instantaneous global barrier after which no request can arrive.

The normal stop signal can depend on image or runtime configuration. Test the deployed process tree and stop contract rather than assuming the behavior of a locally executed program also proves container, sidecar, and proxy coordination.

When it breaks

A shutdown flag that only hides readiness can leave a keep-alive connection admitting new work forever. Conversely, closing every active socket immediately can discard requests that had enough time to finish. Stopping admission and finishing accepted work are separate decisions.

An unbounded join can exceed the supervisor’s allowance. Daemon threads allow process exit without waiting, but that is abandonment, not safe cancellation or transaction rollback. Libraries have their own shutdown contracts: Python’s socketserver shutdown must be called from a different thread than serve_forever or it deadlocks.

Observe accepted, completed, refused, and unfinished work, together with shutdown duration and exit outcome. A clean-looking exit without those distinctions can hide dropped work. The goal is a verified lifecycle, not just a registered signal handler.

Stop new business admission while counting accepted work to its real completion boundary. The source distinguishes a finished request from deliberate daemon-worker abandonment; it does not validate deployed supervisor coordination.

Check it quickly

Save this code as shutdown_check.py and run python3 shutdown_check.py on a Unix-like system with Python 3.10+, keeping assertions enabled. Both language versions ran once on 2026-10-18 using CPython 3.14.4 / Darwin 25.5.0 arm64; other versions remain untested. Each case creates a child and loopback HTTP server; SIGTERM targets that child. POST release coordinates testing, not production administration.

from concurrent.futures import ThreadPoolExecutor
from http.client import HTTPConnection, RemoteDisconnected
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import selectors
import signal
import subprocess
import sys
from threading import Condition, Event, Thread
import time


def child(grace):
 state = {"draining": False, "active": 0}
 condition, release = Condition(), Event()
 stop_requested = False

 def stop(signum, frame):
 nonlocal stop_requested
 stop_requested = True # No lock, I/O, or shutdown call in the signal handler.

 class Handler(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def reply(self, status, value):
 body = json.dumps(value).encode()
 self.send_response(status)
 self.send_header("Content-Type", "application/json")
 self.send_header("Content-Length", str(len(body)))
 self.send_header("Cache-Control", "no-store")
 self.end_headers()
 self.wfile.write(body)

 def do_GET(self):
 if self.path == "/state":
 with condition:
 snapshot = dict(state)
 self.reply(200, snapshot)
 elif self.path == "/readyz":
 with condition:
 ready = not state["draining"]
 self.reply(200 if ready else 503, {"ready": ready})
 elif self.path == "/work":
 with condition:
 admitted = not state["draining"]
 if admitted:
 state["active"] += 1
 if not admitted:
 self.reply(503, {"error": "draining"})
 return
 try:
 if release.wait(timeout=5):
 self.reply(200, {"done": True})
 else:
 self.reply(503, {"error": "fixture-timeout"})
 finally:
 with condition:
 state["active"] -= 1
 condition.notify_all()
 else:
 self.reply(404, {"error": "not-found"})

 def do_POST(self):
 if self.path == "/release": # Loopback-only test coordination, not workload admission.
 self.reply(200, {"released": True})
 release.set() # Finish the control reply before work can complete and the child exits.
 else:
 self.reply(404, {"error": "not-found"})

 signal.signal(signal.SIGTERM, stop)
 server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
 server.daemon_threads = True # Explicitly permits abandonment in the deadline case.
 serving = Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01})
 serving.start()
 print(json.dumps({"port": server.server_port}), flush=True)
 while not stop_requested:
 time.sleep(0.005)
 deadline = time.monotonic() + grace
 with condition:
 state["draining"] = True
 while state["active"] and time.monotonic() < deadline:
 condition.wait(timeout=max(0, deadline - time.monotonic()))
 unfinished = state["active"]
 server.shutdown() # Called outside the thread running serve_forever.
 serving.join(timeout=2)
 server.server_close() # Does not wait for daemon request threads.
 assert not serving.is_alive()
 print(json.dumps({"unfinished": unfinished, "deadline_expired": unfinished > 0}), flush=True)
 return 3 if unfinished else 0 # Fixture policy, not a universal exit-code convention.


def request(port, route, method="GET"):
 connection = HTTPConnection("127.0.0.1", port, timeout=3)
 try:
 connection.request(method, route)
 response = connection.getresponse()
 return response.status, json.loads(response.read())
 finally:
 connection.close()


def wait_state(port, predicate):
 deadline = time.monotonic() + 3
 while time.monotonic() < deadline:
 status, state = request(port, "/state")
 assert status == 200
 if predicate(state):
 return state
 time.sleep(0.01)
 raise AssertionError("child did not reach the expected state")


def exercise(finish):
 process = subprocess.Popen(
 [sys.executable, __file__, "child", "3" if finish else "1"],
 stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
 )
 try:
 with selectors.DefaultSelector() as selector:
 selector.register(process.stdout, selectors.EVENT_READ)
 assert selector.select(timeout=5), "child did not announce its local port"
 port = json.loads(process.stdout.readline())["port"]
 assert request(port, "/readyz") == (200, {"ready": True})
 with ThreadPoolExecutor(max_workers=1) as pool:
 active = pool.submit(request, port, "/work")
 wait_state(port, lambda state: state["active"] == 1)
 process.send_signal(signal.SIGTERM)
 wait_state(port, lambda state: state["draining"])
 assert request(port, "/readyz") == (503, {"ready": False})
 assert request(port, "/work") == (503, {"error": "draining"})
 assert request(port, "/state")[1]["active"] == 1
 if finish:
 assert request(port, "/release", "POST")[0] == 200
 assert active.result(timeout=4) == (200, {"done": True})
 output, errors = process.communicate(timeout=5)
 assert errors == ""
 record = json.loads(output)
 assert process.returncode == (0 if finish else 3)
 assert record == {"unfinished": 0 if finish else 1, "deadline_expired": not finish}
 if not finish:
 try:
 active.result(timeout=4)
 except (OSError, RemoteDisconnected):
 pass
 else:
 raise AssertionError("abandoned work must not report a successful reply")
 print("completed drain" if finish else "expired drain", record, "exit", process.returncode)
 finally:
 if process.poll() is None:
 process.kill() # Safety cleanup targets only this fixture's child.
 process.wait(timeout=5)
 process.stdout.close()
 process.stderr.close()


if __name__ == "__main__":
 if sys.argv[1:2] == ["child"]:
 raise SystemExit(child(float(sys.argv[2])))
 exercise(True)
 exercise(False)

Both cases admit one request before SIGTERM. After the drain flag becomes visible, readiness returns 503, a new business request receives 503, and the admitted request remains counted.

Releasing that request lets it return 200 before the child exits zero with no unfinished work. Without release, the one-second drain allowance expires; the child reports one unfinished request, exits three, and the waiting client receives no successful reply.

The latter case deliberately abandons a daemon worker. It does not prove cooperative cancellation, durability, SIGKILL cleanup, or safe replay. No systemd unit, Kubernetes workload, real transaction, or production traffic was exercised.

The record shows four original SIGTERM calls, 34 parent-observed loopback HTTP connections, and unchanged child arguments. Expired requests ended with RemoteDisconnected. Owned processes/listeners were absent afterward, without supervisor escalation. These are parent Python API observations, not packet capture or child-thread instrumentation.

Sources

Linux signal(7), termination signals and the uncatchable SIGKILL boundary, checked 2026-09-09.
Python signals, main-thread handlers and the warning against locks in handlers, checked 2026-09-09.
Python socketserver, shutdown-thread requirements and daemon request-thread behavior, checked 2026-09-09.
systemd.service, TimeoutStopSec stop waits and notification extensions, checked 2026-09-09.
systemd.kill, KillMode process scope and termination escalation, checked 2026-09-09.
Kubernetes: Pod lifecycle, preStop, termination grace, stop signals, and EndpointSlice transitions, checked 2026-09-09.

Fsync completes a storage request within an update protocol

Fri, 27 Nov 2026 00:00:00 +0000

A successful ordinary write(2) makes its accepted bytes available through the kernel’s file state, but does not by itself establish crash durability. fsync requests synchronization of a file’s data and associated metadata and waits for completion as reported by the storage stack. It is one operation in an update protocol. Correct ordering, checked errors, directory entries, and the storage system’s actual guarantees still matter.

Conceptual Linux replacement sequence. No example was executed; fsync execution remains blocked. Successful calls alone do not establish physical power-loss guarantees. Failure does not undo earlier effects.

When it breaks

User-space buffering creates an earlier boundary. Calling fsync on a descriptor cannot synchronize bytes that a language-level buffer has not yet submitted to the kernel. Flush that buffer first, and check errors from both stages. Ordinary writes can also complete partially, and delayed storage errors may surface during synchronization instead of at the original write.

Synchronizing a newly created file does not necessarily synchronize the directory entry that names it. On Linux, a durable creation or replacement protocol also needs the appropriate directory synchronization. Atomic rename provides a namespace visibility property while the system is running; it does not independently establish that the directory change survived a crash.

fdatasync does not mean no metadata. It synchronizes data and metadata needed for subsequent data retrieval, including a changed file length. It can omit other metadata updates that are not needed for that purpose. Treating it as either identical to fsync in every respect or completely indifferent to metadata obscures the reason for choosing between them.

Check it quickly

Save this as syncfile.py and run python3 syncfile.py on a disposable Linux system with Python and strace. The temporary directory must support regular-file and directory synchronization. The original report names Linux 6.12.76, Python 3.11.2, strace 6.1, and ext4. The 2026-10-15 review could run only the inner file operations: tracing was denied. The complete trace exercise remains blocked and was not retried here; the following results describe the original report and intended assertions, not fresh trace evidence:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,sys

program=r'''
from pathlib import Path
import errno,os,sys
root=Path(sys.argv[1]);current=root/'current';candidate=root/'candidate'
current.write_bytes(b'old');old=current.open('rb')
try:
 with candidate.open('wb') as handle:
 handle.write(b'new');handle.flush();os.fsync(handle.fileno())
 os.replace(candidate,current)
 directory=os.open(root,os.O_RDONLY|os.O_DIRECTORY)
 try:os.fsync(directory)
 finally:os.close(directory)
 assert current.read_bytes()==b'new' and old.read()==b'old'
 with current.open('ab') as handle:
 handle.write(b'-extended');handle.flush();os.fdatasync(handle.fileno())
 assert current.read_bytes()==b'new-extended' and current.stat().st_size==12
 read_end,write_end=os.pipe()
 try:
 try:os.fsync(write_end)
 except OSError as error:assert error.errno==errno.EINVAL
 else:raise AssertionError('fsync on a pipe unexpectedly succeeded')
 finally:os.close(read_end);os.close(write_end)
finally:old.close()
print('replacement content, old open reference, appended length, and unsupported-pipe error verified')
'''
with TemporaryDirectory(prefix='xbcatm-fsync-') as directory:
 root=Path(directory);trace=root/'trace'
 result=subprocess.run(['strace','-yy','-s','200','-e','trace=%file,write,fsync,fdatasync','-o',str(trace),'--',sys.executable,'-c',program,str(root)],capture_output=True,text=True,timeout=10)
 assert result.returncode==0,result.stderr
 lines=trace.read_text().splitlines()
 file_sync=next(i for i,line in enumerate(lines) if 'fsync(' in line and str(root/'candidate') in line and '= 0' in line)
 rename=next(i for i,line in enumerate(lines) if line.startswith('rename') and str(root/'current') in line and '= 0' in line)
 directory_sync=next(i for i,line in enumerate(lines) if 'fsync(' in line and '<'+str(root)+'>' in line and '= 0' in line)
 assert file_sync<rename<directory_sync
 assert any('fdatasync(' in line and str(root/'current') in line and '= 0' in line for line in lines)
 print(result.stdout,end='')
 for line in lines:
 if line.startswith(('fsync(','fdatasync(','rename')):print(line)
 print('syscall order verified: flush data, fsync candidate, replace name, fsync parent directory')

The child creates a current file containing old and keeps an open read reference. It writes new to a separate candidate, flushes the Python buffer, and calls fsync on that candidate. Keeping the candidate in the same directory also keeps the later rename within the same filesystem for this exercise.

It then replaces the current pathname and synchronizes an open descriptor for the containing directory. The trace asserts the sequence: candidate fsync succeeds, rename succeeds, then parent-directory fsync succeeds. These are observed system calls with checked results, rather than a source-code comment claiming that the intended synchronization order was followed.

Reading the current path returns new, while the reference opened earlier still reads old. This distinguishes atomic name replacement from changing every existing open reference. The old reference remains attached to its earlier file object. Readers that need a new version must select it according to the application’s publication protocol.

The child next appends -extended, flushes its buffer, and calls fdatasync. The resulting content is new-extended with length twelve. This exercises successful data synchronization after a size-changing operation, consistent with the contract that metadata needed to retrieve the extended data cannot simply be ignored.

Finally, fsync on a pipe fails with EINVAL. The fixture checks that failure rather than treating every descriptor as a synchronizable regular file. All handles close during cleanup. The trace proves that the local kernel accepted the relevant file and directory requests; it does not independently prove that a particular physical device honors flushes during a power failure.

Define the acknowledgment boundary

Specify which update must survive and what the reader may observe. For a single-file replacement, synchronize the candidate before publication and the affected directory afterward. A move involving different directories can require considering both directory updates. Cross-filesystem movement has different mechanics, so a convenient rename-based protocol cannot be assumed to apply unchanged.

If synchronization fails, do not report the durable update as committed. Earlier writes or a rename may already have taken effect, so failure is not automatic rollback. Retain enough transaction or version information to determine the actual state and recover according to the application’s contract. Fsync alone does not make a sequence of changes to multiple files one atomic transaction.

Batching several operations behind a shared commit boundary can reduce synchronization cost when the application contract permits it. Acknowledge only after the required boundary succeeds. Validate crash recovery on the real storage stack, including its filesystem and device-cache behavior, because a virtualized laboratory’s successful syscall trace establishes API behavior without proving every lower layer’s failure guarantees.

Sources

fsync(2), file and directory synchronization, fdatasync metadata, and errors, checked 2026-09-12.
rename(2), atomic pathname replacement and existing open references, checked 2026-09-12.
write(2), buffered write completion, partial writes, and delayed errors, checked 2026-09-12.

Rate limiting needs a stable client identifier

Thu, 29 Oct 2026 00:00:00 +0000

A rate-limit key defines which requests share a budget. An account, API credential, address, tenant, and authenticated session are different identities, not interchangeable spellings of “client.” Choosing the wrong grouping can punish unrelated users or let one workload obtain fresh allowance simply by changing a field.

This illustrates the article’s already-authenticated, routed fixture requests in one test window. Stable account grouping does not prove per-person fairness, real authentication, trusted proxy forwarding, concurrent counting or distributed enforcement. Historical scoped execution remains passed; no example was rerun for this illustration.

Use the identity the policy actually promises

For a per-account rule, derive the key from the account resolved by authentication, not from a caller-supplied account header or query parameter. A credential’s presence is not proof that it is valid. Authentication must verify it and determine the principal before the account-level decision uses that result.

A per-credential budget is legitimate when that is the intended product contract, but it behaves differently from a per-account budget. Rotating a credential or creating another credential can produce a new credential key while retaining the same account. Decide whether that should renew allowance rather than discovering the distinction through abuse or support tickets.

One account is not necessarily one human or one application, and one person can hold several accounts. A stable account key improves consistency of grouping; it does not prove fairness, prevent account farming, or bound the service’s aggregate demand. Additional scopes need their own justified policies.

An address describes a network path

Several accounts behind a corporate NAT, mobile carrier, or VPN can share a public address. An IP bucket combines their traffic. Conversely, a caller that changes addresses can reach a fresh IP bucket. Neither result means the network address was parsed incorrectly; it means IP and principal identity represent different things.

Address-based limits can still protect public or preauthentication endpoints where no verified account exists. Size and evaluate that policy as an address-level control, including legitimate shared-network traffic. Do not describe it as exact per-person enforcement, or assume a single address can catch a scraper distributed across many networks.

Normalize address syntax before using it as a key. Equivalent IPv6 spellings should not create multiple buckets. Whether to aggregate prefixes or equate particular address families is a separate policy choice with collateral-impact tradeoffs; textual normalization does not answer it.

Forwarded identity needs a trust boundary

An arbitrary Forwarded, X-Forwarded-For, or similar header is not proof of client origin. RFC 7239 explicitly warns that the client and intervening nodes can alter forwarding information. Trust depends on the directly connected peer, the configured proxy chain, and whether that chain removes or safely handles user-supplied values.

NGINX’s real-IP module makes trusted source addresses and recursive processing explicit configuration. “Take the first address in the header” is not an adequate general rule. A proxy accepting and retransmitting an untrusted claim can turn a stable-looking key into caller-controlled budget creation.

The fixture below models exactly one trusted proxy that overwrites a field with one IP literal. It ignores replacement claims from other peers and rejects a multi-address value from that proxy. It is not a general forwarding-chain parser or a deployed NGINX configuration.

When it breaks

Omitting tenant scope can merge unrelated accounts that share an identifier. Keying on raw URLs can split the same operation by query strings; use the stable operation identity resolved by routing when the policy is per operation. Ambiguous string concatenation can also collide, so preserve component boundaries in the key encoding.

Avoid using raw bearer secrets as budget keys or diagnostic labels. Stable internal credential or principal identifiers are more appropriate, and logs still need privacy controls. Unbounded identity creation can consume limiter memory; expiration or state eviction must be evaluated alongside the extra allowance that resetting a bucket can create.

Test realistic identities before selecting thresholds: shared addresses, roaming clients, credential rotation, tenants, and forged claims. Also inspect the source of the supposedly verified principal. No choice of dictionary key can repair authentication data that was actually copied from an untrusted request.

Check it quickly

Save the block as identity_check.py and run python3 identity_check.py with Python 3.10 or newer. It ran with Python 3.12.14 and uses only the standard library. The records represent already authenticated and routed fixture requests. They are not an authentication implementation, and the counters model one test window without servers, network traffic, or file writes.

from collections import Counter
from ipaddress import ip_address


def account_key(request):
 # These two fields come from authentication and routing, not request headers.
 tenant, account = request["verified_actor"]
 return ("account", tenant, account, request["operation"])


def admit(counts, key, limit=2):
 if counts[key] >= limit:
 return False
 counts[key] += 1
 return True


alice = {"verified_actor": ("north", "alice"), "operation": "export",
 "peer": "198.51.100.8", "credential_id": "key-a", "claimed_account": "alice"}
bob = {**alice, "verified_actor": ("north", "bob"), "credential_id": "key-b"}
by_ip, by_account = Counter(), Counter()
assert [admit(by_ip, str(ip_address(request["peer"]))) for request in (alice, alice, bob)] == [
 True, True, False,
]
assert [admit(by_account, account_key(request)) for request in (alice, alice, bob)] == [
 True, True, True,
]
print("two accounts behind one address collide in the IP budget, but not the account budget")

roaming = {**alice, "peer": "203.0.113.9", "credential_id": "key-a-rotated"}
assert admit(by_ip, str(ip_address(roaming["peer"])))
assert not admit(by_account, account_key(roaming))
for claim in ("bob", "new-account", "admin"):
 assert account_key({**roaming, "claimed_account": claim}) == account_key(alice)
print("changing address, credential ID, or an untrusted account claim does not reset this account budget")

other_tenant = {**alice, "verified_actor": ("south", "alice")}
other_operation = {**alice, "operation": "thumbnail"}
assert admit(by_account, account_key(other_tenant))
assert admit(by_account, account_key(other_operation))
assert account_key({**alice, "verified_actor": ("a:b", "c")}) != account_key(
 {**alice, "verified_actor": ("a", "b:c")}
)
print("tenant and operation are explicit dimensions; tuple encoding avoids delimiter collisions")

# One configured proxy that overwrites the header with exactly one IP literal.
trusted_proxy = ip_address("192.0.2.10")


def network_key(peer, replacement):
 peer = ip_address(peer)
 return str(ip_address(replacement)) if peer == trusted_proxy else str(peer)


assert network_key("198.51.100.8", "203.0.113.77") == "198.51.100.8"
assert network_key("198.51.100.8", "not-an-address") == "198.51.100.8"
assert network_key("192.0.2.10", "198.51.100.8") == "198.51.100.8"
assert network_key("2001:0db8:0:0::1", "ignored") == network_key("2001:db8::1", "ignored")
try:
 network_key("192.0.2.10", "198.51.100.8, 203.0.113.9")
except ValueError:
 pass
else:
 raise AssertionError("the single-address proxy contract must reject a chain")
print("untrusted peers cannot choose a forwarded identity; equivalent IPv6 text shares one key")
print("the configured one-address proxy contract rejects a multi-address value")

Alice exhausts an address budget and thereby blocks Bob at the same address, while account grouping leaves Bob’s allowance separate. Alice’s changed address creates new IP allowance, but neither that move, a credential-ID rotation, nor forged account claims reset the account key.

Separate tenants and operations receive separate keys; tuple components avoid delimiter collisions. Untrusted peers cannot choose a replacement address, equivalent IPv6 text normalizes identically, and the single-proxy contract rejects a list of addresses. Authentication, real proxy forwarding, concurrent counting, distributed quotas, and production cardinality are not exercised.

Sources

RFC 6585, section 4 leaves identity and counting scope to the limiter’s policy, checked 2026-09-08.
RFC 7239, forwarding-header integrity, trusted proxies, and information exposure, checked 2026-09-08.
NGINX real-IP module, trusted replacement addresses and recursive processing, checked 2026-09-08.

An error budget names the failures you allow

Thu, 17 Sep 2026 00:00:00 +0000

An objective such as “99.9% reliable” is incomplete until someone can classify an observation. Reliable for which users, at which boundary, over what window, and according to which outcome? An error budget gives the permitted failures a quantity only after those questions have answers. Its first engineering task is defining the population and the good-event rule, not choosing a dashboard color.

The diagram uses the article’s synthetic HTTP rule and exact set counts. Circle areas are schematic; the labeled counts, overlap and arithmetic carry the quantities. The code was read without execution, and production collection completeness is not established.

Count the experience you promise

A service level indicator is a measured quantity; an objective specifies its target and window. For an event-based indicator, divide good eligible events by all eligible events. “Good” might mean a completed business operation within a latency threshold. An HTTP success code alone cannot prove that a checkout was recorded correctly, and an internal server timer may omit the network delay experienced by a customer.

Write the denominator explicitly. It might include all valid customer attempts to one API, while excluding documented synthetic probes. Decide how retries, client cancellations, authentication failures, timeouts, and asynchronous completion are represented. Exclusions should follow the promised service boundary, not be invented after an incident to improve the score. Retain enough evidence to distinguish a deliberately excluded event from one the collector failed to observe.

One bad event can fail several tests

Suppose a good request must return the expected response within 300 milliseconds. A slow server error fails both requirements but remains one bad request. Count the union of failure conditions, or equivalently the complement of the good predicate. Adding separate slow-request and error counters can double-count overlapping failures unless the categories are explicitly disjoint.

For a target S and N eligible requests, the allowance is N multiplied by one minus S. Remaining budget subtracts measured bad requests from that allowance. A request-based budget tracks events, whereas a time-based budget tracks time in a defined state. A 99.9% time objective over thirty days corresponds to 43.2 minutes of bad time; that conversion does not describe arbitrary request traffic. Partial failures and changing load break that equivalence.

A rolling window also changes as old events leave, even when no new request fails. Recompute the allowance and failures over the same window rather than carrying a lifetime debit forward. A release policy must say what a negative balance changes operationally; the arithmetic alone neither authorizes more failures nor automatically blocks a deployment.

When it breaks

Server-only records can miss requests rejected before reaching the application. Measurements of completed requests can also omit requests still hung in a queue. A beautiful ratio over this incomplete population may coexist with a customer outage. Measure at a suitable boundary, include timeout outcomes, and expose coverage or independent checks rather than treating unavailable telemetry as successful service.

Zero eligible events do not produce a meaningful success fraction. Treat the window as having no observations and handle monitoring absence separately. Very small populations also make high targets sensitive to individual failures: the allowance is a mathematical quantity and need not be a whole event. Do not round it upward to manufacture tolerance. Keep the same population and weighting when comparing instances; a simple average of their percentages changes the meaning.

Check it quickly

Save this as slo-population.py and run Python 3. The complete unchanged program passed on 2026-10-19 with Python 3.12.14. The retained figure’s no-execution note describes its earlier artwork review. The synthetic API contract accepts only status 200 within 0.300 seconds. The fixture contains completed customer events and separately identified probes; it tests the classification math, not the completeness of a real collector or the business correctness of an HTTP response.

from fractions import Fraction

# Synthetic events: (kind, status, duration_seconds).
events = ([('user', 200, 0.10)] * 9970
 + [('user', 200, 0.50)] * 10
 + [('user', 503, 0.02)] * 10
 + [('user', 503, 0.50)] * 10
 + [('probe', 503, 0.50)] * 100)
eligible = [(status, duration) for kind, status, duration in events if kind == 'user']
bad_status = sum(status != 200 for status, _ in eligible)
too_slow = sum(duration > 0.30 for _, duration in eligible)
bad = sum(status != 200 or duration > 0.30 for status, duration in eligible)
good = len(eligible) - bad
target = Fraction(999, 1000)
allowed = len(eligible) * (1 - target)
remaining = allowed - bad

assert len(eligible) == 10000 and bad_status == too_slow == 20
assert bad == 30 and bad_status + too_slow == 40
assert allowed == 10 and remaining == -20
assert Fraction(good, len(eligible)) == Fraction(997, 1000)
# The inclusive threshold is part of the synthetic contract.
assert not (200 != 200 or 0.30 > 0.30)
assert 200 != 200 or 0.300001 > 0.30
# Aggregation must weight each eligible event, not each instance equally.
assert Fraction(999 + 0, 999 + 1) == target
assert (Fraction(999, 999) + Fraction(0, 1)) / 2 != target
assert 1 * (1 - target) == Fraction(1, 1000)
print(f"eligible={len(eligible)}; good={good}; bad_union={bad}")
print(f"status_failures={bad_status}; slow={too_slow}; wrong_sum={bad_status + too_slow}")
print(f"SLI={good / len(eligible):.3%}; allowed={float(allowed):.0f}; remaining={float(remaining):.0f}")

There are 10,000 eligible customer events and 100 excluded probes. Twenty eligible events have an unexpected status and twenty are slow, but ten belong to both sets. The correct union contains thirty bad events, leaving 9,970 good ones. Adding the two failure counts would incorrectly report forty.

The printed SLI is 99.700%, below the 99.9% target. The request allowance is ten and the balance is minus twenty. Those are event counts, not outage seconds. Inspecting individual categories remains useful for diagnosis, but the objective must use its single declared predicate. Before connecting this calculation to alerts or a release policy, verify that the production counter uses the same scope, threshold boundary, window, and missing-data treatment.

Sources

Google SRE: Service Level Objectives, SLI populations, measurement, and targets, checked 2026-09-13.
Google SRE: Monitoring Distributed Systems, explicit and implicit request failures, checked 2026-09-13.
Google SRE Workbook: Alerting on SLOs, event-based objectives and budget consumption, checked 2026-09-13.

Clock synchronization bounds time disagreement

Wed, 09 Sep 2026 00:00:00 +0000

Clock synchronization reduces disagreement between machines’ civil-time readings. It supports certificate validity checks, timestamp interpretation, and many expiration policies, but it does not establish a perfect global clock. A host can emit well-formatted UTC while measuring the wrong instant. Applications still need to distinguish elapsed durations, absolute deadlines, and causal ordering rather than making them all depend on raw wall-clock comparisons.

Four timestamps estimate two quantities

An NTP exchange records client send time T1, server receive time T2, server transmit time T3, and client receive time T4. The usual offset estimate is ((T2-T1)+(T3-T4))/2; a positive result means the server appears ahead. The round-trip delay estimate is (T4-T1)-(T3-T2), subtracting time spent processing at the server. These are estimates from observations, not direct measurements of every one-way link delay.

If forward and reverse paths have unequal delay, the offset estimate inherits an error. A small round-trip measurement can be useful without proving symmetry. A synchronization daemon uses repeated samples, selection, filtering, and a clock model rather than treating one packet as unquestionable truth. Source reachability, uncertainty, and source trust all matter; a low stratum number alone is not an accuracy certificate.

Timestamp representation also differs from measurement accuracy. A fractional field can encode very fine increments while the underlying clock or network observation remains much less certain. Printing additional decimal places does not improve that evidence.

Clock control is another stage. A daemon can gradually slew corrections, and configured conditions can permit stepping the clock. Do not promise that NTP never produces a wall-clock discontinuity or always reaches sub-millisecond accuracy. The network, oscillator, platform, and configuration determine achievable behavior. Choose monitoring bounds from application needs and measured conditions instead of adopting one universal offset threshold.

When it breaks

A wall clock can move backward or forward, making naive elapsed-time subtraction negative or misleading. Use a suitable monotonic clock for a local duration or timeout. Its reference point is not a portable timestamp for another host or a later boot. Certificate dates and business deadlines still need civil-time semantics, so monotonic time does not replace all timestamp handling.

Malformed, unrelated, unsynchronized, or policy-denied NTP responses must not be accepted as ordinary samples. Matching a response’s origin timestamp to the request helps reject unrelated packets, but it is not cryptographic authentication. Production clients need the protocol’s fuller validation and source-selection behavior. Synchronization also does not turn a timestamp-based lease into a safe distributed lock or guarantee that close events sort in causal order.

A four-timestamp exchange estimates offset; the two synthetic paths have equal round trips but different offset error.

Check it quickly

Use a fresh disposable Debian 12 Docker container with Python 3 and chrony installed; the package setup is apt-get update followed by apt-get install -y python3 chrony. Save the code as ntp-exchange.py and run it as root with python3 ntp-exchange.py. The original report used chrony 4.3 and Python 3.11.2; these package commands do not pin those versions. Run with assertions enabled, without -O or PYTHONOPTIMIZE. Drop SYS_TIME capability and avoid host networking when creating the container.

from pathlib import Path
from tempfile import TemporaryDirectory
import json
import os
import socket
import struct
import subprocess
import time

assert os.geteuid() == 0 and Path('/.dockerenv').exists(), 'use a disposable root Linux container'
NTP_EPOCH = 2208988800
ERA = 2**32


def stamp(unix_time):
 value = unix_time + NTP_EPOCH
 whole = int(value)
 return struct.pack('!II', whole % ERA, int((value - whole) * ERA))


def seconds(data, reference):
 whole, fraction = struct.unpack('!II', data)
 era = round((reference + NTP_EPOCH - whole) / ERA)
 return whole + era * ERA - NTP_EPOCH + fraction / ERA


def estimate(t1, t2, t3, t4):
 return ((t2 - t1) + (t3 - t4)) / 2, (t4 - t1) - (t3 - t2)


def validate(data, sent):
 if len(data) < 48:
 raise ValueError('short packet')
 if data[0] >> 6 == 3:
 raise ValueError('unsynchronised source')
 if (data[0] & 7) != 4 or ((data[0] >> 3) & 7) != 4:
 raise ValueError('unexpected mode or version')
 if not 1 <= data[1] <= 15:
 raise ValueError('unusable stratum')
 if data[24:32] != sent:
 raise ValueError('origin timestamp mismatch')


with TemporaryDirectory(prefix='ntp-exchange-') as directory:
 root = Path(directory)
 with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as reservation:
 reservation.bind(('127.0.0.1', 0))
 port = reservation.getsockname()[1]
 config = root / 'chrony.conf'
 config.write_text(f'pidfile {root}/pid\ncmdport 0\nbindcmdaddress {root}/command.sock\n'
 f'port {port}\nbindaddress 127.0.0.1\nlocal stratum 8\nallow 127.0.0.1\n')
 process = subprocess.Popen(['chronyd', '-x', '-n', '-u', 'root', '-f', str(config)],
 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 try:
 deadline = time.monotonic() + 5
 while not (root/'command.sock').exists():
 assert process.poll() is None
 if time.monotonic() > deadline:
 raise AssertionError('chronyd did not start')
 time.sleep(0.05)
 with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as client:
 client.settimeout(5)
 client.connect(('127.0.0.1', port))
 request = bytearray(48)
 request[0] = 0x23
 t1, start = time.time(), time.monotonic()
 sent = stamp(t1)
 request[40:48] = sent
 client.send(request)
 data = client.recv(512)
 t4, finish = time.time(), time.monotonic()
 validate(data, sent)
 t2, t3 = seconds(data[32:40], t1), seconds(data[40:48], t1)
 offset, delay = estimate(t1, t2, t3, t4)
 assert data[1] == 8 and finish >= start
 bad_origin = bytearray(data); bad_origin[24:32] = bytes(8)
 alarm = bytearray(data); alarm[0] |= 0xC0
 denied = bytearray(data); denied[1] = 0
 rejected = []
 for candidate in (data[:40], bad_origin, alarm, denied):
 try:
 validate(candidate, sent)
 except ValueError as error:
 rejected.append(str(error))
 else:
 raise AssertionError('invalid packet accepted')
 symmetric = estimate(100, 105.020, 105.022, 100.042)
 asymmetric = estimate(100, 105.010, 105.012, 100.042)
 assert abs(symmetric[0] - 5) < 1e-9 and abs(symmetric[1] - 0.040) < 1e-9
 assert abs(asymmetric[0] - 4.990) < 1e-9 and abs(asymmetric[1] - 0.040) < 1e-9
 print(json.dumps({'actual_loopback_sample': {'stratum': data[1],
 'offset_seconds': offset, 'delay_seconds': delay,
 'monotonic_elapsed_seconds': finish-start},
 'rejected_packets': rejected,
 'synthetic_symmetric_estimate': list(symmetric),
 'synthetic_asymmetric_estimate': list(asymmetric),
 'known_synthetic_offset': 5,
 'system_clock_adjustment_disabled': True,
 'external_UTC_accuracy_tested': False}))
 finally:
 process.terminate()
 process.communicate(timeout=5)

The fixture starts chronyd with -x, which disables system-clock adjustment, and a loopback-only source using local stratum 8. It sends a real version-four client packet and checks the returned mode, stratum, origin timestamp, and alarm state. Expect a stratum-eight sample with measured offset and delay values that vary between runs. Four altered or truncated packets are rejected. Both sockets and the daemon are closed afterward.

Two separate synthetic calculations hold the true server lead at five seconds. With twenty milliseconds in each direction, the estimate is five seconds and the round-trip network delay forty milliseconds. With ten milliseconds outbound and thirty inbound, the same forty-millisecond delay accompanies an estimated lead of 4.990 seconds. The ten-millisecond error demonstrates the symmetry assumption; those values are not measurements of the loopback path.

The original report states that all assertions passed; this review did not rerun the daemon or socket exchange, and its original raw runtime log was not available. The local source shares the test host’s clock and has no external UTC reference. This experiment neither disciplines that clock nor proves external accuracy, NTS authentication, or full NTP client conformance. Keep the distinction between a valid observed exchange, a bounded estimate, and trustworthy operational time visible in diagnostics.

Sources

RFC 5905: Network Time Protocol Version 4, timestamp exchange, offset estimation and delay, checked 2026-09-13.
chronyd 4.3 manual, clock control and the no-adjustment option, checked 2026-09-13.
Python: time, wall-clock and monotonic-clock behavior, checked 2026-09-13.
Python 3.11: assert statement, optimization removes assertion checks, checked 2026-10-18.

Reliability · XB Field Notes

Backpressure works when bounded admission changes the producer’s behavior

When it breaks

Check it quickly

Preserve the admission contract across the pipeline

Sources

Cache invalidation is a push, not a prayer

Expiration is not a broadcast

Identify the scope before invalidating

When it breaks

Check it quickly

Further reading

Sources

Connection timeouts prevent silent hangs

Successful connection does not mean a useful reply

When it breaks

Check it quickly

Sources

A rota needs continuous coverage and a load review

A job that is shared

Handoff is half the craft

Rehearse before the bell rings

When it breaks

Check it quickly

Further reading

Sources

Probes tell the platform when to act

Three questions, different actions

Thresholds operate on consecutive observations

Match the mechanism to the evidence

When it breaks

Check it quickly

Further reading

Sources

Resource limits constrain execution; requests guide placement

CPU and memory fail differently

When it breaks

Check it quickly

Sources

Load shedding protects the system

Protect useful capacity

Choose a justified policy

Bound admission, not only workers

When it breaks

Check it quickly

Further reading

Sources

Retry-After tells when to try again

Interpret the value and its context

Respect the wait without waiting forever

The same delay can synchronize callers

When it breaks

Check it quickly

Further reading

Sources

Graceful degradation keeps partial value

Choose what may be missing

Make stale-data acceptance explicit

When it breaks

Check it quickly

Further reading

Sources

Retries with jitter and backoff

Bound attempts before spreading them

Exponential backoff sets a ceiling

Use one end-to-end deadline

When it breaks

Check it quickly

Further reading

Sources

Circuit breakers stop storming a down service

Name the policy, not just the states

Classify and scope failures

Recovery is a controlled experiment

When it breaks

Check it quickly

Further reading

Sources

Error budgets are a real budget

Give the account a measurement rule