Reliability

Graceful degradation keeps partial value

Graceful degradation preserves a deliberately supported subset of a service when another part fails. An article page might omit recommendations while still showing the requested article. The important word is supported: a partial answer must remain useful and truthful under its product contract. Returning something is not automatically safer than returning an explicit failure. ...

A capped retry ceiling rises and then flattens while sampled full-jitter waits vary below it and sometimes shrink. Attempt caps, deadlines, and wider retry budgets remain separate constraints.

Retries with jitter and backoff

A retry is another unit of work, not an automatic improvement in reliability. It can recover from a transient failure, but can also repeat an unsafe operation or add demand to an overloaded dependency. Decide whether the operation is eligible, how much total effort is allowed, and when that effort should happen before choosing a delay formula. ...

The local breaker model moves from closed to open after two consecutive failures, waits five seconds, then admits one trial. A successful trial returns to closed; the eight-candidate experiment gives one permit and seven refusals.

Circuit breakers stop storming a down service

A circuit breaker decides whether a new call should reach a dependency that appears unhealthy. It can reject promptly instead of spending resources on another likely failure. It does not repair that dependency, cancel calls already running, or prove that rejected work would have failed. Its value depends on the failure signals and recovery policy it actually implements. ...

A rolling thirty-day budget moves from minus 330 to plus 920 as the oldest day leaves. A separate red hold sign remains: an open incident still stops nonessential features in the example policy.

Error budgets are a real budget

An error budget makes a reliability agreement usable during planning. Product and operations choose an objective, measure failures against it, and agree what changes when the allowance is exhausted. The number supports that agreement; it does not predict whether tomorrow’s deployment will cause an incident. Shipping a release creates risk, while observed bad events consume the measured budget. ...

Three sequential states show a writable, then neither a nor b writable after fencing, then b writable after promotion. The writer counts are 1, 0, 1; the safe transition allows zero.

Failover is a practiced switch

A standby helps only if the promotion path works and the old writer cannot corrupt the new history. Failover is a decision, a procedure, and a rehearsal. A healthy-looking replica is not proof that clients can reach it or that it contains every acknowledged write. Define the required recovery time and acceptable data loss before deciding when to switch. ...

A retry enters one transaction boundary enclosing an operation identity and its business effect.

Queues define delivery; consumers must protect effects

A queue’s delivery promise describes a bounded part of a workflow. Getting a message to a consumer is different from committing its business effect. At-least-once delivery allows redelivery; at-most-once delivery permits loss rather than retrying uncertain delivery. Neither label alone guarantees that an order, notification, or database update completes exactly once under every failure. ...

Three attempts carrying the same key converge on one stored order; action identity remains stable across attempts.

Idempotency keys are client promises

An idempotency key names an intended action, not a transmission attempt. When a service supports it, the client sends the same key when retrying that action after an uncertain result. Generating a new key for every retry tells the server something different: this is another action, which may legitimately produce another business effect. ...

A local admission gate stops new work while one accepted request remains counted. Release completes it; deadline expiry explicitly abandons it.

Graceful shutdown stops work, not life

Graceful shutdown is a transition in how a service accepts and finishes work. Receiving a signal is only the trigger. The useful guarantee is narrower than “nothing is lost”: stop admitting new business work, account for what was already accepted, allow bounded completion, and report honestly when that bound expires. ...