An error budget names the failures you allow

Thu, 17 Sep 2026 00:00:00 +0000

An objective such as “99.9% reliable” is incomplete until someone can classify an observation. Reliable for which users, at which boundary, over what window, and according to which outcome? An error budget gives the permitted failures a quantity only after those questions have answers. Its first engineering task is defining the population and the good-event rule, not choosing a dashboard color.

The diagram uses the article’s synthetic HTTP rule and exact set counts. Circle areas are schematic; the labeled counts, overlap and arithmetic carry the quantities. The code was read without execution, and production collection completeness is not established.

Count the experience you promise

A service level indicator is a measured quantity; an objective specifies its target and window. For an event-based indicator, divide good eligible events by all eligible events. “Good” might mean a completed business operation within a latency threshold. An HTTP success code alone cannot prove that a checkout was recorded correctly, and an internal server timer may omit the network delay experienced by a customer.

Write the denominator explicitly. It might include all valid customer attempts to one API, while excluding documented synthetic probes. Decide how retries, client cancellations, authentication failures, timeouts, and asynchronous completion are represented. Exclusions should follow the promised service boundary, not be invented after an incident to improve the score. Retain enough evidence to distinguish a deliberately excluded event from one the collector failed to observe.

One bad event can fail several tests

Suppose a good request must return the expected response within 300 milliseconds. A slow server error fails both requirements but remains one bad request. Count the union of failure conditions, or equivalently the complement of the good predicate. Adding separate slow-request and error counters can double-count overlapping failures unless the categories are explicitly disjoint.

For a target S and N eligible requests, the allowance is N multiplied by one minus S. Remaining budget subtracts measured bad requests from that allowance. A request-based budget tracks events, whereas a time-based budget tracks time in a defined state. A 99.9% time objective over thirty days corresponds to 43.2 minutes of bad time; that conversion does not describe arbitrary request traffic. Partial failures and changing load break that equivalence.

A rolling window also changes as old events leave, even when no new request fails. Recompute the allowance and failures over the same window rather than carrying a lifetime debit forward. A release policy must say what a negative balance changes operationally; the arithmetic alone neither authorizes more failures nor automatically blocks a deployment.

When it breaks

Server-only records can miss requests rejected before reaching the application. Measurements of completed requests can also omit requests still hung in a queue. A beautiful ratio over this incomplete population may coexist with a customer outage. Measure at a suitable boundary, include timeout outcomes, and expose coverage or independent checks rather than treating unavailable telemetry as successful service.

Zero eligible events do not produce a meaningful success fraction. Treat the window as having no observations and handle monitoring absence separately. Very small populations also make high targets sensitive to individual failures: the allowance is a mathematical quantity and need not be a whole event. Do not round it upward to manufacture tolerance. Keep the same population and weighting when comparing instances; a simple average of their percentages changes the meaning.

Check it quickly

Save this as slo-population.py and run Python 3. The complete unchanged program passed on 2026-10-19 with Python 3.12.14. The retained figure’s no-execution note describes its earlier artwork review. The synthetic API contract accepts only status 200 within 0.300 seconds. The fixture contains completed customer events and separately identified probes; it tests the classification math, not the completeness of a real collector or the business correctness of an HTTP response.

from fractions import Fraction

# Synthetic events: (kind, status, duration_seconds).
events = ([('user', 200, 0.10)] * 9970
 + [('user', 200, 0.50)] * 10
 + [('user', 503, 0.02)] * 10
 + [('user', 503, 0.50)] * 10
 + [('probe', 503, 0.50)] * 100)
eligible = [(status, duration) for kind, status, duration in events if kind == 'user']
bad_status = sum(status != 200 for status, _ in eligible)
too_slow = sum(duration > 0.30 for _, duration in eligible)
bad = sum(status != 200 or duration > 0.30 for status, duration in eligible)
good = len(eligible) - bad
target = Fraction(999, 1000)
allowed = len(eligible) * (1 - target)
remaining = allowed - bad

assert len(eligible) == 10000 and bad_status == too_slow == 20
assert bad == 30 and bad_status + too_slow == 40
assert allowed == 10 and remaining == -20
assert Fraction(good, len(eligible)) == Fraction(997, 1000)
# The inclusive threshold is part of the synthetic contract.
assert not (200 != 200 or 0.30 > 0.30)
assert 200 != 200 or 0.300001 > 0.30
# Aggregation must weight each eligible event, not each instance equally.
assert Fraction(999 + 0, 999 + 1) == target
assert (Fraction(999, 999) + Fraction(0, 1)) / 2 != target
assert 1 * (1 - target) == Fraction(1, 1000)
print(f"eligible={len(eligible)}; good={good}; bad_union={bad}")
print(f"status_failures={bad_status}; slow={too_slow}; wrong_sum={bad_status + too_slow}")
print(f"SLI={good / len(eligible):.3%}; allowed={float(allowed):.0f}; remaining={float(remaining):.0f}")

There are 10,000 eligible customer events and 100 excluded probes. Twenty eligible events have an unexpected status and twenty are slow, but ten belong to both sets. The correct union contains thirty bad events, leaving 9,970 good ones. Adding the two failure counts would incorrectly report forty.

The printed SLI is 99.700%, below the 99.9% target. The request allowance is ten and the balance is minus twenty. Those are event counts, not outage seconds. Inspecting individual categories remains useful for diagnosis, but the objective must use its single declared predicate. Before connecting this calculation to alerts or a release policy, verify that the production counter uses the same scope, threshold boundary, window, and missing-data treatment.

Sources