SRE · XB Field Notes

Alertmanager routes alerts through a label-matching tree

Wed, 08 Dec 2027 00:00:00 +0000

Alertmanager selects notification receivers by walking a routing tree against an alert’s labels. The tree expresses ownership and escalation policy, while receivers describe notification integrations. A configuration can parse correctly and still send the wrong class of alert to the wrong destination. Verify the label contract and tree behavior before treating a successful reload as evidence that routing works.

Each route names one receiver; children inherit unspecified settings. Ordered matching selects receivers before a separate delivery pipeline. The diagram follows the article’s expected routing cases; its unchanged Docker/amtool fixture remains blocked and was not rerun.

Follow the tree, including its stopping rule

Every alert enters the root route, which must match all alerts. Child routes inherit settings they do not override. If no child matches, the current node handles the alert. Otherwise, traversal normally stops after the first matching child. Setting continue: true on a matching child allows subsequent siblings to be considered. It does not mean that every receiver in the configuration receives the alert.

One route node names one receiver. A receiver can contain multiple notification integrations, and multiple matching routes can also select different receivers. These are different ways to deliver to several destinations. A route cannot simply replace its receiver field with a list and expect the same behavior. Review the chosen approach for duplicate notifications and intentionally different grouping or escalation settings.

Ownership is a label contract

A route matching team="payments" needs that exact label value on the alert entering Alertmanager. An annotation named team does not satisfy the matcher. Aggregation in a Prometheus expression can remove a needed dimension before the alert reaches routing. Inspect the final alert label set, including alert relabeling, rather than assuming the raw metric’s labels survived unchanged.

Keep the root receiver useful for alerts that miss more specific children. An unmatched alert still needs an observable destination and an owner who can correct its classification. Test absent labels as well as recognized teams. A missing team on a warning can take the root route, while a critical alert without that team can still match a separate severity route.

Route order is part of behavior. A broad early route can prevent a narrower later route from being reached unless continuation is enabled. Moving a block of YAML can therefore change recipients without changing any matcher text. Review ordering changes like policy changes, and keep representative label fixtures beside the configuration so the intended selection remains explicit.

Time-based muting does not make a matching route disappear from traversal. A muted child can still stop later siblings when continuation is false. Consequently, an office-hours route is not automatically a fallback to an on-call route outside its active window. Check schedule conditions and tree traversal together before trusting an escalation policy.

When it breaks

A typo or a label renamed by an exporter can send alerts to the root receiver. A new catch-all child can capture alerts intended for a later team. Excessive continuation can create multiple notifications for one event. Diagnose these failures with the actual labels and the entire matching path, not by reading one route in isolation.

Correct receiver selection still does not establish delivery. Grouping delays, silence or inhibition, template errors, transport failures, and destination behavior can intervene afterward. Inspect routing first, then follow the selected receiver through the notification pipeline. Increasing the rule’s waiting duration will not fix an incorrect team matcher, and reducing grouping delay will not repair missing credentials at a receiver.

Check it quickly

Save this as routing.py and run python3 routing.py. It requires Docker and the public Alertmanager 0.28.1 image, which may be downloaded if absent. The real amtool validates the configuration and checks seven label cases. Receivers deliberately have names without delivery integrations, so the experiment evaluates routing without contacting any person or external service.

import subprocess
import uuid

# Requires Docker and Alertmanager 0.28.1. No receiver has a delivery integration.
script = r"""set -eu
cat > /tmp/routes.yml <<'ROUTES'
route:
 receiver: default
 group_by: [alertname, service]
 routes:
 - receiver: payments
 matchers: ['team="payments"']
 continue: true
 - receiver: pager
 matchers: ['severity="critical"']
receivers:
 - name: default
 - name: payments
 - name: pager

ROUTES
/bin/amtool check-config /tmp/routes.yml
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=payments,pager team=payments severity=critical
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=payments team=payments severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=pager team=data severity=critical
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default team=data severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default team=payments-preview severity=warning
sed 's/continue: true/continue: false/' /tmp/routes.yml > /tmp/stop.yml
/bin/amtool config routes test --config.file=/tmp/stop.yml --verify.receivers=payments team=payments severity=critical
"""
name = "xbcatm-route-test-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-s"],
 input=script, text=True, capture_output=True, timeout=30,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 raise RuntimeError(result.stderr or result.stdout)
print("Verified seven route cases, including continuation and unmatched labels.")

The first case resolves to payments,pager: the payments route matches and continues, then the critical route also matches. A payments warning selects only payments; a critical data-team alert selects pager. Unmatched and missing team values select the root receiver. The payments-preview value does not match the exact payments value.

The final check changes continuation to false and confirms that the payments critical alert now selects only payments. Each expected receiver set is enforced by --verify.receivers, so a mismatch fails execution. The container has no external network or host mounts and is removed on exit. Notification delivery requires a separate controlled test of the real receiver integration.

The operational fixture is unchanged here and was not rerun in this review. Its expected outputs and earlier observations remain distinct from current execution evidence.

Sources

Alertmanager: configuration, route inheritance, sibling continuation and receiver definitions, checked 2026-10-15.
Alertmanager 0.28.1: README, amtool configuration and routing checks, checked 2026-10-15.
Prometheus: Alertmanager, separation of routing, grouping and suppression, checked 2026-10-15.

The horizontal autoscaler scales on demand

Thu, 04 Nov 2027 00:00:00 +0000

The HorizontalPodAutoscaler adjusts a scalable workload toward configured metric targets. Its output is a desired replica count, not a guarantee that additional Pods are already serving traffic. Scheduling, image startup, initialization, readiness, and available node capacity determine how quickly that recommendation becomes useful capacity. Design the control loop together with those delays.

Source arithmetic and list-state model only. Historical Python-only execution remains passed; it did not exercise tolerance, rate policies, per-Pod handling, Kubernetes reconciliation or useful ready capacity. No new run was performed.

Metrics produce recommendations

For a simple per-Pod average, desired replicas are the ceiling of current replicas multiplied by current metric divided by target metric. A ratio of two suggests doubling before additional controls apply. A ratio below one can suggest shrinking, but tolerance, missing measurements, readiness handling, bounds, and scaling behavior can prevent or limit the action.

CPU utilization uses resource requests as its denominator. Changing requests can change utilization without changing consumed CPU, so coordinating vertical and horizontal scaling matters. The metrics API must be available and the target must support the scale subresource. HPA changes a Deployment’s count; the Deployment controller manages its ReplicaSets and Pods.

The default reconciliation interval is fifteen seconds, but that is not an end-to-end response promise. Metrics have their own collection windows and delays. Distinguish average usage across Pods from temporal averaging in the measurement pipeline, and distinguish both from stabilization of recommendations. A stale but valid number can mislead an otherwise healthy controller.

Stabilize recommendations, not the last resize

Downscale stabilization retains recent replica recommendations and selects the highest within the configured window. The default downscale window is five minutes; autoscaling/v2 behavior can customize it. This is not a simple cooldown measured from the last replica change. A high recommendation that left the count unchanged can still postpone a later shrink.

Rate policies separately bound how many Pods or what percentage may change over a period. Scale-up can therefore be limited too; it is not necessarily immediate or unlimited. Set a realistic floor for startup latency and a maximum consistent with capacity and cost. Stabilization reduces churn but does not compensate for a target that tracks the wrong bottleneck.

Choose pressure that extra replicas relieve

CPU can be useful for compute-bound requests, while queue depth or work per replica may better describe workers. Neither latency nor an arbitrary resource graph automatically makes a good control input. If all replicas wait on a saturated shared database, multiplying them can increase contention instead of reducing delay.

Evaluate what happens after one replica is added: can work move to it, does per-replica pressure fall, and does downstream capacity remain sufficient? Long-lived connections, uneven partitions, and cache warmup can all delay the intended effect. A controller cannot redistribute an application’s work simply by changing a number.

When it breaks

Missing metrics are unknown demand, not zero demand. With multiple metrics, a valid scale-up recommendation can still be used even if another source fails; a failed source prevents a shrink suggested by the remaining sources. Missing samples for individual Pods also receive conservative treatment. Use conditions and events to distinguish these cases instead of assuming every missing sample stops every action.

A maxReplicas ceiling can leave queues growing while ScalingLimited reports that the calculated recommendation was constrained. A high minimum can hide low demand. Repeated fixed replicas writes from deployment automation can fight HPA. Also inspect CPU request changes and startup readiness before blaming normal rolling updates: HPA supports Deployment rollouts and does not simply count every initializing Pod as zero-load capacity for unrestricted shrink.

Check it quickly

Save this as hpa_loop.py and run python3 hpa_loop.py. It is a standard-library model of ratio arithmetic, bounds, and a highest-recent-recommendation downscale window. It deliberately omits tolerance, rate policies, individual Pod samples, and live Kubernetes reconciliation.

import math

TARGET = 50.0
MIN_REPLICAS = 2
MAX_REPLICAS = 10
WINDOW_S = 300


def desired(replicas, value):
 raw = math.ceil(replicas * value / TARGET)
 return max(MIN_REPLICAS, min(MAX_REPLICAS, raw))


def step(replicas, history, now, value):
 if value is None:
 print(f"t={now}s metric=missing hold={replicas}")
 return replicas
 recommendation = desired(replicas, value)
 history[:] = [(t, count) for t, count in history if now - t <= WINDOW_S]
 history.append((now, recommendation))
 if recommendation < replicas:
 result = min(replicas, max(count for _, count in history))
 else:
 result = recommendation
 print(f"t={now}s raw={recommendation} {replicas}->{result}")
 return result


def main():
 history = []
 replicas = step(3, history, 0, 100)
 assert replicas == 6
 replicas = step(replicas, history, 240, 50)
 assert replicas == 6
 replicas = step(replicas, history, 400, 15)
 assert replicas == 6 # A recent recommendation, not a recent scale event.
 replicas = step(replicas, history, 541, 15)
 assert replicas == 2
 replicas = step(replicas, history, 600, None)
 assert replicas == 2
 assert desired(2, 500) == 10
 assert desired(2, 5) == 2
 print("PASS: recommendation history holds 6 at t=400, then shrinks to 2")


if __name__ == "__main__":
 main()

At time zero the model grows from three to six. At 240 seconds it recommends six again without resizing. At 400 seconds a new recommendation of two is held at six because the recent recommendation remains in the window, even though the last actual resize was more than five minutes ago. At 541 seconds that high recommendation has expired and the model shrinks to two.

Both language copies passed with Python 3.12.14 in this review, including missing-input hold and both bounds. Those assertions establish the model’s stated behavior only. For cluster tuning, compare real HPA conditions, recommendations, replica transitions, Pod readiness, and application latency over time. No live controller or metrics pipeline was exercised here.

Sources

Kubernetes horizontal pod autoscaling, how HPA derives replica counts, checked 2026-10-15.
HorizontalPodAutoscaler walkthrough, worked example with multiple metrics and status conditions, checked 2026-10-15.

Error budgets are a real budget

Thu, 25 Mar 2027 00:00:00 +0000

An error budget makes a reliability agreement usable during planning. Product and operations choose an objective, measure failures against it, and agree what changes when the allowance is exhausted. The number supports that agreement; it does not predict whether tomorrow’s deployment will cause an incident. Shipping a release creates risk, while observed bad events consume the measured budget.

Request counts and policy cases come from the article’s synthetic fixture. The bars share a request-count scale. This drawing does not execute the code or grant deployment approval; the positive balance only makes a closed-incident case eligible for review.

Give the account a measurement rule

For a request-based objective, allowed bad events equal eligible requests multiplied by one minus the target. A 99.9% objective over three million requests permits three thousand bad requests. Specify which requests count, which outcomes are bad, the measurement location, and the window before attaching a release rule. A traffic-weighted allowance is not automatically a fixed number of downtime minutes.

A rolling thirty-day account drops old observations as new ones arrive. Its denominator changes when traffic changes, so the allowance can change even when the target stays fixed. A calendar-month account resets at a calendar boundary instead. Those approaches answer different questions; switching between them during an incident makes an apparently precise balance difficult to audit. Keep the definition versioned with the policy and show the window beside the balance.

Decide the response before the argument

An example policy might hold nonessential feature releases when the budget is overspent, prioritize reliability work, and allow urgent security or incident fixes through review. Name who owns the decision, what evidence permits an exception, and when ordinary release review can resume. Google’s published example is a policy for its example service, not a universal requirement to copy every threshold or escalation rule.

Remaining budget alone does not establish release readiness. An open incident, missing monitoring, a failed regression, or unsafe rollback can still prevent a change. Conversely, a necessary repair should not be blocked merely because the incident it repairs exhausted the budget. Record these decisions explicitly. The useful result is a shared response to customer impact, not an automated mechanism for blaming the team whose dashboard turned red.

When it breaks

A team can appear to recover simply because yesterday’s large incident leaves the rolling window. The arithmetic is correct, but it says nothing about whether the defect was repaired. Changing exclusions, silently replacing unavailable samples with successes, or averaging differently sized service percentages can also create a misleading surplus. Display measurement coverage and investigate definition changes before interpreting an improved balance.

Short-window burn rate supplies another view: the observed bad-event fraction divided by the allowed fraction. A high burn can warn before the longer account is exhausted. Estimating time to exhaustion from that rate assumes comparable traffic and continued behavior; it is not a promised deadline. Pair actionable alerts with the long-window account and account for sparse traffic, rather than paging on every isolated failure.

Check it quickly

Save the following as budget-policy.py and run it with Python 3. It uses synthetic daily request counts to verify rolling-window accounting and a deliberately small example policy. It does not change a deployment system or estimate the risk of a real release.

from fractions import Fraction

target = Fraction(999, 1000)
# Every synthetic daily bin has 100,000 eligible requests.
days = [(100000, 1300)] + [(100000, 70)] * 29

def account(window):
 eligible = sum(total for total, _ in window)
 bad = sum(errors for _, errors in window)
 allowed = eligible * (1 - target)
 return bad, allowed, allowed - bad

def feature_policy(remaining, incident_open):
 return "hold" if remaining < 0 or incident_open else "eligible for review"

before = account(days)
after = account(days[1:] + [(100000, 50)])
assert before == (3330, 3000, -330)
assert after == (2080, 3000, 920)
assert feature_policy(before[2], False) == "hold"
assert feature_policy(after[2], True) == "hold"
assert feature_policy(after[2], False) == "eligible for review"
for label, (bad, allowed, remaining) in [("before", before), ("after", after)]:
 print(f"{label}: bad={bad}, allowed={float(allowed):.0f}, remaining={float(remaining):.0f}, spent={float(bad / allowed):.2%}")
print("after rotation, incident still open:", feature_policy(after[2], True))
print("after rotation, incident closed:", feature_policy(after[2], False))

The first account has 3,330 bad requests against an allowance of 3,000: 111% spent and a balance of minus 330. Removing the oldest day and adding the new day leaves 2,080 bad requests, 69.33% spent, and 920 remaining. Both windows contain the same three million eligible requests in this fixture; real traffic need not stay constant.

The policy still prints hold while an incident is open, despite the positive new balance. Only the final case becomes eligible for review, which intentionally stops short of granting deployment approval. This separates a verified calculation from the operational judgment that uses it. Review the definition, coverage, incident evidence, and release checks together before acting on the account.

Sources

Google SRE Workbook: Example Error Budget Policy, release restrictions, exceptions, and ownership, checked 2026-09-13.
Google SRE: Service Level Objectives, measurement boundaries and reliability targets, checked 2026-09-13.
Google SRE Workbook: Alerting on SLOs, burn rate and alerting windows, checked 2026-09-13.

An error budget names the failures you allow

Thu, 17 Sep 2026 00:00:00 +0000

An objective such as “99.9% reliable” is incomplete until someone can classify an observation. Reliable for which users, at which boundary, over what window, and according to which outcome? An error budget gives the permitted failures a quantity only after those questions have answers. Its first engineering task is defining the population and the good-event rule, not choosing a dashboard color.

The diagram uses the article’s synthetic HTTP rule and exact set counts. Circle areas are schematic; the labeled counts, overlap and arithmetic carry the quantities. The code was read without execution, and production collection completeness is not established.

Count the experience you promise

A service level indicator is a measured quantity; an objective specifies its target and window. For an event-based indicator, divide good eligible events by all eligible events. “Good” might mean a completed business operation within a latency threshold. An HTTP success code alone cannot prove that a checkout was recorded correctly, and an internal server timer may omit the network delay experienced by a customer.

Write the denominator explicitly. It might include all valid customer attempts to one API, while excluding documented synthetic probes. Decide how retries, client cancellations, authentication failures, timeouts, and asynchronous completion are represented. Exclusions should follow the promised service boundary, not be invented after an incident to improve the score. Retain enough evidence to distinguish a deliberately excluded event from one the collector failed to observe.

One bad event can fail several tests

Suppose a good request must return the expected response within 300 milliseconds. A slow server error fails both requirements but remains one bad request. Count the union of failure conditions, or equivalently the complement of the good predicate. Adding separate slow-request and error counters can double-count overlapping failures unless the categories are explicitly disjoint.

For a target S and N eligible requests, the allowance is N multiplied by one minus S. Remaining budget subtracts measured bad requests from that allowance. A request-based budget tracks events, whereas a time-based budget tracks time in a defined state. A 99.9% time objective over thirty days corresponds to 43.2 minutes of bad time; that conversion does not describe arbitrary request traffic. Partial failures and changing load break that equivalence.

A rolling window also changes as old events leave, even when no new request fails. Recompute the allowance and failures over the same window rather than carrying a lifetime debit forward. A release policy must say what a negative balance changes operationally; the arithmetic alone neither authorizes more failures nor automatically blocks a deployment.

When it breaks

Server-only records can miss requests rejected before reaching the application. Measurements of completed requests can also omit requests still hung in a queue. A beautiful ratio over this incomplete population may coexist with a customer outage. Measure at a suitable boundary, include timeout outcomes, and expose coverage or independent checks rather than treating unavailable telemetry as successful service.

Zero eligible events do not produce a meaningful success fraction. Treat the window as having no observations and handle monitoring absence separately. Very small populations also make high targets sensitive to individual failures: the allowance is a mathematical quantity and need not be a whole event. Do not round it upward to manufacture tolerance. Keep the same population and weighting when comparing instances; a simple average of their percentages changes the meaning.

Check it quickly

Save this as slo-population.py and run Python 3. The complete unchanged program passed on 2026-10-19 with Python 3.12.14. The retained figure’s no-execution note describes its earlier artwork review. The synthetic API contract accepts only status 200 within 0.300 seconds. The fixture contains completed customer events and separately identified probes; it tests the classification math, not the completeness of a real collector or the business correctness of an HTTP response.

from fractions import Fraction

# Synthetic events: (kind, status, duration_seconds).
events = ([('user', 200, 0.10)] * 9970
 + [('user', 200, 0.50)] * 10
 + [('user', 503, 0.02)] * 10
 + [('user', 503, 0.50)] * 10
 + [('probe', 503, 0.50)] * 100)
eligible = [(status, duration) for kind, status, duration in events if kind == 'user']
bad_status = sum(status != 200 for status, _ in eligible)
too_slow = sum(duration > 0.30 for _, duration in eligible)
bad = sum(status != 200 or duration > 0.30 for status, duration in eligible)
good = len(eligible) - bad
target = Fraction(999, 1000)
allowed = len(eligible) * (1 - target)
remaining = allowed - bad

assert len(eligible) == 10000 and bad_status == too_slow == 20
assert bad == 30 and bad_status + too_slow == 40
assert allowed == 10 and remaining == -20
assert Fraction(good, len(eligible)) == Fraction(997, 1000)
# The inclusive threshold is part of the synthetic contract.
assert not (200 != 200 or 0.30 > 0.30)
assert 200 != 200 or 0.300001 > 0.30
# Aggregation must weight each eligible event, not each instance equally.
assert Fraction(999 + 0, 999 + 1) == target
assert (Fraction(999, 999) + Fraction(0, 1)) / 2 != target
assert 1 * (1 - target) == Fraction(1, 1000)
print(f"eligible={len(eligible)}; good={good}; bad_union={bad}")
print(f"status_failures={bad_status}; slow={too_slow}; wrong_sum={bad_status + too_slow}")
print(f"SLI={good / len(eligible):.3%}; allowed={float(allowed):.0f}; remaining={float(remaining):.0f}")

There are 10,000 eligible customer events and 100 excluded probes. Twenty eligible events have an unexpected status and twenty are slow, but ten belong to both sets. The correct union contains thirty bad events, leaving 9,970 good ones. Adding the two failure counts would incorrectly report forty.

The printed SLI is 99.700%, below the 99.9% target. The request allowance is ten and the balance is minus twenty. Those are event counts, not outage seconds. Inspecting individual categories remains useful for diagnosis, but the objective must use its single declared predicate. Before connecting this calculation to alerts or a release policy, verify that the production counter uses the same scope, threshold boundary, window, and missing-data treatment.

Sources

Google SRE: Service Level Objectives, SLI populations, measurement, and targets, checked 2026-09-13.
Google SRE: Monitoring Distributed Systems, explicit and implicit request failures, checked 2026-09-13.
Google SRE Workbook: Alerting on SLOs, event-based objectives and budget consumption, checked 2026-09-13.

SRE · XB Field Notes

Alertmanager routes alerts through a label-matching tree

Follow the tree, including its stopping rule

Ownership is a label contract

When it breaks

Check it quickly

Sources

The horizontal autoscaler scales on demand

Metrics produce recommendations

Stabilize recommendations, not the last resize

Choose pressure that extra replicas relieve

When it breaks

Check it quickly

Further reading

Sources

Error budgets are a real budget

Give the account a measurement rule

Decide the response before the argument

When it breaks

Check it quickly

Further reading

Sources

An error budget names the failures you allow

Count the experience you promise

One bad event can fail several tests

When it breaks

Check it quickly

Sources