SRE | XB Field Notes

An alert labeled payments and critical enters the root. The payments match continues to the later critical match, selecting payments and pager receivers. The arrows describe ordered selection, not proven delivery.

Alertmanager routes alerts through a label-matching tree

Alertmanager selects notification receivers by walking a routing tree against an alert’s labels. The tree expresses ownership and escalation policy, while receivers describe notification integrations. A configuration can parse correctly and still send the wrong class of alert to the wrong destination. Verify the label contract and tree behavior before treating a successful reload as evidence that routing works. ...

A high recommendation at 240 seconds holds six replicas through a lower recommendation at 400 seconds; after that recommendation expires, the model drops to two at 541 seconds.

The horizontal autoscaler scales on demand

The HorizontalPodAutoscaler adjusts a scalable workload toward configured metric targets. Its output is a desired replica count, not a guarantee that additional Pods are already serving traffic. Scheduling, image startup, initialization, readiness, and available node capacity determine how quickly that recommendation becomes useful capacity. Design the control loop together with those delays. ...

A rolling thirty-day budget moves from minus 330 to plus 920 as the oldest day leaves. A separate red hold sign remains: an open incident still stops nonessential features in the example policy.

Error budgets are a real budget

An error budget makes a reliability agreement usable during planning. Product and operations choose an objective, measure failures against it, and agree what changes when the allowance is exhausted. The number supports that agreement; it does not predict whether tomorrow’s deployment will cause an incident. Shipping a release creates risk, while observed bad events consume the measured budget. ...

Two overlapping failure sets each contain twenty requests. Ten requests fail both tests, so the union is thirty bad requests rather than the incorrect sum of forty.

An error budget names the failures you allow

An objective such as “99.9% reliable” is incomplete until someone can classify an observation. Reliable for which users, at which boundary, over what window, and according to which outcome? An error budget gives the permitted failures a quantity only after those questions have answers. Its first engineering task is defining the population and the good-event rule, not choosing a dashboard color. ...