Alerting

Two versions of the same api-a alert retain the same blue identity-label block and fingerprint while the gold annotation summary changes from queue six to queue eight. Only explanatory context changes.

Annotations explain an alert without changing its identity

Alert labels identify which condition is being tracked and provide fields for routing and grouping. Annotations explain that condition: a current value, a concise summary, or a runbook link. Putting changing prose in labels can split one ongoing problem into many identities. Putting routing fields only in annotations can leave a route unable to recognize the alert’s intended owner. ...

An alert labeled payments and critical enters the root. The payments match continues to the later critical match, selecting payments and pager receivers. The arrows describe ordered selection, not proven delivery.

Alertmanager routes alerts through a label-matching tree

Alertmanager selects notification receivers by walking a routing tree against an alert’s labels. The tree expresses ownership and escalation policy, while receivers describe notification integrations. A configuration can parse correctly and still send the wrong class of alert to the wrong destination. Verify the label contract and tree behavior before treating a successful reload as evidence that routing works. ...

A gold pending interval begins at minute one, reaches a red firing state at minute three after two evaluated minutes, then becomes inactive at minute four. These are rule states, before notification delivery.

Prometheus rules track alert state before notification delivery

An alerting rule evaluates a PromQL expression and tracks the returned label sets over time. Prometheus decides whether each resulting alert is pending or firing. Alertmanager receives alert information and handles notification routing, grouping, silencing, and inhibition. A reliable investigation follows those stages separately: a firing alert is evidence of rule state, not proof that a person received a message. ...

Alertmanager grouping batches alerts while silences suppress notifications

One incident can produce many alerts, but a group is not proof that those alerts have one cause. Alertmanager groups alerts according to configured labels and routing context so their notifications can be handled together. The original alert identities remain available inside the group. Silences add another operation: they suppress matching notifications for a time window without resolving the underlying conditions. ...

The source’s sample appears at minute zero. A stale marker stops instant selection at minute one, while the prior sample can remain in a three-minute range. Range absence begins at minute three; the separate two-minute alert wait reaches firing at minute five.

Prometheus staleness and alert waiting are different mechanisms

A missing metric is not a numeric zero. Prometheus distinguishes a series with a recent value, a series marked stale, and a selector with no usable sample. Alerting then applies its own evaluation and waiting rules. Treating all these mechanisms as one universal grace period produces alerts that miss disappearing targets or report an outage without establishing what actually disappeared. ...