Monitoring · XB Field Notes

iotop shows which process is burning the disks

Wed, 15 Mar 2028 00:00:00 +0000

iotop connects task-level kernel accounting with a system-wide storage investigation. It can identify tasks reporting substantial IO or waiting, but those are different observations. A task blocked behind other work may be a victim rather than the source of contention. Kernel counters do not provide perfect attribution for every filesystem, cache, or storage layer.

Use iotop beside device and application measurements. Neither an active writer nor a high wait value alone proves a storage bottleneck. Compare the same time window before deciding which component deserves closer inspection.

Per-process counters

Check the installed implementation and version. In the cited iotop-c, main-process rows already include thread activity; -P hides separate thread rows. Other variants need their own manual. The -o option filters for observed IO activity; it does not classify harmful work. Read/write byte rates and IO-wait percentages have different units and should not be ranked as interchangeable measures.

Linux read_bytes accounts storage-layer reads; cached reads need not produce them. write_bytes is accounted when pages are dirtied, not necessarily when the device completes a write. Cancelled writes and later writeback can separate task totals from device throughput. Delay accounting records task waiting, including synchronous block-IO delay; CPU iowait is a separate aggregate and is not a reliable per-task blame measure.

See whatever sweeps the disk

Choose an explicit sort column and observe long enough to distinguish sustained work from a burst. A high write rate can be an expected backup, index build, or logging workload. Correlate process ownership, application role, and current work before changing scheduling or limits. A high rank identifies a candidate, not the correct remedy.

Resolve the task in the relevant PID namespace and verify that it still refers to the intended process. Command lines, parent relationships, and open files provide context, but an open file is not proof that all observed writes went there. PIDs can be reused between measurements; a stale number can point at unrelated work.

Watch before you act

Observation has costs and access requirements. Additional syscall tracing may perturb the process and expose arguments, so choose a separately authorized, scoped investigation when needed. An append flag belongs to opening a file; it is not a flag on each write syscall. Avoid presenting an imagined combination of outputs as a measured diagnosis.

Record collection conditions for a useful baseline: implementation, interval, process/thread mode, accumulation mode, and comparable workload. With the documented default one-second delay, -b -n 60 is roughly a minute, not five minutes. Device await includes queueing and service time; near-100-percent utilization does not establish saturation on every parallel SSD or RAID device.

When it breaks

Missing permissions, kernel accounting options, or enabled task delay accounting can cause failure or unavailable fields. An empty display does not exonerate either your processes or other users’ processes. Record the actual limitation and required environment; do not silently change capabilities or kernel settings to make the tool work. Some delay-accounting changes affect only tasks started afterward.

Bursty activity can disappear in averages, and short-lived tasks may escape sampling. Varying intervals changes what is visible. Accumulated mode summarizes observed accounting within its applicable lifetime; it does not recover an unrecorded previous hour or guarantee complete coverage of exited tasks. State the observation window with every conclusion.

Compare task bytes, task waiting and device observations over the same interval. Capture and identity checks remain unexecuted; a fresh snapshot cannot prove identity at an earlier sample.

Check it quickly

This unexecuted proposal targets the cited iotop-c interface and sysstat iostat. Confirm the installed implementations first. It captures full reports, stderr and both statuses without selecting a process from text. A fresh configuration directory prevents saved iotop preferences from silently changing the view.

(
 capture=$(mktemp -d) || exit 1
 printf 'Evidence directory: %s\n' "$capture"
 iotop --version || exit $?
 iostat -V || exit $?
 LC_ALL=C iostat -x -y -t 2 5 > "$capture/iostat.txt" 2> "$capture/iostat.err" &
 stat_pid=$!
 if LC_ALL=C XDG_CONFIG_HOME="$capture" iotop -b -P -k -t -d 2 -n 5 \
 --no-only --no-accumulated --no-accum-bw \
 > "$capture/iotop.txt" 2> "$capture/iotop.err"; then
 task_status=0
 else
 task_status=$?
 fi
 if wait "$stat_pid"; then stat_status=0; else stat_status=$?; fi
 printf 'iotop_status=%s iostat_status=%s\n' "$task_status" "$stat_status"
 cat "$capture/iotop.err" "$capture/iostat.err"
 [ "$task_status" -eq 0 ] && [ "$stat_status" -eq 0 ] || exit 1
 cat "$capture/iotop.txt" "$capture/iostat.txt"
)

In iotop-c, -P hides separate thread rows; their activity is already included in process totals. Fixed kilobyte units and non-accumulated mode support comparisons. This batch capture does not promise descending write order. Compare the labelled DISK WRITE values within each report; for an interactive ranking, explicitly select that column and descending direction. Never treat the first printed row as the largest writer.

Compare overlapping timestamped intervals; launch times are not perfectly synchronized. -y excludes iostat’s since-boot report. Missing counters, nonzero status or incomplete windows invalidate the comparison.

Save this helper as inspect_io_process.py. Select a process PID manually from the full report, in the same PID namespace. Python 3.9+, Linux pidfd support and readable proc entries are required:

"""Read-only Linux identity snapshot; requires Python 3.9+ and pidfd support."""
import json
import os
import re
import select
import sys


def snapshot(token, proc_root="/proc", ops=os, poll_factory=select.poll):
 if not re.fullmatch(r"[1-9][0-9]*", token):
 raise ValueError("PID must be a positive decimal process ID")
 pid = int(token)
 pidfd = ops.pidfd_open(pid, 0)
 directory = None
 try:
 directory = ops.open(f"{proc_root}/{pid}", os.O_RDONLY | os.O_DIRECTORY)
 watcher = poll_factory()
 watcher.register(pidfd, select.POLLIN)

 def read(name, limit):
 fd = ops.open(name, os.O_RDONLY, dir_fd=directory)
 with ops.fdopen(fd, "rb") as source:
 data = source.read(limit + 1)
 if len(data) > limit:
 raise ValueError(f"{name} exceeds the snapshot limit")
 return data

 def identity():
 head, tail = read("stat", 65536).rsplit(b")", 1)
 if int(head.split(b" ", 1)[0]) != pid:
 raise ValueError("stat PID mismatch")
 fields = tail.split() # field 3 is index 0; field 22 is index 19
 if fields[0] in (b"Z", b"X", b"x"):
 raise ProcessLookupError("process has exited")
 return int(fields[19])

 before = identity()
 executable = ops.readlink("exe", dir_fd=directory)
 raw_command = read("cmdline", 65536)
 # Remove one final NUL; empty input means no argument data.
 command = raw_command[:-1] if raw_command.endswith(b"\0") else raw_command
 command = command.split(b"\0") if raw_command else []
 after = identity()
 if before != after or watcher.poll(0):
 raise ProcessLookupError("identity changed or process exited")
 return {"pid": pid, "starttime_ticks": before, "executable": executable,
 "argv": [arg.decode("utf-8", "backslashreplace") for arg in command]}
 finally:
 if directory is not None:
 ops.close(directory)
 ops.close(pidfd)


if __name__ == "__main__":
 try:
 if len(sys.argv) != 2:
 raise ValueError("usage: inspect_io_process.py PID")
 print(json.dumps(snapshot(sys.argv[1]), ensure_ascii=True))
 except (OSError, ValueError, IndexError, AttributeError) as error:
 sys.exit(f"Identity snapshot failed: {error}")

python3 inspect_io_process.py "${PID:?Set PID from the reviewed process row}"

A successful snapshot reports an executable, arguments and stable start-time value while checking a pidfd for exit. Permission, missing-process or unavailable-pidfd errors stop the check. PID reuse before acquiring the handle cannot be excluded; this snapshot does not prove identity at the earlier sample. Proc fields can change and are not atomic. Recheck before any later action; nothing is killed or reprioritized here.

Sources

iotop man page, rendered iotop-c manual; verify the installed variant, checked 2026-10-18.
vmstat man page, documents system-wide CPU and IO-wait reporting, checked 2026-09-05.
Linux delay accounting, taskstats delay scope and enablement prerequisites, checked 2026-10-18.
Linux proc filesystem, storage byte accounting, cancelled writes, and CPU iowait limits, checked 2026-10-18.
iostat manual, first report, await, and parallel-device utilization limits, checked 2026-10-18.
iotop-c upstream manual, process aggregation, batch options, configuration and interactive sorting, checked 2026-10-19.
sysstat upstream iostat manual, -y omits the since-boot report and -t timestamps interval reports, checked 2026-10-19.
pidfd_open manual, bounded identity or controlled-operation contract, checked 2026-10-19.
Python os interfaces, bounded identity or controlled-operation contract, checked 2026-10-19.
proc_pid_stat manual, bounded identity or controlled-operation contract, checked 2026-10-19.

Grafana variables need the query language to agree

Wed, 23 Feb 2028 00:00:00 +0000

Grafana dashboard variables let one dashboard select an environment, service, or instance without duplicating every panel. A query variable obtains options from a data source; a custom variable can hold a deliberately fixed list. Neither choice is universally better. Use query-backed options when discovery should follow the source, and a maintained static list when the allowed choices are intentionally stable and small.

A variable is substituted into a query before the data source executes it. That makes its formatting part of query correctness. For Prometheus multi-value selections, use a regular-expression label matcher such as environment=~"$environment". Grafana’s Prometheus interpolation handles selected values as regular-expression alternatives with the required escaping. An exact matcher treats the resulting combined text as one label value and therefore does not implement the intended multi-selection. The variable editor’s label_values helper discovers options; it is not a PromQL function to paste into a panel expression.

Check the resulting expression

There are two relevant syntactic layers: the regular expression and the PromQL string containing it. A literal dot in production.eu must not become a wildcard that also matches productionXeu. Let the data source’s interpolation perform its documented formatting, and inspect the expression actually sent when using an explicit formatter. The raw formatter bypasses that data-source formatting; it is not a universal fix for difficult variable values.

The All option also has semantics. With no custom All value, this example expands the available, filtered options. Its menu includes production.eu and staging, so All covers those two environments. A custom expression such as .* would instead match whatever values the backend regex admits. Decide whether “all” means the menu’s current choices or a broader query, and test that decision with deliberately excluded data.

When it breaks

Query variables are not guaranteed to stay fresh forever. Their refresh setting determines when Grafana updates the option query, and time-dependent options may need refresh on time-range changes. Chained variables can still produce an empty downstream selection when the data contains no matching combination. Distinguish a legitimately empty result from a stale selection or a broken query before presenting “No data” as a service outage.

Variable filters are not authorization controls. A menu that hides an environment does not remove that environment from the data source or stop another permitted query from requesting it. Enforce access at the appropriate Grafana and backend boundaries. Shared dashboard URLs also carry selections rather than permissions; multiple values use repeated var-environment parameters, not an invented comma-separated permission list.

Grafana query formatting: mechanism and evidence boundaries.

Check it quickly

Save this as grafana-variables.py. With Python, a local Docker daemon, and both images preloaded, run python3 grafana-variables.py --serve. Disposable Grafana 12.1.0 and Prometheus 3.5.1 containers share a dedicated bridge; only Grafana is published on loopback, without host mounts. Open the printed URL. Serve mode lasts 120 seconds; --serve-seconds accepts 1–300. Control-C enters cleanup and remains an interruption, not normal completion. Docker calls have 30-second timeouts; readiness retries stop after a 20-second deadline plus the in-flight request. Cleanup checks exact owned names and fails on removal, inventory, or command-timeout or in-flight-interruption uncertainty. Without --serve, it checks only handwritten backend queries. Python disables proxy discovery and redirects; browser proxy policy remains a separate prerequisite.

import argparse
import json
import secrets
import subprocess
import time
import uuid
from http.client import HTTPException
from threading import Event
from urllib.error import HTTPError, URLError
from urllib.parse import quote
from urllib.request import HTTPRedirectHandler, ProxyHandler, build_opener

parser = argparse.ArgumentParser()
parser.add_argument("--serve", action="store_true")
parser.add_argument("--serve-seconds", type=int, default=120)
args = parser.parse_args()
if not 1 <= args.serve_seconds <= 300:
 parser.error("--serve-seconds must be between 1 and 300")

def require(condition, message):
 if not condition:
 raise AssertionError(message)

uncertain_commands = []

prefix = "xbcatm-variable-" + uuid.uuid4().hex[:10]
network, prom, grafana = [prefix + suffix for suffix in ("-net", "-prom", "-grafana")]
print(json.dumps({"owned_containers": [grafana, prom], "owned_network": network}), flush=True)


def command(*args):
 try:
 result = subprocess.run(["docker", *args], capture_output=True, text=True, timeout=30)
 except subprocess.TimeoutExpired as error:
 uncertain_commands.append(args)
 raise RuntimeError("Docker " + args[0] + " timed out") from error
 except KeyboardInterrupt:
 uncertain_commands.append(args)
 raise
 if result.returncode:
 raise RuntimeError(result.stderr.strip())
 return result.stdout.strip()


class NoRedirect(HTTPRedirectHandler):
 def redirect_request(self, req, fp, code, msg, headers, newurl):
 raise RuntimeError("Unexpected redirect in local fixture")


opener = build_opener(ProxyHandler({}), NoRedirect())


def get(path):
 with opener.open(origin + path, timeout=5) as response:
 require(response.status == 200, "Unexpected HTTP status")
 return json.load(response)


def eventually(check):
 deadline = time.monotonic() + 20
 while time.monotonic() < deadline:
 try:
 value = check()
 if value:
 return value
 except (HTTPError, URLError, TimeoutError, ConnectionError, HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Local lab did not become ready")


def cleanup_owned(groups):
 # Inspect exact owned names; absence is not inferred from a failed inspect.
 errors = []
 for kind, names, format_string in groups:
 try:
 listing = command(kind, "ls", *( ["--all"] if kind == "container" else [] ),
 "--format", format_string).splitlines()
 present = sorted(set(names) & set(listing))
 if present:
 command(kind, "rm", *( ["--force"] if kind == "container" else [] ), *present)
 except Exception as error:
 errors.append(f"{kind} removal: {error}")
 try:
 remaining = command(kind, "ls", *( ["--all"] if kind == "container" else [] ),
 "--format", format_string).splitlines()
 if set(names) & set(remaining):
 errors.append(f"{kind} remains: {sorted(set(names) & set(remaining))}")
 except Exception as error:
 errors.append(f"{kind} inventory unverified: {error}")
 if uncertain_commands:
 errors.append("A Docker command timed out or was interrupted; daemon-side completion remains uncertain")
 require(not errors, "Cleanup not verified: " + "; ".join(errors))
 print(json.dumps({"cleanup": "owned names absent in final inventories"}), flush=True)


source = {"type": "prometheus", "uid": "lab_prom"}
dashboard = {
 "uid": "variable-lab", "title": "Variable escaping lab", "schemaVersion": 41,
 "version": 1, "timezone": "utc", "refresh": "", "time": {"from": "now-5m", "to": "now"},
 "templating": {"list": [{
 "name": "environment", "label": "Environment", "type": "query", "datasource": source,
 "query": {"query": "label_values(lab_active_workers, environment)", "refId": "VariableQuery"},
 "definition": "label_values(lab_active_workers, environment)",
 "regex": "/^(production[.]eu|staging)$/", "refresh": 1, "sort": 1,
 "multi": True, "includeAll": True, "allValue": "", "options": [],
 "current": {"text": ["production.eu"], "value": ["production.eu"], "selected": True},
 }]},
 "panels": [{
 "id": number, "title": title, "type": "stat", "datasource": source,
 "gridPos": {"h": 7, "w": 12, "x": (number - 1) * 12, "y": 0},
 "targets": [{"refId": "A", "expr": expression, "instant": True, "range": False}],
 "options": {"reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": False}},
 "fieldConfig": {"defaults": {"decimals": 0}, "overrides": []},
 } for number, title, expression in (
 (1, "Selected workers", 'sum(lab_active_workers{environment=~"$environment"})'),
 (2, "Unescaped control", 'sum(lab_active_workers{environment=~"${environment:raw}"})'),
 )],
}
prom_script = """set -eu
mkdir -p /tmp/www
cat > /tmp/www/metrics.txt <<'METRICS'
# HELP lab_active_workers Synthetic active worker count.
# TYPE lab_active_workers gauge
lab_active_workers{environment="production.eu"} 10
lab_active_workers{environment="staging"} 2
lab_active_workers{environment="productionXeu"} 100
METRICS
cat > /tmp/prom.yml <<'CONFIG'
global:
 scrape_interval: 1s
scrape_configs:
 - job_name: lab
 metrics_path: /metrics.txt
 static_configs:
 - targets: ['127.0.0.1:8080']
CONFIG
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data
"""
grafana_script = """set -eu
mkdir -p /tmp/data /tmp/provisioning/datasources /tmp/provisioning/dashboards /tmp/dashboards
mkdir -p /tmp/provisioning/plugins /tmp/provisioning/alerting
cat > /tmp/provisioning/datasources/lab.yml <<'DATASOURCE'
apiVersion: 1
datasources:
 - name: Lab Prometheus
 type: prometheus
 uid: lab_prom
 access: proxy
 url: http://metrics:9090
 isDefault: true
 jsonData:
 timeInterval: 1s
DATASOURCE
cat > /tmp/provisioning/dashboards/lab.yml <<'PROVIDER'
apiVersion: 1
providers:
 - name: lab
 type: file
 options:
 path: /tmp/dashboards
PROVIDER
cat > /tmp/dashboards/lab.json <<'DASHBOARD'
""" + json.dumps(dashboard) + """
DASHBOARD
exec /run.sh
"""
try:
 # Use preloaded images; this is a local disposable fixture, not a download step.
 for image in ("prom/prometheus:v3.5.1", "grafana/grafana:12.1.0"):
 command("image", "inspect", image)
 command("network", "create", network)
 command("run", "--pull=never", "-d", "--rm", "--name", prom, "--network", network, "--network-alias", "metrics",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", prom_script)
 command("run", "--pull=never", "-d", "--rm", "--name", grafana, "--network", network,
 "-p", "127.0.0.1::3000", "-e", "GF_PATHS_DATA=/tmp/data",
 "-e", "GF_PATHS_PROVISIONING=/tmp/provisioning",
 "-e", "GF_AUTH_ANONYMOUS_ENABLED=true", "-e", "GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer",
 "-e", "GF_AUTH_DISABLE_LOGIN_FORM=true", "-e", "GF_AUTH_BASIC_ENABLED=false",
 "-e", "GF_SECURITY_ADMIN_PASSWORD=" + secrets.token_urlsafe(24),
 "-e", "GF_ANALYTICS_REPORTING_ENABLED=false", "-e", "GF_ANALYTICS_CHECK_FOR_UPDATES=false",
 "-e", "GF_ANALYTICS_CHECK_FOR_PLUGIN_UPDATES=false", "-e", "GF_PLUGINS_PREINSTALL_DISABLED=true",
 "--entrypoint", "/bin/sh", "grafana/grafana:12.1.0", "-c", grafana_script)
 origin = "http://" + command("port", grafana, "3000/tcp").splitlines()[0]
 health = eventually(lambda: get("/api/health"))
 require(health["version"] == "12.1.0" and health["database"] == "ok", "Unexpected Grafana health")
 loaded = eventually(lambda: get("/api/dashboards/uid/variable-lab"))
 require(loaded["dashboard"]["templating"]["list"][0]["multi"] is True, "Dashboard provisioning mismatch")
 checks = {}
 cases = ((r"production\\.eu", 10), ("staging", 2),
 (r"(production\\.eu|staging)", 12), ("production.eu", 110),
 ("production.eu,staging", None), (".*", 112))
 # Wait for the synthetic series to be scraped before asserting empty results.
 def ready_series():
 result = get("/api/datasources/proxy/uid/lab_prom/api/v1/query?query=" +
 quote("count(lab_active_workers)", safe=""))["data"]["result"]
 return len(result) == 1 and float(result[0]["value"][1]) == 3
 eventually(ready_series)
 for pattern, expected in cases:
 expression = 'sum(lab_active_workers{environment=~"' + pattern + '"})'
 result = get("/api/datasources/proxy/uid/lab_prom/api/v1/query?query=" +
 quote(expression, safe=""))["data"]["result"]
 value = None if not result else float(result[0]["value"][1])
 require(len(result) <= 1 and value == expected, (expression, result))
 checks[expression] = value
 print(json.dumps({"app_url": origin + "/d/variable-lab/variable-escaping-lab",
 "grafana_version": health["version"],
 "backend_handwritten_query_checks": checks,
 "dashboard_interpolation": "NOT CHECKED",
 "serve_seconds": args.serve_seconds if args.serve else 0}), flush=True)
 if args.serve:
 Event().wait(args.serve_seconds)
finally:
 cleanup_owned((("container", (grafana, prom), "{{.Names}}"),
 ("network", (network,), "{{.Name}}")))

Interrupted or timed-out commands remain uncertain despite empty inventories. Cleanup failures chain the primary exception. Repeated interruption or abrupt death can prevent cleanup; final inventories cannot exclude delayed daemon work or concurrent changes.

Explicit checks remain active under Python optimization. The Python checks submit handwritten PromQL through the data-source proxy; they do not exercise dropdown interpolation. The comma-joined and wildcard controls expect no series and 112 respectively; neither is a dropdown test. The synthetic series hold ten workers for production.eu, two for staging, and one hundred for productionXeu. With only production.eu selected, “Selected workers” must show ten, while the deliberately unescaped control shows one hundred ten. Select both available options, then All: the correct panel must show twelve in each case. The raw control becomes empty because its comma-joined text does not match the intended alternatives.

For each browser selection, retain Query Inspector’s outgoing expression and returned value alongside the displayed selection. Check repeated URL parameters when selecting both values. Backend output explicitly labels dashboard interpolation “NOT CHECKED.” The earlier author reports Chrome 152 checks of these selections and fields. Exact browser captures were not recovered for this review, and neither containers nor browser were rerun. Those historical observations concern the provisioned synthetic dashboard, not tenant isolation or changing-discovery freshness. When adapting the dashboard, keep the excluded environment as a regression case and verify the final expression whenever variable type, formatter, refresh policy, or data source changes.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
Grafana variables, variable types and dashboard reuse, checked 2026-09-13.
Grafana Prometheus template variables, multi-value regex matching and refresh behavior, checked 2026-09-13.
Grafana variable syntax, data-source interpolation and explicit formatting overrides, checked 2026-09-13.
Grafana dashboard URL variables, repeated URL parameters for multiple selected values, checked 2026-09-13.
Python urllib.request, private opener and disabled automatic proxies, checked 2026-10-19.
Python subprocess, subprocess timeout scope, checked 2026-10-19.

Prometheus rate estimates counter growth per second

Fri, 24 Dec 2027 00:00:00 +0000

A request counter answers how many events an instrumented process has accumulated. A rate answers how quickly those events are arriving. Prometheus rate estimates average growth per second over a selected range, adjusts observed counter resets, and extrapolates toward the range boundaries. The result is useful precisely because it has a time unit, but it is not an exact reconstruction of every event between scrapes.

The numbers come from the article’s synthetic promtool fixture, whose historical local execution remains distinct from this drawing. The five-minute range excludes minute zero, adjusts the observed reset and extrapolates. Rate each counter before summing. No new query engine or live Prometheus server was run.

Sampling determines what can be inferred

An ordinary counter increases until it resets, commonly after a restart. A decrease between observed samples is treated as a reset by the counter functions. A restart and recovery entirely between two scrapes may be invisible if the new value already exceeds the previous sample. More frequent scraping improves observation resolution; no query can recover an unobserved history with certainty.

Apply rate to individual counter series before aggregating them. Summing first can let one instance’s growth hide another instance’s reset. Labels define series identity, so a replacement instance with different labels starts a different series rather than continuing the old one. A gauge such as queue depth can fall normally and should not be interpreted as a resetting counter.

The selected range is part of the meaning

A range selector is open at its left boundary and closed at its right boundary. At minute five, [5m] excludes a sample exactly at minute zero. The float-counter calculation needs at least two usable samples. A range containing one sample produces no rate value; it does not establish zero traffic. Leave enough room for the configured cadence and realistic missed scrapes.

Longer windows generally smooth short bursts and delay their disappearance from the average. Shorter windows respond sooner but offer fewer observations and can become empty. Choose the window from the event duration and detection objective, then label it explicitly. irate uses the last two samples, whereas rate uses the selected range. Both can be useful on the same metric if panels explain their different questions. Prefer rate for alerting: brief dips in irate can reset a pending alert’s for timer even while load remains broadly elevated.

The query evaluation interval controls when a rule asks the question, not which samples were scraped. Refreshing a dashboard every second cannot create new observations from a target scraped every minute. Inspect these two cadences separately when assessing alert responsiveness.

When it breaks

A newly created series may vanish from a rate panel until a second sample arrives. Filling every missing result with zero can conceal collection failure. Check the raw series, timestamps, and scrape status before treating a blank panel as idle traffic. A very smooth long-window line can also hide a brief overload that matters to users.

Reset adjustment is not immunity to gaps. Missing samples can hide bursts or multiple restarts, and boundary extrapolation introduces estimation. increase applies the corresponding range estimate in event-count units; it may return a fractional count even for integer counters. Use application accounting records when exact totals are required, rather than promising that a dashboard reconstructs them.

Check it quickly

Save this as rate.py and run python3 rate.py with an installed promtool on PATH, or set PROMTOOL to its executable path. The recorded earlier execution used Python 3.12.14 and promtool 3.14.0; it was not rerun during this source review. The real query engine evaluates synthetic samples in a temporary directory; no server, listener, container or external service is started.

from pathlib import Path
from tempfile import TemporaryDirectory
import os
import shutil
import subprocess

# Uses a real, installed promtool; no server or Docker runtime is started.
promtool = shutil.which(os.environ.get("PROMTOOL", "promtool"))
if not promtool:
 raise SystemExit("Set PROMTOOL to an installed promtool executable")

tests = r"""evaluation_interval: 1m
tests:
 - interval: 1m
 input_series:
 - series: "demo_requests_total{instance=\"steady\"}"
 values: "0+60x5"
 - series: "demo_requests_total{instance=\"restart\"}"
 values: "100 160 220 20 80 140"
 - series: "demo_requests_total{instance=\"gaps\"}"
 values: "0 60 _ 180 _ 300"
 - series: "demo_requests_total{instance=\"one\"}"
 values: "_ _ _ _ _ 7"
 - series: "combined_requests_total"
 values: "100 220 340 200 320 440"
 promql_expr_test:
 - expr: "rate(demo_requests_total{instance=\"steady\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"steady\"}"
 value: 1
 - expr: "abs(rate(demo_requests_total{instance=\"restart\"}[5m]) - 5/6) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"restart\"}"
 value: 1
 - expr: "increase(demo_requests_total{instance=\"restart\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"restart\"}"
 value: 250
 - expr: "irate(demo_requests_total{instance=\"restart\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"restart\"}"
 value: 1
 - expr: "rate(demo_requests_total{instance=\"gaps\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"gaps\"}"
 value: 1
 - expr: "rate(demo_requests_total{instance=\"one\"}[5m])"
 eval_time: 5m
 exp_samples: []
 - expr: 'abs(sum(rate(demo_requests_total{instance=~"steady|restart"}[5m])) - 11/6) < bool 1e-12'
 eval_time: 5m
 exp_samples:
 - labels: '{}'
 value: 1
 - expr: 'abs(rate(combined_requests_total[5m]) - 7/3) < bool 1e-12'
 eval_time: 5m
 exp_samples:
 - labels: '{}'
 value: 1
"""
rules = r""""""

with TemporaryDirectory(prefix="prometheus-query-check-") as directory:
 root = Path(directory)
 (root / "tests.yml").write_text(tests, encoding="utf-8")
 if rules:
 (root / "rules.yml").write_text(rules, encoding="utf-8")
 result = subprocess.run([promtool, "test", "rules", "tests.yml"],
 cwd=root, text=True, capture_output=True, timeout=45)
 print(result.stdout, end="")
 if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

Successful assertions print SUCCESS. The steady series grows at one request per second. For samples 100 160 220 20 80 140, the five-minute rate is 5/6, increase is 250, and irate is one. These differ because the selected range excludes the first sample, corrects the observed reset, and extrapolates its average across the requested duration.

The deliberately linear series with gaps still yields one per second; that controlled result does not prove arbitrary gaps harmless. The one-sample series yields an empty vector. The test checks values and absence separately, using numerical tolerance for the repeating fraction. Before deploying a rule, add representative restart, sparse-traffic, and missing-data cases from the service whose counter you will monitor.

The last two checks expose an aggregation error: the steady and restarting series together yield 11/6 requests per second when rated separately. Their pre-summed sample stream yields 7/3 because reset correction is now applied to the mixture. Both calculations execute successfully; a valid query is not proof that its counter identity was preserved.

Sources

Prometheus: query functions, rate, increase, irate and reset adjustment, checked 2026-10-15.
Prometheus: querying basics, range boundaries and sample selection, checked 2026-10-15.
Prometheus: unit testing rules, synthetic samples and expression assertions, checked 2026-10-15.

Prometheus rules track alert state before notification delivery

Wed, 01 Sep 2027 00:00:00 +0000

An alerting rule evaluates a PromQL expression and tracks the returned label sets over time. Prometheus decides whether each resulting alert is pending or firing. Alertmanager receives alert information and handles notification routing, grouping, silencing, and inhibition. A reliable investigation follows those stages separately: a firing alert is evidence of rule state, not proof that a person received a message.

The timing is the article’s synthetic rule fixture: one stable queue identity, one-minute evaluations and for: 2m. Default recovery is used; keep_firing_for is absent. The historical local promtool result is preserved, with no new execution here. Grouping, suppression and delivery need separate evidence.

Vector membership activates the alert

Each element returned by the expression represents an active alert candidate. A comparison such as queue_depth > 5 filters out values that do not meet the threshold. Adding the bool modifier changes comparison behavior: false comparisons remain as elements with value zero. An alert rule tests the presence of elements, so using that modifier can activate an alert even when the number is zero.

Labels identify alert instances. Keep stable routing dimensions, such as service, queue, and severity, in labels. Put changing explanatory values in annotations. If a label includes the current queue size, every change can create a different identity and restart the pending period. An annotation can include that size without redefining which alert is being tracked.

Waiting is measured at evaluations

With for: 2m, a newly active label set is pending until it has remained active for two minutes of evaluated time. A rule evaluated every minute first seeing the condition at minute one can fire at minute three. The engine observes evaluations, not continuous physical reality between samples. The timing therefore includes the wait until the next evaluation as well as the configured duration.

Omitting for permits an active expression to fire on its first evaluation. That may suit an unequivocal failure, but it also makes a single bad sample sufficient; choose deliberately rather than relying on a forgotten default.

If the expression stops returning that label set before firing, pending state clears. With the default behavior used here, a firing alert also clears when its condition disappears. An explicitly configured keep_firing_for can extend firing after the condition clears; that option is not present in this example. Do not describe all alerts as latched incidents that require manual acknowledgement to resolve.

When it breaks

A threshold over a missing series usually returns nothing. That can clear an alert even though the application has stopped exporting data. Decide whether absence needs its own rule rather than blindly treating it as recovery. Test label changes and missing samples alongside threshold transitions. A syntax check alone cannot reveal these semantic failures.

A firing alert can produce no visible page because a silence, an inhibition rule, routing, grouping delay, or delivery failure intervened. These mechanisms belong to the notification path and do not change Prometheus’s threshold evaluation. Inspect the active alert labels and then the matching Alertmanager route and delivery evidence. Increasing for does not repair a wrong receiver or a missing route label.

Check it quickly

Save this as alert-state.py and run python3 alert-state.py with an installed promtool in PATH, or set PROMTOOL to its executable. This complete fixture passed with promtool 3.14.0 and Python 3.12.14. It evaluates actual rules locally, without Docker, a listener, Alertmanager, or an external notification destination.

from pathlib import Path
from tempfile import TemporaryDirectory
import os
import shutil
import subprocess

promtool = shutil.which(os.environ.get("PROMTOOL", "promtool"))
if promtool is None:
 raise SystemExit("Set PROMTOOL to an installed promtool executable")
promtool = str(Path(promtool).resolve())
tests = r"""evaluation_interval: 1m
rule_files:
 - rules.yml
tests:
 - interval: 1m
 input_series:
 - series: "queue_depth{queue=\"orders\"}"
 values: "0 6 6 6 0 6 0 0"
 promql_expr_test:
 - expr: 'sum(ALERTS{alertname="BooleanTrap",alertstate="firing"}) or vector(0)'
 eval_time: 0m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"pending\"}) or vector(0)"
 eval_time: 1m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"firing\"}) or vector(0)"
 eval_time: 2m
 exp_samples:
 - labels: "{}"
 value: 0
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"firing\"}) or vector(0)"
 eval_time: 3m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\"}) or vector(0)"
 eval_time: 4m
 exp_samples:
 - labels: "{}"
 value: 0
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"pending\"}) or vector(0)"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\"}) or vector(0)"
 eval_time: 6m
 exp_samples:
 - labels: "{}"
 value: 0
 - interval: 1m
 input_series:
 - series: 'queue_depth{queue="orders"}'
 values: "6 6 stale"
 promql_expr_test:
 - expr: 'sum(ALERTS{alertname="QueueBacklog"}) or vector(0)'
 eval_time: 2m
 exp_samples:
 - labels: "{}"
 value: 0
"""
rules = r"""groups:
 - name: queue-alerts
 interval: 1m
 rules:
 - alert: QueueBacklog
 expr: queue_depth{queue="orders"} > 5
 for: 2m
 labels:
 severity: page
 annotations:
 summary: Orders queue remains above five
 - alert: BooleanTrap
 expr: queue_depth{queue="orders"} > bool 5
 annotations:
 summary: Deliberately incorrect boolean alert for this test
"""
with TemporaryDirectory(prefix="prom-rule-test-") as directory:
 work = Path(directory)
 (work / "tests.yml").write_text(tests, encoding="utf-8")
 (work / "rules.yml").write_text(rules, encoding="utf-8")
 subprocess.run([promtool, "test", "rules", "tests.yml"], cwd=work,
 check=True, timeout=45)

The expected output is SUCCESS. At minute one the alert is pending, at minute two it is not firing, and at minute three it fires. At minute four the condition recovers and no active ALERTS element remains. A second spike starts pending at minute five but clears at minute six, so it never reaches firing.

The assertions inspect the synthetic ALERTS series; their or vector(0) only makes absence countable inside the test. It is not part of the production alert expression. A deliberately wrong BooleanTrap rule fires even at minute zero, when the comparison is false. A separate stale-marker fixture clears pending state before firing; this tests explicit staleness, not every scrape-failure timeline. Temporary rule files are removed after execution. These checks verify the supplied timelines. Receiver matching, grouping delays, and actual message delivery require separate tests with the intended Alertmanager configuration.

Sources

Prometheus: alerting rules, active label sets, for and synthetic ALERTS series, checked 2026-10-15.
Prometheus: unit testing rules, evaluation timelines and assertion fixtures, checked 2026-10-15.
Prometheus: Alertmanager, notification grouping, routing, silences and inhibition, checked 2026-10-15.

Cardinality is a metric budget

Wed, 21 Jul 2027 00:00:00 +0000

A metric costs more than its short name suggests. Prometheus identifies a time series by its metric name and complete label set. A new observed label combination creates a different series, which must be ingested, indexed, retained, and queried. Cardinality budgeting asks which distinctions are worth paying for and whether their number stays bounded as the service grows.

The article’s exercise is a series-count model: all eight bounded combinations occur, and each replica has those combinations. Nine core classic-histogram components and twenty replicas multiply both alternatives equally. These counts are not measured memory or disk use.

Count combinations that actually exist

Label dimensions multiply the possible combinations, but the product of their distinct values is an upper bound unless every combination occurs. Two routes, two methods, and two status categories allow eight combinations. A service that only accepts one method on each route may emit fewer. Count observed combinations as well as theoretical limits, and include target labels such as instance when estimating what the server will ingest.

The value set matters more than the label’s friendly name. A normalized route such as /orders/{order_id} groups operations; a raw path containing each order identifier creates a growing dimension. Request IDs, timestamps, user identifiers, and arbitrary query strings can grow with event volume. A label named route is not bounded merely because the developer intended it to describe a route. Check the values produced on real traffic, including errors and uncommon paths.

Include the metric representation

A counter usually contributes one series for each observed label combination, before optional companion metrics. A classic histogram expands the same combination into its bucket, sum, and count series. With six finite bucket boundaries, one positive-infinity bucket, a sum, and a count, that is nine core series per combination. Some clients expose an additional creation-time series; include it in a real inventory rather than assuming this simplified count is complete.

Replicas add their own instance dimension. Scrape interval changes samples per series, while retention changes how long historical samples remain; neither is the same as the number of distinct series. Rapidly changing identifiers also create churn: yesterday’s short-lived combinations may remain in storage even after their producers disappear. Monitor active series, creation rate, ingestion volume, memory, and query behavior instead of using one fixed bytes-per-series estimate for every deployment.

When it breaks

An innocent-looking diagnostic label can turn an aggregate metric into a per-request event store. Removing that label from a dashboard query does not undo the cost already paid to ingest and retain it. Hashing each identifier still preserves its distinctness, and truncating a random identifier generally only postpones the problem while introducing collisions. Choose a deliberately bounded category when aggregation is the actual requirement.

Do not collapse away distinctions needed to detect customer impact. Keep useful bounded outcome or service dimensions, and move request-level investigation to appropriately controlled logs or traces. If unwanted labels are removed during collection, check for duplicate resulting series within a scrape and for lost diagnostic meaning. Prefer fixing the producer where possible, then verify the new label set and plan how historical series age out.

Check it quickly

Save this as cardinality.py and run Python 3. The synthetic events exercise every route, method, and status combination fifty times. One version retains only those bounded dimensions; another also includes a unique request ID. The calculation assumes twenty replicas each expose the same combinations as classic histograms with six finite boundaries.

from itertools import product

routes = ["/orders/{order_id}", "/products/{product_id}"]
methods = ["GET", "POST"]
statuses = ["200", "500"]
events = list(product(routes, methods, statuses, range(50)))
bounded = {(route, method, status) for route, method, status, _ in events}
with_request_id = {(route, method, status, f"request-{i}")
 for i, (route, method, status, _) in enumerate(events)}

finite_buckets = 6
# Core classic-histogram series: finite buckets, +Inf, _sum, _count.
# Optional _created series are deliberately excluded from this calculation.
series_per_combination = finite_buckets + 3
replicas = 20
bounded_series = len(bounded) * series_per_combination * replicas
request_id_series = len(with_request_id) * series_per_combination * replicas

assert len(events) == 400 and len(bounded) == 8
assert len(with_request_id) == 400
assert bounded_series == 1440 and request_id_series == 72000
print(f"observed label sets: bounded={len(bounded)}, request_id={len(with_request_id)}")
print(f"core classic-histogram series: bounded={bounded_series}, request_id={request_id_series}")
print(f"multiplier={request_id_series // bounded_series}x")

Four hundred events collapse into eight bounded combinations. Adding request IDs keeps four hundred distinct combinations, giving 1,440 versus 72,000 core histogram series across the twenty replicas: a fiftyfold increase. Optional creation-time series and other metric families are excluded explicitly, so these are modeled series counts rather than a measurement of Prometheus memory or storage use.

The comparison demonstrates why aggregating events at instrumentation time changes the cost profile. Before approving another label, state the operational question it answers, its maximum expected value set, the combinations actually emitted, and the effect on each metric family. Then validate that estimate against the collector and server as the workload changes.

Sources

Prometheus: Data Model, series identity and changes in label sets, checked 2026-09-13.
Prometheus: Metric and Label Naming, bounded labels and high-cardinality dimensions, checked 2026-09-13.
Prometheus: Instrumentation, cardinality costs and instrumentation choices, checked 2026-09-13.

Dashboards summarize; they do not magnify

Sun, 18 Jul 2027 00:00:00 +0000

A useful dashboard helps a reader decide what to inspect next. It reduces a large collection of measurements into a small set of questions: are users succeeding, where is the impact, and is capacity becoming constrained? Reduction must preserve important differences. A green global percentage can conceal a failing region, and a blank panel can mean missing telemetry rather than a quiet, healthy service.

Give the page a reading order

Start with the service outcome and observation coverage, then show traffic, latency, errors, and relevant saturation. Provide drilldowns for regions, operations, or dependencies when those distinctions change the next action. Put deployment annotations and links to diagnostic material near the relevant views. Correlated lines can suggest where to look, but their timing alone does not prove that a deployment or dependency caused the incident.

Each panel should answer a clear question, not necessarily contain exactly one metric. Comparing queue depth with its configured limit can be useful when both share a unit. Mixing unrelated units on a complicated dual-axis chart makes apparent relationships easy to invent. State the unit, aggregation window, population, and threshold meaning. A reader should not need to remember whether an axis represents milliseconds, seconds, fractions, or percentages.

Preserve weighting and coverage

Compute a service error fraction from the total bad events divided by total eligible events over matching windows. An unweighted mean of regional percentages gives a quiet region the same influence as a busy one. That is a different statistic, not a harmless display choice. Even a correctly weighted result should have a useful breakdown when small populations matter to the service commitment.

Freshness is another dimension of scope. Display the last successful observation or an explicit coverage signal, and define when stale data becomes unknown. A service-specific freshness threshold is separate from the storage system’s staleness rules. Filling gaps with zero can turn an unavailable exporter into an apparently perfect error rate or an empty queue. Missing data should remain visible, with a next step to investigate collection or the service itself.

When it breaks

Automatically chosen axis ranges can make a tiny variation look like a major incident, while an overly broad fixed range can conceal a meaningful threshold crossing. Select scales according to the operational question and keep comparable panels consistent. An average latency also hides a slow minority; use appropriate distributions and counts without averaging precomputed percentiles across instances.

An attractive dashboard can still become a collection of unrelated snapshots. Different time zones, filters, refresh intervals, and aggregation windows prevent a coherent comparison. Shared variables should have explicit defaults and visible scope, especially for production environments. Test the page with a regional failure, sparse traffic, and absent measurements. If those situations all appear green, the page is compressing away information needed for decisions.

The weighted 1% overview, eastern 50% error fraction and unknown southern coverage must remain visible together. These are synthetic aggregation values, not a validated dashboard screen.

Check it quickly

Save this as dashboard.py and run Python 3. It computes panel-ready values from synthetic counts for one completed sixty-second interval. Two regions have fresh observations; a third has missing values and an older observation time. The ninety-second freshness threshold is an example dashboard policy, not a claim about Prometheus defaults. No Grafana installation is required for this aggregation exercise.

import json
from fractions import Fraction

# Counts cover the same completed 60-second interval.
rows = [
 {"region": "west", "requests": 9800, "errors": 0, "age_seconds": 10},
 {"region": "east", "requests": 200, "errors": 100, "age_seconds": 10},
 {"region": "south", "requests": None, "errors": None, "age_seconds": 240},
]
fresh = [r for r in rows if r["age_seconds"] <= 90
 and r["requests"] is not None and r["errors"] is not None]
missing = [r["region"] for r in rows if r not in fresh]
requests = sum(r["requests"] for r in fresh)
errors = sum(r["errors"] for r in fresh)
regional = {r["region"]: Fraction(r["errors"], r["requests"]) for r in fresh}
known_ratio = Fraction(errors, requests)
unweighted = sum(regional.values()) / len(regional)

assert known_ratio == Fraction(1, 100)
assert regional["east"] == Fraction(1, 2) and unweighted == Fraction(1, 4)
assert missing == ["south"]
print(json.dumps({
 "coverage": "incomplete",
 "fresh_requests_per_second": round(requests / 60, 3),
 "fresh_error_percent": float(known_ratio * 100),
 "regional_error_percent": {k: float(v * 100) for k, v in regional.items()},
 "wrong_unweighted_percent": float(unweighted * 100),
 "unknown_regions": missing,
}, indent=2))

Fresh observations contain 10,000 requests and 100 errors, giving 166.667 requests per second and a weighted 1% error fraction. The small eastern region is nevertheless failing half its requests. Averaging the two regional percentages would incorrectly label the combined request fraction as 25%. Show the 1% overview with its regional breakdown instead of choosing one statistic to hide the other.

The output also declares coverage incomplete and lists the southern region as unknown. Neither its requests nor its errors are invented as zero. This fixture checks the arithmetic and missing-data meaning behind the panels; it does not validate an actual dashboard query or screen. Carry the same definitions into the rendered dashboard and review it against known incident and telemetry-loss cases.

Sources

Grafana: Dashboard Best Practices, goals, focused questions, and a coherent reading order, checked 2026-09-13.
Google SRE: Monitoring Distributed Systems, service-oriented dashboards and signals, checked 2026-09-13.
Prometheus: Metric and Label Naming, units and meaningful aggregation, checked 2026-09-13.

External labels identify outgoing Prometheus data

Wed, 09 Jun 2027 00:00:00 +0000

When several Prometheus servers send data to a common system, downstream readers need to know which collector or environment supplied it. Global external_labels can add that context during communication with external systems. They do not rewrite every locally stored series, and they do not override labels already present on the outgoing series. Understanding both boundaries prevents misleading configuration checks.

External labels supplement missing names at the relevant outgoing boundary; they do not rewrite locally stored series or replace existing values. The exact label values illustrate the source-side federation fixture. Its historical integration gap is unchanged, and this artwork is not a new federation, remote-write, alert-delivery or deduplication test.

Local storage and outgoing identity differ

A local query of up generally shows the labels created by scraping and relabeling. An external label such as prom_region need not appear there. Its absence from that query does not prove the configuration failed. Compare the appropriate outgoing interface with the local result; federation is a direct way to observe this difference without configuring a remote storage backend.

The configuration contract also covers communication such as remote storage and alerting, but each downstream path has its own relabeling and processing. Inspect the actual consumer when validating that path. A locally evaluated recording rule does not acquire every external label merely because it stores a new series. External labels are added at the relevant communication boundary, not as a universal local query transformation.

Existing labels take precedence

If a target series already has cluster="target-cluster", configuring an external cluster="collector-cluster" does not replace that value. External labels supplement missing label names. This is defined behavior, not a promise that newer versions always reject collisions or log an error. A local metric can therefore export a different cluster value from the one an operator expected the global setting to enforce.

Reserve distinct collector label names when that distinction matters, and check exporter and relabeling conventions for collisions. For federation, the destination’s honor_labels setting determines how it handles source labels that conflict with its own scrape labels. That destination behavior is a separate issue from how the source adds external labels to its federation response.

Stable labels do not provide deduplication

Changing a global label value changes the identity of subsequently exported series. Previously stored downstream history remains under its old labels; it is not rewritten. Update selectors and dashboards with that historical boundary in mind. Adding many labels is not automatically useful: a constant label adds context, while different values across collectors create additional distinctions consumers must handle.

Two collectors scraping the same targets can export duplicate observations under different replica labels. A naive sum across replicas can double apparent traffic. External labels make that distinction expressible but do not configure the downstream system’s deduplication. Nor are labels a tenant authorization boundary. Verify the storage system’s actual isolation and query behavior when those requirements apply.

Alert relabeling happens after external labels are added. High-availability collectors can therefore remove only the replica discriminator before sending equivalent alerts, while retaining it on outgoing metrics for a storage system that needs it. Verify that the remaining alert labels really are equivalent; dropping a genuine ownership distinction can merge unrelated incidents.

When it breaks

A team may look for an external label in a local query, find none, and repeatedly change a working configuration. The opposite mistake is seeing the desired label in a federation response and assuming remote write or alert routing has also been tested. Check the specific path and final labels, including any downstream relabeling, instead of using one observation as evidence for all integrations.

Check it quickly

Save this as external-labels.py and run python3 external-labels.py. It requires Docker and the Prometheus 3.5.1 image; Docker may download an uncached image. One disposable Prometheus scrapes itself, then the program compares its local query response with its text federation output. All HTTP requests stay on container loopback, with no host ports or external network.

import json
import re
import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
# All HTTP traffic stays on loopback inside one disposable container.
script = r"""set -eu
cat > /tmp/prom.yml <<'CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
 external_labels:
 cluster: collector-cluster
 prom_region: lab-region
 replica: one
scrape_configs:
 - job_name: self
 static_configs:
 - targets: ['127.0.0.1:9090']
 labels:
 cluster: target-cluster
CONFIG
/bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data \
 --web.listen-address=127.0.0.1:9090 > /tmp/prom.log 2>&1 &
for attempt in $(seq 1 30); do
 if wget -qO /tmp/ready.json 'http://127.0.0.1:9090/api/v1/query?query=up%20%3D%3D%201'; then
 if grep -q '"status":"success"' /tmp/ready.json && ! grep -q '"result":\[\]' /tmp/ready.json; then
 wget -qO /tmp/local.json 'http://127.0.0.1:9090/api/v1/query?query=up'
 wget -qO /tmp/federated.txt --header='Accept: text/plain; version=0.0.4' \
 'http://127.0.0.1:9090/federate?match%5B%5D=up'
 cat /tmp/local.json
 printf '\n'
 cat /tmp/federated.txt
 exit 0
 fi
 fi
 sleep 1
done
cat /tmp/prom.log >&2
exit 1
"""
name = "xbcatm-prom-live-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if result.returncode:
 raise RuntimeError(result.stderr or result.stdout)
local_json, _, exported = result.stdout.partition("\n")
payload = json.loads(local_json)
assert payload["status"] == "success"
local = next(a["metric"] for a in payload["data"]["result"]
 if a["metric"]["job"] == "self")
line = next(line for line in exported.splitlines() if line.startswith("up{"))
# The fixture's controlled label values contain no quotes or escape sequences.
outbound = dict(re.findall(r'(\w+)="([^"]*)"', line))
assert outbound["prom_region"] == "lab-region" and outbound["replica"] == "one"
assert outbound["cluster"] == local["cluster"] == "target-cluster"
assert "prom_region" not in local and "replica" not in local
print(json.dumps({"federated_labels": outbound, "local_labels": local}, indent=2))

The local labels contain cluster="target-cluster" but no prom_region or replica. Federation adds prom_region="lab-region" and replica="one", while the existing cluster value remains unchanged. Assertions verify all three facts. The parser is deliberately limited to the fixture’s controlled label values, which contain no quotes or escape sequences; it is not a general exposition parser.

The uniquely named container is removed on exit. This establishes source-side federation behavior and the local/outgoing distinction. It does not validate a federation destination, remote write receiver, high-availability deduplication, or notification delivery. Use the corresponding downstream interface and configuration to complete those separate integration checks.

The operational fixture is unchanged here and was not rerun in this review. Its expected outputs and earlier observations remain distinct from current execution evidence.

Sources

Prometheus: configuration, external labels supplement labels during outbound communication, checked 2026-10-15.
Prometheus: federation, federation selectors and preservation of upstream labels, checked 2026-10-15.
Prometheus: HTTP API, local query results used for comparison, checked 2026-10-15.

Prometheus staleness and alert waiting are different mechanisms

Wed, 03 Feb 2027 00:00:00 +0000

A missing metric is not a numeric zero. Prometheus distinguishes a series with a recent value, a series marked stale, and a selector with no usable sample. Alerting then applies its own evaluation and waiting rules. Treating all these mechanisms as one universal grace period produces alerts that miss disappearing targets or report an outage without establishing what actually disappeared.

These are the article’s explicit synthetic samples, not a universal failed-scrape or discovery-removal timetable. At 3m, the range (0m,3m] excludes the sample at 0m; for: 2m is a separate alert layer. Historical local promtool evidence is unchanged. Live collection, expected membership and notification delivery remain separate checks.

A marker is not a five-minute timer

An instant selector normally uses the newest eligible sample within its lookback period, which defaults to five minutes. A stale marker tells the engine to stop returning that series in instant selections after the marker. It does not require the previous value to remain visible for another five minutes. Historical samples are still stored and can remain relevant to range queries.

Collection circumstances matter. A series no longer exported, a target removed from discovery, and an exporter assigning its own timestamps need not follow identical timing. Explicit timestamps and staleness configuration can alter behavior. Determine which path your exporter uses before predicting disappearance time from the scrape interval alone. A configurable lookback is a query selection rule, not a universal service outage detector.

Choose the absence question deliberately

absent(selector) returns a value of one when the selector has no elements. It returns an empty vector when matching data exists, even if that data’s numeric value is zero. absent_over_time(selector[3m]) asks whether the selected range contains any samples. A real sample from before a stale marker can keep that range-based absence expression empty until the sample leaves the window.

The alert rule’s for duration is another layer: the expression must keep returning the relevant alert label set at successive evaluations before it fires. It does not configure staleness or extend the query’s lookback. Combining a three-minute absence window with a waiting duration delays notification further, with evaluation cadence and collection timing also contributing. Choose that delay from the detection objective.

When it breaks

absent(up{job="api"}) is not a failed-scrape alert. If the configured targets are present with up=0, the selector still has elements, so absence is false. A comparison against zero addresses that case. If a target disappears from discovery altogether, checking only zero can miss it once its series stops appearing. Monitor expected membership separately when that distinction matters.

A broad selector also cannot enumerate missing members. If one replica survives, absent over the whole job remains empty even when several expected replicas are gone. Match a deliberately known identity or compare observed series with an authoritative inventory. A label rename can look like an outage under the old matcher while the process continues working under a new identity.

Check it quickly

Save this as stale.py and run python3 stale.py with an installed promtool on PATH, or set PROMTOOL to its executable path. The recorded earlier execution used Python 3.12.14 and promtool 3.14.0; it was not rerun during this source review. The real query engine evaluates synthetic samples in a temporary directory; no server, listener, container or external service is started.

from pathlib import Path
from tempfile import TemporaryDirectory
import os
import shutil
import subprocess

# Uses a real, installed promtool; no server or Docker runtime is started.
promtool = shutil.which(os.environ.get("PROMTOOL", "promtool"))
if not promtool:
 raise SystemExit("Set PROMTOOL to an installed promtool executable")

tests = r"""evaluation_interval: 1m
rule_files: [rules.yml]
tests:
 - interval: 1m
 input_series:
 - series: "up{job=\"api\",instance=\"gone\"}"
 values: "1 0 0 0 0"
 - series: "service_events_total{job=\"api\",instance=\"gone\"}"
 values: "10 stale _ _ _"
 - series: "service_events_total{job=\"api\",instance=\"zero\"}"
 values: "0 0 0 0 0"
 promql_expr_test:
 - expr: "service_events_total{job=\"api\",instance=\"gone\"}"
 eval_time: 1m
 exp_samples: []
 - expr: "up{job=\"api\",instance=\"gone\"}"
 eval_time: 1m
 exp_samples:
 - labels: "{__name__=\"up\",job=\"api\",instance=\"gone\"}"
 value: 0
 - expr: "absent(service_events_total{job=\"api\",instance=\"gone\"})"
 eval_time: 1m
 exp_samples:
 - labels: "{job=\"api\",instance=\"gone\"}"
 value: 1
 - expr: "absent_over_time(service_events_total{job=\"api\",instance=\"gone\"}[3m])"
 eval_time: 2m
 exp_samples: []
 - expr: "absent_over_time(service_events_total{job=\"api\",instance=\"gone\"}[3m])"
 eval_time: 4m
 exp_samples:
 - labels: "{job=\"api\",instance=\"gone\"}"
 value: 1
 - expr: "absent(service_events_total{job=\"api\",instance=\"zero\"})"
 eval_time: 4m
 exp_samples: []
 - expr: "absent(up{job=\"api\"})"
 eval_time: 4m
 exp_samples: []
 - expr: 'absent_over_time(service_events_total{job="api",instance="gone"}[3m])'
 eval_time: 3m
 exp_samples:
 - labels: '{job="api",instance="gone"}'
 value: 1
 - expr: 'ALERTS{alertname="BusinessMetricMissing",alertstate="pending"}'
 eval_time: 3m
 exp_samples:
 - labels: '{__name__="ALERTS",alertname="BusinessMetricMissing",alertstate="pending",job="api",instance="gone"}'
 value: 1
 alert_rule_test:
 - eval_time: 4m
 alertname: BusinessMetricMissing
 exp_alerts: []
 - eval_time: 5m
 alertname: BusinessMetricMissing
 exp_alerts:
 - exp_labels: {job: api, instance: gone}
 exp_annotations: {}
"""
rules = r"""groups:
 - name: absence
 rules:
 - alert: BusinessMetricMissing
 expr: absent_over_time(service_events_total{job="api",instance="gone"}[3m])
 for: 2m
"""

with TemporaryDirectory(prefix="prometheus-query-check-") as directory:
 root = Path(directory)
 (root / "tests.yml").write_text(tests, encoding="utf-8")
 if rules:
 (root / "rules.yml").write_text(rules, encoding="utf-8")
 result = subprocess.run([promtool, "test", "rules", "tests.yml"],
 cwd=root, text=True, capture_output=True, timeout=45)
 print(result.stdout, end="")
 if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The test prints SUCCESS. At minute one, the marked business series is absent from an instant selection, while its separately supplied up series exists with value zero. Instant absent returns one. At minute two, the three-minute range still contains the original business sample, so absent_over_time returns nothing; at minute three the left-open boundary already excludes the original sample, and at minute four it still returns one.

The continuously exported zero-valued business series is never absent, and the job-wide up selector is not absent either. These assertions distinguish missing values from failing values instead of converting both to zero. The temporary test files are removed on exit. The added alert starts pending at minute three, is not firing at minute four, and fires at minute five after its separate two-minute for duration. Assertions check both pending state and firing timing. Live collection, target removal and notification delivery remain separate integration tests.

Sources

Prometheus: querying basics, instant selectors, lookback and staleness, checked 2026-10-15.
Prometheus: query functions, absent and absent_over_time behavior, checked 2026-10-15.
Prometheus: unit testing rules, missing and stale sample notation, checked 2026-10-15.
Prometheus: alerting rules, pending duration and synthetic ALERTS series, checked 2026-10-15.

Prometheus histograms retain a bucketed distribution

Wed, 06 Jan 2027 00:00:00 +0000

A latency mean can conceal a slow subset of requests. A histogram retains more distribution information by counting observations into buckets, but it still discards detail within each bucket. Prometheus can estimate quantiles from those counts without storing every observation. The estimate depends on bucket boundaries and aggregation; it is not a recovery of the complete original latency distribution.

For classic histograms, rate each counter before merging compatible buckets, preserve le, and then estimate the quantile. The article’s synthetic fixture yields fleet p99 4.94 seconds, threshold fraction 50% at two seconds, and mean 2.085 seconds. A +Inf-tail result is not a measured maximum or a threshold guarantee. No promtool or server was run for this drawing.

A classic histogram exports cumulative _bucket counters, a _count, and a _sum. The le label identifies each inclusive bucket upper bound. An observation of 1.5 seconds increments the two-second bucket and every larger bucket, so bucket values must not be summed as though they were disjoint counts. The positive-infinity bucket includes all observations and corresponds to the count.

For nonnegative latency observations, applying rate to the sum and count gives total seconds accumulated per second and observations per second. Their ratio is mean latency in seconds. Bucket rates describe the distribution over the selected time range. Apply the counter rate before aggregating instances so each series’ observed reset is handled independently.

Aggregate buckets before estimating a quantile

For one fleet percentile, sum the bucket rates while retaining le, then call histogram_quantile. Keep any grouping dimensions required by the question, such as service. Compatible bucket boundaries and matching event definitions are necessary for meaningful aggregation. Removing le destroys the bucket identity the classic quantile function needs.

Do not average instance percentiles. A percentile is a position in a distribution, not an additive count, and even equal request volumes do not make percentile averaging valid. The combined bucket curve can have a very different shape from either instance. If the question is the fraction meeting a two-second objective, a bucket exactly at two seconds divided by the total count answers that threshold question directly.

A summary may export client-computed quantiles. Those values cannot be merged into a fleet quantile by summing or averaging them. Its count and sum can still contribute to a combined mean when event definitions match. Select the metric type from the aggregation you need, not solely from a panel’s preferred display.

When it breaks

A wide bucket near the desired percentile can make the estimate look more precise than the measurements justify. Classic quantile calculation interpolates within the selected bucket; it does not know where the individual observations occurred inside it. Keep boundaries useful around service objectives, and review them before a rollout that mixes different schemas across replicas. Refining future buckets cannot recover detail already discarded from historical observations.

A missing bucket or exporter is also different from zero observations. An apparently better percentile can result from losing the slow instance’s metrics. Check scrape completeness, contributing series, and sample counts before interpreting improvement. A present histogram with zero observations yields NaN from histogram_quantile; a missing input can produce no result at all. Neither means zero latency, and sparse observations still make a percentile fragile.

Check it quickly

Save this as histogram-query.py and run python3 histogram-query.py with an installed promtool on PATH, or set PROMTOOL to its executable path. The recorded earlier execution used Python 3.12.14 and promtool 3.14.0; it was not rerun during this source review. The real query engine evaluates synthetic samples in a temporary directory; no server, listener, container or external service is started.

from pathlib import Path
from tempfile import TemporaryDirectory
import os
import shutil
import subprocess

# Uses a real, installed promtool; no server or Docker runtime is started.
promtool = shutil.which(os.environ.get("PROMTOOL", "promtool"))
if not promtool:
 raise SystemExit("Set PROMTOOL to an installed promtool executable")

tests = r"""evaluation_interval: 1m
tests:
 - interval: 1m
 input_series:
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"1\"}"
 values: "0+95x5"
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"2\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"5\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"+Inf\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_count{instance=\"fast\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_sum{instance=\"fast\"}"
 values: "0+17x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"1\"}"
 values: "0+0x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"2\"}"
 values: "0+0x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"5\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"+Inf\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_count{instance=\"slow\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_sum{instance=\"slow\"}"
 values: "0+400x5"
 - series: 'tail_duration_seconds_bucket{le="1"}'
 values: "0+0x5"
 - series: 'tail_duration_seconds_bucket{le="5"}'
 values: "0+90x5"
 - series: 'tail_duration_seconds_bucket{le="+Inf"}'
 values: "0+100x5"
 promql_expr_test:
 - expr: "abs(histogram_quantile(0.99, sum by (le) (rate(api_duration_seconds_bucket[2m]))) - 4.94) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "abs(avg(histogram_quantile(0.99, rate(api_duration_seconds_bucket[2m]))) - 3.385) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(rate(api_duration_seconds_bucket{le=\"2\"}[2m])) / sum(rate(api_duration_seconds_count[2m]))"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 0.5
 - expr: "abs(sum(rate(api_duration_seconds_sum[2m])) / sum(rate(api_duration_seconds_count[2m])) - 2.085) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: 'histogram_quantile(0.99, rate(tail_duration_seconds_bucket[2m]))'
 eval_time: 5m
 exp_samples:
 - labels: '{}'
 value: 5
"""
rules = r""""""

with TemporaryDirectory(prefix="prometheus-query-check-") as directory:
 root = Path(directory)
 (root / "tests.yml").write_text(tests, encoding="utf-8")
 if rules:
 (root / "rules.yml").write_text(rules, encoding="utf-8")
 result = subprocess.run([promtool, "test", "rules", "tests.yml"],
 cwd=root, text=True, capture_output=True, timeout=45)
 print(result.stdout, end="")
 if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The expected output is SUCCESS. The correctly combined p99 estimate is 4.94 seconds. Averaging the two instance p99 estimates produces 3.385 seconds, which is not the fleet p99. Half the observations meet the two-second threshold, and the combined mean is 2.085 seconds. Numerical assertions allow a tiny floating-point tolerance rather than requiring an identical binary representation of decimal fractions.

The fixture validates PromQL arithmetic and aggregation, not actual request instrumentation or the distribution within a bucket. It removes its temporary test files after execution. Current Prometheus guidance also discusses native histograms, which encode buckets differently and can simplify queries when the instrumentation and storage path support them. This example deliberately targets classic histograms: retain le, aggregate their counter rates correctly, and describe the returned percentile as an estimate.

A second histogram places ten percent of observations beyond its largest finite boundary of five seconds. Its p99 still returns five: for a classic quantile falling in the positive-infinity bucket, the function returns the preceding finite upper bound. That number is not a measured maximum or proof that ninety-nine percent finished within five seconds. The added assertion makes this reporting trap explicit.

Sources

Prometheus: histograms and summaries, bucketed observations, aggregation and metric choices, checked 2026-10-15.
Prometheus: query functions, classic histogram_quantile interpolation and rate ordering, checked 2026-10-15.
Prometheus: unit testing rules, executing expected PromQL values, checked 2026-10-15.

The golden signals of health

Sun, 29 Nov 2026 00:00:00 +0000

Latency, traffic, errors, and saturation are four complementary ways to ask whether a service is serving users well. They are a starting set of questions, not four universal counters that prove health by themselves. Their usefulness depends on where they are measured, which work they describe, and how their time windows align. A fast response can still be a failed operation.

Separate experience from demand

Latency measures how long work takes at a chosen boundary. Track successful and failed operations separately so that immediate failures do not make the overall distribution appear healthier. Include useful tail information, but also preserve the observation count: a p99 calculated from a tiny population does not establish a stable view of rare behavior. Client latency and server processing time describe different portions of the experience.

Traffic describes demand or completed work in an appropriate unit, such as requests per second or bytes per second. Define whether the measurement counts arrivals, completions, or retries. A completion rate can fall while demand continues rising if a queue is stuck. Concurrent requests are a count, not a request rate; use them as additional context rather than swapping their units into an otherwise identical traffic panel.

Explain failure and resource pressure

Errors measure work that fails the promised outcome. HTTP status failures are one signal, but a nominally successful response containing incorrect data can also violate the contract. Decide which outcomes and latency violations belong in the indicator, and use a denominator with the same scope. Display counts as well as ratios so that one error in a quiet period is not mistaken for thousands of affected requests.

Saturation measures how close a constrained resource is to its usable limit. CPU, memory, connection pools, worker occupancy, and queue age may reveal different bottlenecks. A thread pool with every worker busy is not automatically unhealthy, but a growing queue and aging work show why that occupancy matters. A snapshot is not an interval average, and a low average can conceal brief overload. Capacity thresholds need workload context and measurement, not a decorative red line.

When it breaks

Mixing populations can create an impossible-looking story: latency from successful application requests, errors from an edge proxy, and traffic from another region may change independently. Name boundaries and filters explicitly before correlating them. Fast rejections can improve an all-request latency average while errors rise. Measuring only completions can also omit stalled requests, making both latency and throughput miss the work users are still waiting for.

None of the four signals proves the root cause of a deviation. CPU can be quiet while a database connection pool is exhausted, and queue growth can follow either more demand or slower downstream work. Use the signals to choose a next investigation, then inspect traces, logs, dependencies, and change evidence. Alert on actionable user impact or impending exhaustion; a momentary deviation in any one gauge need not wake a human.

Fast failures lower the combined mean while remaining failures. Compare aligned populations and units; the worker and queue gauges are a separate end-of-window snapshot.

Check it quickly

Save the following as golden.py and run Python 3. It uses one synthetic ten-second interval of completed requests and a separately labeled end-of-interval worker snapshot. The fixture verifies aggregation and interpretation of the four signals. It neither generates load nor measures a live server, and its exact nearest-rank percentile is not an estimate from histogram buckets.

from math import ceil
from statistics import mean

# Synthetic completed requests in one 10-second observation interval.
records = [(True, 0.05)] * 90 + [(True, 0.8)] * 5 + [(False, 0.005)] * 5
success = sorted(duration for ok, duration in records if ok)
failure = [duration for ok, duration in records if not ok]
p99_success = success[ceil(0.99 * len(success)) - 1]
traffic_rps = len(records) / 10
error_ratio = len(failure) / len(records)
# These are end-of-interval gauges, not interval averages.
busy_workers, total_workers = 4, 4
queued_requests, oldest_queue_seconds = 12, 1.2

assert p99_success == 0.8 and traffic_rps == 10 and error_ratio == 0.05
assert mean(failure) < mean(success)
assert busy_workers / total_workers == 1
print(f"traffic={traffic_rps:.1f} completed requests/s; errors={error_ratio:.1%}")
print(f"success p99={p99_success:.3f}s; failed mean={mean(failure):.3f}s")
print(f"success mean={mean(success):.6f}s; all-request mean={mean(d for _, d in records):.6f}s")
print(f"worker snapshot={busy_workers}/{total_workers}; queue={queued_requests}; oldest={oldest_queue_seconds}s")

The output shows ten completed requests per second, a 5% error ratio, successful-request p99 of 0.800 seconds, and failed-request mean of 0.005 seconds. The successful mean is about 0.089474 seconds, while including the fast errors lowers the all-request mean to 0.085250 seconds. A lower combined mean therefore does not imply more useful service.

All four workers are busy in the snapshot, with twelve queued requests and an oldest wait of 1.2 seconds. Those gauges provide pressure context, not proof that occupancy stayed at 100% throughout the interval. Read them with the latency, demand definition, and subsequent queue trend before deciding whether to scale, shed load, or investigate a slower dependency.

Sources

Google SRE: Monitoring Distributed Systems, the four golden signals and successful versus failed request latency, checked 2026-09-13.
Prometheus: Instrumentation, request-serving systems and thread-pool instrumentation, checked 2026-09-13.
Google SRE: Handling Overload, resource limits, queuing, and changing request cost, checked 2026-09-13.

Metrics and logs need an explicit population

Fri, 16 Oct 2026 00:00:00 +0000

Metrics and logs are different representations of observed behavior, not competing definitions of truth. A metric usually summarizes measurements under a set of dimensions. A log record describes an event and can carry detailed context. Both can answer aggregate questions if their coverage and semantics support the calculation. Neither becomes authoritative merely because its dashboard looks cleaner or its records contain more fields.

For a checkout service, a completed-request counter can classify every terminal outcome as success or error. A few bounded dimensions, such as route and outcome, make it practical to compare traffic and failure rates over time. A diagnostic log can then explain a particular failed request using its correlation identifier, error category, and dependency response. These serve related questions, but their denominators must still be established separately.

The article’s controlled batch fixes membership and one terminal outcome per request. Its five diagnostic records are deliberately selected, not established as representative. Historical arithmetic and format validation remain unchanged; this illustration adds no runtime or production coverage evidence.

Decide what the measurement includes

A cumulative counter is not itself a requests-per-second measurement. A live rate calculation needs observations across time and handling for resets. A complete synthetic batch, however, can be counted directly because its beginning, end, and membership are known. Do not silently replace a windowed production rate with a ratio of unrelated snapshots, or compare a sampled event stream against a counter covering every request.

Logs are not limited to manual reading. Consistently structured, complete terminal events can be aggregated into the same request totals as a counter. The tradeoff includes event volume, indexing, query latency, retention, and operational complexity. Metrics are often economical for repeated aggregate questions, but a metric with a new label value for every request can be expensive. The representation alone does not settle cost.

When it breaks

Consider a hundred completed requests with four failures. An incident-focused log selection retains five requests, including two failures. The complete population has a four-percent failure fraction, while the selected logs have a forty-percent fraction. Both calculations are arithmetically correct. Presenting the second as the service-wide failure rate is wrong because the five records were not a complete or representative population.

Counting retry failures as terminal request failures creates another mismatch: a request may fail an attempt and later succeed. Missing logs, duplicated deliveries, differing time windows, and incompatible route definitions can also make apparently related charts disagree. Define one terminal outcome per request for the reliability measure, then keep attempt-level evidence for diagnosis. Never repair the discrepancy merely by changing the chart label or averaging the two percentages.

Check it quickly

Save this as metrics-logs.py and run it with Python 3 and an official promtool binary on PATH. The retained 2026-10-15 review ran the complete example with Python 3.12.14 and promtool 3.14.0. The actual validator reads the generated exposition from standard input; no server, container, or network connection is created. The synthetic population is complete by construction, while its diagnostic log subset is deliberately selected rather than randomly sampled.

import json
import subprocess
from collections import Counter

# A complete synthetic population, with exactly one terminal outcome per request.
requests = [{"request_id": f"r{i}", "route": "/checkout",
 "outcome": "error" if i in {7, 18, 53, 94} else "success"}
 for i in range(1, 101)]
counter = Counter(request["outcome"] for request in requests)
selected_logs = [request for i, request in enumerate(requests, 1)
 if i in {1, 7, 18, 33, 55}]
logged_errors = sum(record["outcome"] == "error" for record in selected_logs)
text = "# HELP demo_requests_total Completed synthetic requests.\n# TYPE demo_requests_total counter\n"
for outcome, count in sorted(counter.items()):
 text += f'demo_requests_total{{route="/checkout",outcome="{outcome}"}} {count}\n'
version = subprocess.run(["promtool", "--version"], capture_output=True, text=True, check=True).stdout.splitlines()[0]
result = subprocess.run(["promtool", "check", "metrics"], input=text,
 text=True, capture_output=True, timeout=30)
assert result.returncode == 0, result.stdout + result.stderr
population_rate = counter["error"] / len(requests)
selected_rate = logged_errors / len(selected_logs)
assert population_rate == 0.04 and selected_rate == 0.4
assert "request_id" not in text and len(counter) == 2
# Full event coverage would recover the same aggregate; these five logs cannot.
assert sum(r["outcome"] == "error" for r in requests) == counter["error"]
print(json.dumps({"requests": len(requests), "errors": counter["error"],
 "population_error_rate": population_rate, "exposed_metric_series": len(counter),
 "selected_logs": len(selected_logs), "selected_log_errors": logged_errors,
 "selected_log_error_rate": selected_rate,
 "exposition_validated_by": version}, indent=2))

The result should report one hundred requests, four errors, and a population fraction of 0.04. Five selected log records contain two errors, producing 0.4. The exposition has two counter series distinguished by outcome, and its labels contain no request identifier. Promtool validates the exported format; the Python assertions establish the arithmetic and the deliberate difference in populations. This is not a live scrape, counter-reset test, or storage-cost benchmark.

For real investigations, use bounded metric dimensions to locate the affected route, outcome, or deployment. Follow a correlation identifier into logs or traces when individual context is needed. Keep that identifier out of ordinary metric labels unless a justified cardinality budget supports it; exemplars can provide a separate link when the stack supports them. Finally, document event completeness and selection rules next to derived rates. A chart becomes trustworthy when its unit, population, and time basis are clear enough for another person to reproduce its meaning.

Sources

OpenTelemetry metrics, measurements and aggregation over a population, checked 2026-09-13.
OpenTelemetry logs, event records and their relationship to other telemetry, checked 2026-09-13.
Prometheus metric and label naming, bounded labels and the cost of high-cardinality dimensions, checked 2026-09-13.

Alerts tell you what to read

Tue, 22 Sep 2026 00:00:00 +0000

An alert should give its reader a useful first investigation, not merely a red indicator. Name the observed condition, affected scope, persistence window, responsible team, and next evidence to inspect. These details are part of the rule’s behavior: test them with the trigger instead of treating notification text as decoration added after the incident.

State exactly what was observed and make the next check accessible. Scrape failure alone does not prove API downtime, and a recent change is a lead rather than a cause. The historically passed synthetic promtool checks cover rules and annotations only; no new example, live scrape or delivery test was run.

Separate observations from explanations

A failed scrape means Prometheus could not collect that target’s metrics. It does not, by itself, prove that the public API is unavailable. A service can answer users while its metrics handler fails, and a successful scrape can coexist with broken business requests. Combine symptom-oriented checks with internal evidence, and state precisely which observation triggered this alert.

Recent releases are useful investigation leads, not proof of causality. A certificate expiry, dependency failure, resource shortage, or network fault can occur without a deployment. Link the relevant change history alongside status, logs, and user-facing measurements. Do not label the latest commit as the cause before establishing the connection.

Make identity stable and context useful

Prometheus evaluates each resulting label set as an alert instance. Put stable routing attributes, such as service and team, in labels. Put a current value, explanation, and runbook link in annotations. A changing timestamp or measurement in a label creates changing identities and can prevent the pending duration from completing. An annotation can explain a changing situation without redefining which alert it is.

The runbook should begin with accessible, read-only checks: inspect the target’s reported error, compare the scrape endpoint with the public request path, then correlate relevant logs and recent changes. It should identify a healthy result, the next branch if that result is absent, and the point for escalation. Keep credentials and sensitive query parameters out of notification links. Confirm that the on-call reader can actually access the destination.

When it breaks

An alert can fire correctly but route nowhere, link to a removed runbook, or lose its instance label during aggregation. A noisy threshold interrupts people for conditions requiring no decision. Choose a page only when timely human action is justified; otherwise use the appropriate dashboard, ticket, or automated response. Grouping and inhibition in Alertmanager are separate from deciding whether a Prometheus rule is firing.

The optional for duration requires the expression to remain active across evaluations before firing. It is not a moving-average window and does not include notification delivery delays. If the expression stops producing the label set, this rule deactivates. Missing data is especially dangerous: a stale or absent series is not an explicit zero. Silence from the rule is therefore not sufficient proof of recovery; monitor collection and expected target inventory separately.

Check it quickly

Use promtool from a Prometheus release; this example was executed with version 3.14.0. In an empty local directory save the next two blocks as alerts.yml and alerts.test.yml. The rule deliberately models scrape availability, not an API availability SLO. Its severity and team labels are test data, and running these tests sends no notification.

groups:
 - name: api-scrape
 rules:
 - alert: APIScrapeUnavailable
 expr: up{job="api"} == 0
 for: 2m
 labels:
 severity: page
 team: platform
 annotations:
 summary: 'Cannot scrape {{ $labels.job }} on {{ $labels.instance }}'
 description: 'up == 0 for 2m. Inspect the target error and /metrics handler; scrape failure alone does not prove API downtime.'
 runbook_url: https://www.xbcatm.com/posts/a-purpose-built-health-check-is-worth-keeping/

The tests use synthetic one-minute samples. A zero begins pending at minute zero and becomes firing at minute two. Minute three is healthy; a later failure must wait through its own pending period. Expected annotations verify the expanded target name, troubleshooting text, and runbook URL rather than merely counting alerts.

rule_files:
 - alerts.yml
evaluation_interval: 1m
tests:
 - name: sustained failure, recovery, and recurrence
 interval: 1m
 input_series:
 - series: 'up{job="api",instance="api.example:9090"}'
 values: '0 0 0 1 0 0 0'
 alert_rule_test:
 - eval_time: 0m
 alertname: APIScrapeUnavailable
 exp_alerts: []
 - eval_time: 1m
 alertname: APIScrapeUnavailable
 exp_alerts: []
 - eval_time: 2m
 alertname: APIScrapeUnavailable
 exp_alerts: &firing
 - exp_labels:
 job: api
 instance: api.example:9090
 severity: page
 team: platform
 exp_annotations:
 summary: 'Cannot scrape api on api.example:9090'
 description: 'up == 0 for 2m. Inspect the target error and /metrics handler; scrape failure alone does not prove API downtime.'
 runbook_url: https://www.xbcatm.com/posts/a-purpose-built-health-check-is-worth-keeping/
 - eval_time: 3m
 alertname: APIScrapeUnavailable
 exp_alerts: []
 - eval_time: 5m
 alertname: APIScrapeUnavailable
 exp_alerts: []
 - eval_time: 6m
 alertname: APIScrapeUnavailable
 exp_alerts: *firing
 promql_expr_test:
 - expr: 'ALERTS{alertname="APIScrapeUnavailable",alertstate="pending"}'
 eval_time: 1m
 exp_samples:
 - labels: 'ALERTS{alertname="APIScrapeUnavailable",alertstate="pending",instance="api.example:9090",job="api",severity="page",team="platform"}'
 value: 1
 - name: transient failure does not fire
 interval: 1m
 input_series:
 - series: 'up{job="api",instance="api.example:9090"}'
 values: '1 0 1 1'
 alert_rule_test:
 - eval_time: 2m
 alertname: APIScrapeUnavailable
 exp_alerts: []
 - eval_time: 3m
 alertname: APIScrapeUnavailable
 exp_alerts: []
 - name: stale series is not an explicit zero
 interval: 1m
 input_series:
 - series: 'up{job="api",instance="api.example:9090"}'
 values: '0 stale _ _'
 alert_rule_test:
 - eval_time: 2m
 alertname: APIScrapeUnavailable
 exp_alerts: []
 promql_expr_test:
 - expr: 'ALERTS{alertname="APIScrapeUnavailable"}'
 eval_time: 2m
 exp_samples: []

Run these commands from that directory:

promtool --version
promtool check rules alerts.yml
promtool test rules alerts.test.yml

The rule check and unit tests must report SUCCESS. The cases cover sustained failure, pending state, recovery, recurrence, a transient failure, and a stale target that disappears before firing. The stale case intentionally proves a detection gap instead of pretending missing samples are healthy. All expected labels and annotations must match.

For a negative control, change the rule’s runbook URL while leaving the expectations unchanged; the test should fail. Restore it and repeat successfully. Changing for from two minutes to one must fail the minute-one expectation; smaller changes may be invisible at this evaluation interval. These checks validate rule evaluation and explanatory context, not live scraping, authorization on linked pages, Alertmanager grouping, or actual pager delivery. Exercise those separately before enabling the rule in production.

Sources

Prometheus: alerting rules, label identity, pending duration, and templated annotations, checked 2026-09-09.
Prometheus: unit testing rules, synthetic samples, stale markers, and expected alert output, checked 2026-09-09.
Google SRE: monitoring distributed systems, actionable pages, symptoms, and low-noise monitoring, checked 2026-09-09.

Histograms preserve the latency shape

Sun, 20 Sep 2026 00:00:00 +0000

A mean can make an uncomfortable service look fast. If ninety-five requests take 50 milliseconds and five take two seconds, the average is only 147.5 milliseconds. A latency histogram retains more information about that slow minority without storing every request. What it preserves is a bucketed approximation of the distribution, not the complete original observations or an automatically exact percentile.

The same source observations produce different classic-histogram estimates when the bucket boundaries change. Narrower buckets reduce the error in this fixture but do not recreate the raw percentile. An aligned threshold count is exact for captured observations. Standard exponential native buckets use a different interpolation model. Neither the Python fixture nor its historical Prometheus comparison was rerun for this drawing.

Know what the buckets contain

In a Prometheus classic histogram, a bucket’s le boundary includes every observation less than or equal to that boundary. Counts are cumulative: a 50-millisecond request contributes to the 100-millisecond bucket and every larger one. The positive-infinity bucket covers all observations, while the count and sum components provide the sample population and total observed duration. Adding cumulative bucket counts together double-counts observations.

Subtracting neighboring cumulative counts recovers the counts in disjoint intervals. You still do not know where observations lie inside each interval. Compatible histograms can be aggregated by adding corresponding bucket counts before estimating a percentile. Averaging independently calculated instance percentiles does not recover the combined distribution, even when every individual percentile was accurately calculated.

Resolution determines the answer

For an ordinary positive classic bucket, histogram_quantile interpolates inside the bucket containing the requested rank, assuming observations are distributed uniformly within that interval. Real latency may cluster near either edge. The returned number is therefore an estimate tied to the bucket width, not a newly recovered measurement. Place useful boundaries near important operating thresholds instead of spreading every boundary across irrelevant ranges.

A bucket exactly at a 300-millisecond objective can directly count the observations meeting that threshold. This can answer an SLO question more faithfully than asking whether an interpolated p99 happened to land below it. Choosing more boundaries increases the number of classic bucket series per label combination. Modern Prometheus also supports native histograms, which its current guidance prefers when the instrumentation and surrounding system support them. Standard exponential native buckets use a different interpolation model; the exercise here deliberately isolates classic buckets.

When it breaks

An upper bucket that is too wide makes a small movement of observations produce an implausibly large movement of the estimated tail. Missing the required positive-infinity bucket can make the classic quantile calculation invalid. If the requested rank falls into that final infinite bucket, Prometheus returns the preceding finite boundary rather than inventing an infinite latency estimate. None of these behaviors identifies the exact slowest request.

Empty populations have no observed percentile. Mismatched boundaries across instances, mixed observation units, or invalid nonmonotonic cumulative counts also undermine interpretation. When querying cumulative counters over time, calculate appropriate increases or rates before aggregation so resets are handled. A distribution over all requests additionally needs explicit outcome semantics: fast failures can improve the apparent latency while the service becomes less useful.

Check it quickly

Save this as histogram-shape.py and run Python 3. It calculates exact nearest-rank p99 from a synthetic raw sample, then independently estimates p99 from two classic bucket layouts. The compact interpolation function is restricted to this positive, nonempty fixture and asserts that the selected bucket is finite; it is not a replacement for Prometheus’s complete quantile implementation.

from math import ceil, inf
from statistics import mean

values = [0.05] * 95 + [2.0] * 5
coarse = [0.1, 0.3, 1.0, 5.0, inf]
fine = [0.1, 0.3, 1.0, 2.0, 5.0, inf]

def cumulative(bounds):
 return [sum(value <= bound for value in values) for bound in bounds]

def estimate_p99(bounds):
 counts = cumulative(bounds)
 rank = 0.99 * counts[-1]
 lower, previous = 0.0, 0
 for upper, count in zip(bounds, counts):
 if count >= rank:
 assert upper != inf and count > previous
 return lower + (upper - lower) * (rank - previous) / (count - previous)
 lower, previous = upper, count
 raise AssertionError('rank not covered')

exact = sorted(values)[ceil(0.99 * len(values)) - 1]
counts = cumulative(coarse)
increments = [counts[0]] + [right - left for left, right in zip(counts, counts[1:])]
assert counts == [95, 95, 95, 100, 100]
assert increments == [95, 0, 0, 5, 0] and sum(increments) == 100
assert exact == 2.0 and abs(estimate_p99(coarse) - 4.2) < 1e-12
assert abs(estimate_p99(fine) - 1.8) < 1e-12
print(f"mean={mean(values):.4f}s; raw nearest-rank p99={exact:.1f}s")
print(f"coarse cumulative counts={counts}; sum_of_buckets={sum(counts)}")
print(f"classic linear p99: coarse={estimate_p99(coarse):.1f}s; fine={estimate_p99(fine):.1f}s")
print(f"at or below 0.3s={counts[1]}/{counts[-1]}")

The raw p99 is two seconds, but the coarse buckets estimate 4.2 seconds because the relevant observations are somewhere between one and five seconds. Adding a two-second boundary changes the estimate to 1.8 seconds. The narrower bucket improves the error in this sample, yet the estimate still differs from the exact raw percentile; extra resolution does not recreate the original values. The original recorded comparison also matched histogram_quantile in Prometheus 3.5.1 within a numerical tolerance of 1e-12.

The cumulative counts sum to 485 although only one hundred requests exist. Their disjoint increments sum to one hundred, and the 0.3-second bucket gives the exact threshold result of 95 out of 100 for this dataset. Read those outputs together: histogram counts can answer aligned threshold questions precisely while percentile interpolation remains approximate.

Sources

Prometheus: Histograms and Summaries, distribution representations, aggregation, and native histograms, checked 2026-09-13.
Prometheus: histogram_quantile, classic bucket interpolation and boundary cases, checked 2026-09-13.
Google SRE: Monitoring Distributed Systems, latency tails and meaningful monitoring, checked 2026-09-13.

HEAD should describe the same representation as GET

Mon, 22 Jun 2026 00:00:00 +0000

The HEAD method asks for the response a server would send to GET, without transferring the response content. It is useful for checks that need status, validators, type, or length, but only when the metadata remains aligned with the real GET path.

Match the URL, request headers, release, and content coding before comparing HEAD with GET. Permitted omissions are not automatic failures. Keep a small GET body or checksum check, because plausible metadata cannot prove that the payload is intact.

Compare a normalized header set

Dynamic headers such as Date will differ between requests. Compare the fields that describe the selected representation:

url=https://www.example.test/downloads/manual.pdf

curl -sS -I "$url" > head.headers
curl -sS -D get.headers -o /dev/null "$url"

for file in head.headers get.headers; do
 printf '== %s\n' "$file"
 awk '
 /^HTTP\// ||
 tolower($1) ~ /^(content-type:|content-length:|content-encoding:|cache-control:|etag:|last-modified:|vary:)$/ { print }
 ' "$file"
done

Run this in a disposable working directory; the two header files are overwritten. Match the URL, request headers, and release version. Compression negotiation belongs in the comparison: send the same Accept-Encoding value to both requests.

HEAD should describe the corresponding GET, but RFC 9110 permits omission of fields determined only while generating content. Missing Content-Length alone is not a failure. If present on HEAD, it must be the length the corresponding GET content would have, not zero just because HEAD has no body. Compare encoded lengths consistently and investigate changed validators or types rather than requiring byte-identical header lists.

Do not use HEAD as the only release check

A server can synthesize plausible headers while the file is missing, unreadable, or truncated. Keep one small GET in the smoke test and verify its body or checksum. HEAD reduces transfer cost; it does not prove the payload.

Check redirects explicitly

Without --location, curl -I reports the first response. That is useful when testing redirect policy. With --location, it prints header blocks for the followed responses, including intermediate redirects and the final destination. Record which question the command answers:

curl -sS -I --max-redirs 0 https://old.example.test/manual
curl -sS -I --location https://old.example.test/manual

Treat method-specific failures as configuration defects

A static page that returns 200 to GET and 405 or 404 to HEAD usually exposes a proxy or routing inconsistency. Fix the method handling instead of teaching monitors to ignore it, unless the endpoint contract deliberately disallows HEAD.

Sources

RFC 9110 — HTTP Semantics, HEAD method semantics, checked 2026-10-15.
curl manual, --head, --include, and redirect options, checked 2026-10-15.

Certificate expiry is a date, not a dashboard color

Wed, 10 Jun 2026 00:00:00 +0000

An HTTPS process can be healthy while the public certificate is close to expiry, issued for the wrong name, or served differently on another edge. A useful check asks the public endpoint for the certificate it would give to a real client.

Inspect the certificate a real client sees, then verify separately. Expiry alerts need a service-specific response threshold, handshake checks, renewal-failure alerts, and sampling of relevant edges and address families.

Include the hostname

TLS virtual hosting depends on the server name. Pass the hostname as SNI and inspect the peer certificate:

host=www.example.test
openssl s_client -connect "$host:443" -servername "$host" </dev/null 2>/dev/null \
 | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256

The command is read-only. It does not renew anything and it does not prove that every intermediate certificate is accepted by every client. It does give you the not-before, not-after, subject, issuer, and fingerprint of the certificate presented for that hostname.

Alert on a threshold

Choose a threshold that leaves time for a failed renewal, a DNS problem, and a human response. The exact number belongs to the service’s recovery target; a universal “30 days” rule is not a substitute for that decision. The companion TLS expiry check prints the expiry and exits non-zero when the remaining interval is below the threshold you pass.

Run the check from outside the host when possible. A local process may see a different listener, certificate file, or proxy path than a visitor on the public network.

Separate observation from verification

SNI selects a virtual host; it does not verify that the certificate matches that name. This pipeline prints a certificate even when s_client reports verification problems, and stderr is suppressed. Use a separate hostname-and-chain verification check with visible diagnostics and a known trust store. Do not turn this inspection pipeline’s final exit code into a TLS-health verdict.

The companion helper tests notAfter with OpenSSL -checkend; it does not establish hostname validity, trust, revocation status, or that notBefore has arrived. Pair expiry alerts with handshake checks and renewal-failure alerts. Choose a threshold smaller than the certificate’s normal lifetime, especially for short-lived certificates, and sample each relevant edge/address family.

Sources

Caddy Automatic HTTPS, automatic TLS certificate management, checked 2026-10-15.
OpenSSL s_client documentation, SNI and certificate inspection options, checked 2026-10-15.
OpenSSL x509 documentation, certificate parsing, dates, and fingerprints, checked 2026-10-15.

Monitoring · XB Field Notes

iotop shows which process is burning the disks

Per-process counters

See whatever sweeps the disk

Watch before you act

When it breaks

Check it quickly

Sources

Grafana variables need the query language to agree

Check the resulting expression

When it breaks

Check it quickly

Sources

Prometheus rate estimates counter growth per second

Sampling determines what can be inferred

The selected range is part of the meaning

When it breaks

Check it quickly

Sources

Prometheus rules track alert state before notification delivery

Vector membership activates the alert

Waiting is measured at evaluations

When it breaks

Check it quickly

Sources

Cardinality is a metric budget

Count combinations that actually exist

Include the metric representation

When it breaks

Check it quickly

Sources

Dashboards summarize; they do not magnify

Give the page a reading order

Preserve weighting and coverage

When it breaks

Check it quickly

Sources

External labels identify outgoing Prometheus data

Local storage and outgoing identity differ

Existing labels take precedence

Stable labels do not provide deduplication

When it breaks

Check it quickly

Sources

Prometheus staleness and alert waiting are different mechanisms

A marker is not a five-minute timer

Choose the absence question deliberately

When it breaks

Check it quickly

Sources

Prometheus histograms retain a bucketed distribution

Classic histograms are several related series

Aggregate buckets before estimating a quantile

When it breaks

Check it quickly

Sources

The golden signals of health

Separate experience from demand

Explain failure and resource pressure

When it breaks

Check it quickly

Further reading

Sources

Metrics and logs need an explicit population

Decide what the measurement includes

When it breaks

Check it quickly

Further reading

Sources

Alerts tell you what to read

Separate observations from explanations

Make identity stable and context useful

When it breaks

Check it quickly

Further reading

Sources

Histograms preserve the latency shape

Know what the buckets contain

Resolution determines the answer

When it breaks