Observability · XB Field Notes

Baggage carries context across an explicit boundary

Mon, 27 Mar 2028 00:00:00 +0000

OpenTelemetry baggage is a context-associated collection of string key-value pairs. It lets information known near a request’s entry point travel to downstream operations: a deployment region or a rollout cohort, for example. Unlike a span attribute, baggage is intended for propagation. Unlike trace context, it describes application-selected information rather than the identifiers and flags connecting spans. These mechanisms can travel together without being the same data structure.

Propagation still needs a working boundary. An outgoing client injects context into a carrier such as HTTP headers; the receiver extracts that carrier into its local context. Instrumentation can perform these steps automatically when configured, while manually instrumented code must do them explicitly. Merely creating baggage in one process does not deliver it to an unrelated process, a background task, or an external service.

Choose what crosses and what gets recorded

Baggage entries do not automatically become span, log, or metric attributes. A downstream operation must read the value and copy it deliberately, or use a configured processor implementing that policy. This distinction matters when a team expects a searchable attribute but has only enabled propagation. It also prevents the opposite mistake: copying every incoming key into telemetry without considering its meaning, size, or cardinality.

Context operations return a new context rather than changing every existing reference. Keep the returned context and pass or attach it at the intended boundary. When removing entries for an outgoing request, preserve the trace context that should accompany that request. Treat an outbound allowlist and a telemetry-attribute allowlist as explicit decisions; the values appropriate for one receiver need not be suitable for every later destination.

When it breaks

Baggage has no built-in proof that a value was issued by a trusted component. A caller can supply a header claiming an expensive account tier or a privileged role. Parsing that header successfully establishes its representation, not its authority. Authentication and authorization must use their own trusted inputs. Small, plausible-looking values deserve the same trust analysis as obviously suspicious ones.

Unbounded baggage also increases request-header sizes and can spread sensitive data to unintended recipients, including third-party APIs reached by automatic instrumentation. Limit names and values, review the destinations that receive them, and use propagators for wire encoding. A successful round trip between two copies of one SDK is narrower evidence than interoperability across every language and proxy in a production path. Test those actual boundaries when they matter.

Baggage propagation and trust: mechanism and evidence boundaries.

Check it quickly

Create a virtual environment, install opentelemetry-sdk==1.35.0, save the script as baggage.py, and execute it with that environment’s Python. The author reported an earlier Python 3.14.4 run; retained exact records cover Python 3.12.14 with SDK 1.35.0. This source review did not rerun it. It uses the real W3C trace-context and baggage propagators with an in-memory span exporter. An explicit always-on sampler avoids inherited sampling settings. Keep assertions enabled, without -O or PYTHONOPTIMIZE: force_flush is called inside assert. It does not start an HTTP server: the dictionary is the carrier that an HTTP client would subsequently send.

import json

from opentelemetry import baggage, trace
from opentelemetry.context import Context
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ALWAYS_ON
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

base = Context()
raw = baggage.set_baggage("region", "eu-west", context=base)
raw = baggage.set_baggage("release", "canary/2", context=raw)
raw = baggage.set_baggage("internal_note", "synthetic-only", context=raw)
assert not baggage.get_all(base)
propagator = CompositePropagator([TraceContextTextMapPropagator(), W3CBaggagePropagator()])
provider = TracerProvider(sampler=ALWAYS_ON)
exporter = InMemorySpanExporter()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("baggage-lab")
with tracer.start_as_current_span("parent", context=Context()) as parent:
 outgoing = trace.set_span_in_context(parent, baggage.clear(context=raw))
 for key in ("region", "release"):
 outgoing = baggage.set_baggage(key, baggage.get_baggage(key, raw), context=outgoing)
 carrier = {}
 propagator.inject(carrier, context=outgoing)
 received = propagator.extract(carrier, context=Context())
 assert set(baggage.get_all(received)) == {"region", "release"}
 assert baggage.get_baggage("region", received) == "eu-west"
 assert baggage.get_baggage("release", received) == "canary/2"
 assert baggage.get_baggage("internal_note", raw) == "synthetic-only"
 assert trace.get_current_span(received).get_span_context().trace_id == parent.get_span_context().trace_id
 with tracer.start_as_current_span("inherited", context=received):
 pass
 with tracer.start_as_current_span("copied", context=received) as span:
 span.set_attribute("deployment.region", baggage.get_baggage("region", received))
assert provider.force_flush()
spans = {span.name: span for span in exporter.get_finished_spans()}
assert not spans["inherited"].attributes
assert spans["copied"].attributes == {"deployment.region": "eu-west"}
independent = propagator.extract({"traceparent": "invalid", "baggage": "region=eu-west"},
 context=Context())
assert not trace.get_current_span(independent).get_span_context().is_valid
assert baggage.get_baggage("region", independent) == "eu-west"
untrusted = propagator.extract({"baggage": "tier=gold"}, context=Context())
assert baggage.get_baggage("tier", untrusted) == "gold"
print(json.dumps({"wire_fields": sorted(carrier),
 "wire_baggage": carrier["baggage"],
 "automatic_span_attributes": dict(spans["inherited"].attributes),
 "explicit_span_attributes": dict(spans["copied"].attributes),
 "baggage_survives_invalid_traceparent": True,
 "caller_supplied_tier_was_parsed": True}, indent=2))
provider.shutdown()

The original context contains region, release, and a synthetic internal note. The outgoing context keeps only the first two entries while retaining the parent span. The extracted region is eu-west, and the release value canary/2 round-trips through the encoded header. Assertions also confirm that constructing the filtered context did not remove the note from the original context or change the empty base context.

Two child spans demonstrate the recording boundary. inherited receives baggage but has no attributes; copied contains only the explicitly assigned deployment.region. A malformed traceparent is tested separately: its trace context is invalid, while the independently encoded baggage still extracts successfully. Finally, a caller-supplied tier=gold is accepted as data without any identity check. These results cover serialization, filtering, and recording in this SDK fixture; the trace assertion compares trace_id, not every trace-context field. Before relying on the same policy across services, verify the real client instrumentation, receiver extraction, outbound filtering, and the attribute mapping used by the destination backend.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
OpenTelemetry baggage, baggage is separate from telemetry attributes and lacks built-in integrity checks, checked 2026-10-15.
W3C Baggage, HTTP representation, value encoding, and propagation limits, checked 2026-10-15.
W3C Trace Context, trace identity travels separately from application baggage, checked 2026-10-15.
OpenTelemetry Python sampling, explicit sampler selection and environment defaults, checked 2026-10-15.

Annotations explain an alert without changing its identity

Tue, 22 Feb 2028 00:00:00 +0000

Alert labels identify which condition is being tracked and provide fields for routing and grouping. Annotations explain that condition: a current value, a concise summary, or a runbook link. Putting changing prose in labels can split one ongoing problem into many identities. Putting routing fields only in annotations can leave a route unable to recognize the alert’s intended owner.

The illustration follows the article’s unchanged Alertmanager 0.28.1 fixture and its explicitly supplied template data. Stable labels preserve identity; common fields are an intersection, and member details remain in Alerts. The integration rerun is historically blocked and was not executed here. Annotation changes alone do not guarantee immediate notification or prove delivery.

Separate identity from the current explanation

A queue alert might keep alertname, service, and instance as stable labels while its summary changes from six waiting items to eight. Alertmanager can update the annotation for the same alert fingerprint rather than adding another alert. A fingerprint is derived from identity labels; it is not a hash of the rendered notification text.

This distinction also matters before Alertmanager receives the alert. Prometheus tracks its pending duration for an alert label set. A label containing a changing current value can keep creating new identities and prevent the intended waiting period from completing. Put that changing value in an annotation unless it deliberately defines a different alert condition. Review templates and rule labels together when diagnosing repeated pending alerts.

Notification templates receive several scopes of data. GroupLabels contains the labels chosen for grouping. CommonLabels contains label pairs common to all alerts in the notification. CommonAnnotations similarly contains annotations shared by the group. The Alerts list retains the labels and annotations of individual members, allowing a notification to explain their different states.

If two replicas have summaries describing different queue sizes, there is no shared summary to put in CommonAnnotations. A template that prints only that field can omit the very information an operator needs. Use common context for the heading, then iterate over individual alerts for changing summaries. A shared runbook link can remain common even when the individual values differ.

Do not assume a grouping label such as instance exists in every group. Grouping by service and alert name intentionally combines instances. A template should reflect the selected grouping contract and handle an absent common field deliberately. Representative fixtures need multiple alerts, unequal annotations, and the relevant firing or resolved states, not just one idealized alert where every field happens to be common.

An annotation update is not itself a promise of an immediate notification. In Alertmanager 0.28.1, deduplication compares firing and resolved alert identities and the repeat interval; changing a summary alone does not create a new fingerprint. Keep rapidly changing detail available in the alert view without promising one message for every observed value.

When it breaks

A dynamic label creates a new fingerprint for every value change, increasing alert churn and potentially restarting pending state. A route matching team cannot use a team annotation in place of a label. An overly broad notification heading can also describe one instance while listing alerts from several services if grouping and template assumptions drift apart.

A template that parses successfully can still render an empty or misleading message for actual group data. Check the rendered text, not only syntax. Conversely, rendering successfully does not prove the integration accepts the message or that a notification was delivered. Keep credentials and sensitive request details out of annotations because they can be copied to every receiver selected by routing.

Check it quickly

Save this as annotations.py and run python3 annotations.py with Docker available. It uses Alertmanager 0.28.1, downloading the image if needed. The isolated server has a receiver with no delivery integrations. Two synthetic alerts are submitted through its actual API; one annotation is then updated without changing any labels.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


port = 9093
script = r"""set -eu
cat > /tmp/alertmanager.yml <<'ALERT_CONFIG'
route:
 receiver: preview
 group_by: [alertname, service]
 group_wait: 1h
 group_interval: 5m
 repeat_interval: 4h
receivers:
 - name: preview
ALERT_CONFIG
exec /bin/alertmanager --config.file=/tmp/alertmanager.yml --storage.path=/tmp/data \
 --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: request("GET", "/-/ready")[0] == 200)
 alerts = [{"labels": {"alertname": "QueueHigh", "service": "checkout", "instance": instance},
 "annotations": {"summary": "queue=" + str(depth), "runbook": "https://example.invalid/queue"}}
 for instance, depth in [("api-a", 6), ("api-b", 9)]]
 assert request("POST", "/api/v2/alerts", json.dumps(alerts))[0] == 200
 before = eventually(lambda: (a if len(a := get_json("/api/v2/alerts")) == 2 else False))
 fingerprints = {a["labels"]["instance"]: a["fingerprint"] for a in before}
 alerts[0]["annotations"]["summary"] = "queue=8"
 assert request("POST", "/api/v2/alerts", json.dumps([alerts[0]]))[0] == 200
 def updated_annotations():
 items = get_json("/api/v2/alerts")
 changed = any(x["labels"]["instance"] == "api-a" and
 x["annotations"]["summary"] == "queue=8" for x in items)
 return items if changed else None
 updated = eventually(updated_annotations)
 assert len(updated) == 2
 assert {a["labels"]["instance"]: a["fingerprint"] for a in updated} == fingerprints
 # Supply the documented notification Data shape to the real template renderer.
 data = {"receiver": "preview", "status": "firing",
 "groupLabels": {"alertname": "QueueHigh", "service": "checkout"},
 "commonLabels": {"alertname": "QueueHigh", "service": "checkout"},
 "commonAnnotations": {"runbook": "https://example.invalid/queue"},
 "alerts": [{"status": "firing", "labels": a["labels"], "annotations": a["annotations"]}
 for a in sorted(updated, key=lambda a: a["labels"]["instance"])]}
 template = '{{define "preview"}}{{.GroupLabels.service}}|common-summary={{with .CommonAnnotations.summary}}{{.}}{{else}}none{{end}}{{range .Alerts}}\n{{.Labels.instance}}:{{.Annotations.summary}}{{end}}{{end}}'
 for path, content in [("/tmp/preview.tmpl", template), ("/tmp/data.json", json.dumps(data))]:
 subprocess.run(["docker", "exec", "-i", name, "/bin/sh", "-c", "cat > " + path],
 input=content, text=True, capture_output=True, check=True, timeout=5)
 rendered = subprocess.run(["docker", "exec", name, "/bin/amtool", "template", "render",
 "--template.glob=/tmp/preview.tmpl", "--template.data=/tmp/data.json",
 '--template.text={{ template "preview" . }}'],
 text=True, capture_output=True, check=True, timeout=5).stdout.strip()
 assert rendered == "checkout|common-summary=none\napi-a:queue=8\napi-b:queue=9", rendered
 print(json.dumps({"alert_count": 2, "fingerprints_unchanged": True, "rendered": rendered}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The result confirms that two alerts remain and their fingerprints are unchanged after the update. The program then supplies the documented notification data structure to the real amtool template renderer. Its common annotations intentionally include the shared runbook but exclude the unequal summaries. This explicit template input tests rendering, not the server’s notification assembly or delivery pipeline.

The rendered heading says common-summary=none, followed by api-a:queue=8 and api-b:queue=9. Assertions verify the updated value and the separate per-alert details. All HTTP traffic remains on container loopback, with no host mounts or published ports, and the container is removed on exit. A production notification still needs validation using its actual route, template data, and receiver integration.

The operational fixture is unchanged here and was not rerun in this review. Its expected outputs and earlier observations remain distinct from current execution evidence.

Sources

Alertmanager: notification template reference, group labels, common annotations and individual alert data, checked 2026-10-15.
Prometheus: alerting rules, labels identify alerts while annotations carry explanatory content, checked 2026-10-15.
Alertmanager 0.28.1: README, local template rendering with amtool, checked 2026-10-15.
Alertmanager 0.28.1 notification pipeline, DedupStage identity and repeat checks, checked 2026-10-15.

pg_stat_statements ranks query work within a collection window

Sun, 12 Dec 2027 00:00:00 +0000

pg_stat_statements groups statement executions and accumulates their costs. It can identify a frequently executed short query whose total time is substantial, as well as an individually slow query. These are different questions. Neither ranking alone proves which change will improve application latency, and execution time is not the same as CPU consumption, storage latency, or end-to-end request duration.

Grouping is useful and bounded

PostgreSQL 18 normally replaces literal constants in representative query text with parameter symbols and groups statements with equivalent analyzed structures. The entry key includes user, database, query identifier, and whether execution is top-level. Text alone is not that identity. Different object resolution can separate identical text, and rare hash collisions can combine unrelated structures within the relevant grouping scope.

The representative text is diagnostic data, not an audit trail of every submitted statement. Some utility statements or normalization failures can retain literal values, so restrict access and avoid embedding secrets. Query identifiers are not promised stable across major versions or arbitrary catalog changes. PostgreSQL 18 also normalizes certain constant lists of differing lengths; do not assume identical behavior in another version without checking its manual.

Rank according to the question

Total execution time helps prioritize aggregate database work. Mean, maximum, and dispersion help describe individual calls, but they are not percentiles and cannot identify a cause by themselves. High variance could involve parameters, cache state, contention, or plan changes. Inspect the relevant plan and waits before declaring a bad plan or lock problem from one statistic.

Accumulated execution durations can exceed elapsed wall-clock time because concurrent sessions overlap. That is not proof of a restart or invalid measurement. Planning time is tracked separately when planning statistics are enabled. Shared-block reads mean blocks requested through PostgreSQL’s buffer manager; they can be satisfied by the operating-system cache. Multiplying by the server’s block size gives a requested-byte quantity, not physical-device traffic. I/O timing adds time measurements when enabled; the basic block counts do not depend on it.

When it breaks

The module must be loaded through shared_preload_libraries at server startup and its SQL extension created in the database used for inspection. Query identifier calculation must be available, normally through compute_query_id=auto or on; an external module can also supply identifiers. Missing configuration, restricted visibility, and disabled tracking are different failures. Read their actual errors rather than assuming every absent row means no workload occurred.

A useful delta requires comparable entry identities and collection windows. PostgreSQL 18 exposes per-entry stats_since, a separate minimum/maximum start time, and aggregate reset and deallocation information. A targeted reset or eviction can invalidate one entry without resetting the whole view. Nonnegative subtraction is not sufficient proof of continuity: enough work after a reset can hide the discontinuity. Treat new or changed windows as new baselines instead of subtracting across them.

Normalization is bounded by entry identity and collection continuity. The revised fixture binds all four identity fields; execution remains held, with no fresh SQL or timing result.

Check it quickly

Use a fresh, otherwise idle disposable PostgreSQL 18 cluster and database, with the module preloaded, built-in query identifiers enabled, top-level tracking, and the extension visible and installed or installable by the test role. Keep the role, search path and objects unchanged; allow no concurrent workload, resets or eviction. Run this SQL in one fresh session with errors treated as failures. Success removes its tables and view; errors can leave them, and a newly installed extension remains. No existing statistics are reset.

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
CREATE TABLE pgss_demo (id integer);
INSERT INTO pgss_demo SELECT generate_series(1,1000);
CREATE TEMP TABLE pgss_key (
 userid oid NOT NULL, dbid oid NOT NULL,
 toplevel boolean NOT NULL, queryid bigint NOT NULL CHECK (queryid <> 0),
 PRIMARY KEY (userid,dbid,toplevel,queryid)
);
DO $$
DECLARE first_plan json; second_plan json; first_id bigint; second_id bigint;
BEGIN
 IF current_setting('compute_query_id') NOT IN ('auto','on')
 OR current_setting('pg_stat_statements.track') <> 'top' THEN
 RAISE EXCEPTION 'requires built-in query identifiers and top-level tracking';
 END IF;
 EXECUTE 'EXPLAIN (VERBOSE, FORMAT JSON) SELECT sum(id) FROM pgss_demo WHERE id < 100'
 INTO STRICT first_plan;
 EXECUTE 'EXPLAIN (VERBOSE, FORMAT JSON) SELECT sum(id) FROM pgss_demo WHERE id < 500'
 INTO STRICT second_plan;
 first_id := (first_plan->0->>'Query Identifier')::bigint;
 second_id := (second_plan->0->>'Query Identifier')::bigint;
 IF first_id IS NULL OR first_id = 0 OR second_id IS DISTINCT FROM first_id THEN
 RAISE EXCEPTION 'expected matching nonzero query identifiers';
 END IF;
 INSERT INTO pgss_key
 SELECT (SELECT oid FROM pg_roles WHERE rolname=current_user),
 (SELECT oid FROM pg_database WHERE datname=current_database()),
 true,first_id;
END $$;
CREATE TEMP VIEW pgss_target AS
SELECT s.* FROM pg_stat_statements AS s JOIN pgss_key AS k
USING (userid,dbid,toplevel,queryid);
CREATE TEMP TABLE pgss_window AS
SELECT dealloc,stats_reset FROM pg_stat_statements_info;
SELECT sum(id) FROM pgss_demo WHERE id < 100;
CREATE TEMP TABLE pgss_first AS
SELECT userid,dbid,toplevel,queryid,calls,stats_since FROM pgss_target;
SELECT sum(id) FROM pgss_demo WHERE id < 500;
SELECT userid,dbid,toplevel,queryid,query,calls,rows,
 round(total_exec_time::numeric,3) AS total_ms,
 round(mean_exec_time::numeric,3) AS mean_ms,stats_since
FROM pgss_target;
DO $$ BEGIN
 IF (SELECT count(*) FROM pgss_first) <> 1
 OR (SELECT count(*) FROM pgss_target) <> 1
 OR (SELECT count(*) FROM pgss_window) <> 1 THEN
 RAISE EXCEPTION 'expected one identity-bound entry and collection baseline';
 END IF;
 IF NOT EXISTS (
 SELECT 1 FROM pgss_first AS a JOIN pgss_target AS b
 USING (userid,dbid,toplevel,queryid)
 CROSS JOIN pgss_window AS w CROSS JOIN pg_stat_statements_info AS i
 WHERE a.calls=1 AND b.calls=2 AND b.calls-a.calls=1
 AND b.stats_since=a.stats_since
 AND i.dealloc=w.dealloc AND i.stats_reset=w.stats_reset) THEN
 RAISE EXCEPTION 'expected calls 1 then 2 in one unchanged collection window';
 END IF;
END $$;
SELECT userid,dbid,toplevel,queryid,calls,
 round(total_exec_time::numeric,3) AS total_ms,
 round(mean_exec_time::numeric,3) AS mean_ms
FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;
SELECT userid,dbid,toplevel,queryid,calls,
 round(mean_exec_time::numeric,3) AS mean_ms
FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;
SELECT dealloc,stats_reset FROM pg_stat_statements_info;
SELECT round((sum(shared_blks_read)::numeric *
 current_setting('block_size')::numeric)/1073741824,6) AS requested_read_gib
FROM pg_stat_statements;
DROP VIEW pgss_target;
DROP TABLE pgss_first,pgss_window,pgss_key;
DROP TABLE pgss_demo;
SELECT 'pgss fixture completed' AS completion;

The sums should be 4,950 and 124,750. Non-ANALYZE EXPLAIN obtains both query identifiers without executing the sums; they must match and be nonzero. The target view joins that identifier with the current user, database and top-level flag, never representative text. Snapshots require one call then two, with unchanged stats_since and global reset/deallocation metadata. These guards rely on the idle-cluster assumptions; they are not collision-proof attribution or concurrent monitoring. The general rankings may agree; their sort keys differ. Numeric casts permit two-argument rounding of double-precision timings.

Retained PostgreSQL 18.6 logs cover the predecessor text-filter SQL in single-user mode; they do not execute this revised identity-bound fixture. Pure source and synthetic checks add no PostgreSQL runtime evidence. Client transport, concurrent sessions, production overhead and monitoring permissions remain untested. Durations are observations, not stable benchmarks. Before acting on production rankings, collect comparable snapshots, scope the user and database, inspect the workload, and measure application outcomes after changes.

Sources

PostgreSQL 18 pg_stat_statements, group keys, timing types, normalization, reset and per-entry windows, checked 2026-10-15.
PostgreSQL 18 monitoring statistics, cumulative observations and buffer reads versus physical I/O, checked 2026-10-15.
PostgreSQL 18 statistics configuration, query identifiers and optional I/O timing, checked 2026-10-15.
PostgreSQL 18 EXPLAIN, VERBOSE query identifiers and non-ANALYZE planning, checked 2026-10-19.
PostgreSQL 18 PL/pgSQL statements, dynamic EXPLAIN and strict single-row capture, checked 2026-10-19.

Read procfs as documented views of a changing process

Fri, 29 Oct 2027 00:00:00 +0000

Linux procfs exposes process and kernel information through filesystem interfaces. Many entries generate their contents when read instead of storing ordinary disk-file bytes. Each entry has a documented meaning, permission model, and format. Reading several entries does not freeze the process or create a transactionally consistent snapshot, so useful observations must be tied to the right process instance and interpretation.

A successful readlink does not guarantee a successful open through that link. Linux separately checks the referenced inode’s permissions. Conversely, inspecting a descriptor does not consume its owner’s data merely by reading the link text. Distinguish inspecting metadata from opening and reading the referenced object, especially for pipes and other stateful streams.

Each procfs entry is a documented, permission-checked view. Read times, process identity, entry-specific parsing and object lifetime determine what the observation can establish.

When it breaks

Ordinary file assumptions can fail immediately. A proc entry can report size zero yet return useful content when read. A per-process directory can disappear as the process exits, and a numerical PID can eventually identify another instance. Treating a failed read as proof of a broken installation, or assuming a PID forever names one workload, can misdiagnose normal lifecycle changes.

Formats require entry-specific parsing. status contains labeled fields, while stat uses a positional layout with a parenthesized command name that can contain spaces and closing parentheses. Splitting the whole stat line on whitespace can shift the expected fields. cmdline and environ use NUL separators, so ordinary line or space splitting loses argument and value boundaries.

The views also have semantic limits. environ describes the environment region established at program execution, not a guaranteed live dictionary of every later environment change. Command-line contents can be changed by the process. A descriptor link ending in (deleted) does not mean the open object has vanished. These entries are evidence about specific state, not interchangeable proofs of identity or access.

Check it quickly

Save this as procview.py and run python3 procview.py in a disposable Linux environment. The 2026-10-15 review recorded this fixture passing on Linux 6.18.44 and Python 3.12.14; it was not rerun here. The script creates one controlled child with a small, explicitly supplied environment, reads only its own and that child’s proc entries, and performs no writes to kernel tuning interfaces:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,subprocess,sys

worker=r'''
from pathlib import Path
import ctypes,json,os,sys
assert ctypes.CDLL(None).prctl(15,b'worker ) sample',0,0,0)==0
os.environ['XB_PROC_TEST']='changed-after-start'
path=Path(sys.argv[1]);path.write_bytes(b'fixture-data')
with path.open('rb') as handle:
 info=os.fstat(handle.fileno());path.unlink()
 print(json.dumps({'fd':handle.fileno(),'device':info.st_dev,'inode':info.st_ino,'current_environment':os.environ['XB_PROC_TEST']}),flush=True)
 assert sys.stdin.readline()=='done\n'
'''
with TemporaryDirectory(prefix='xbcatm-proc-') as directory:
 path=Path(directory)/'data'
 process=subprocess.Popen([sys.executable,'-c',worker,str(path),'argument with spaces'],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True,env={'PATH':'/usr/bin:/bin','XB_PROC_TEST':'fixture-value'})
 try:
 metadata=json.loads(process.stdout.readline());root=Path('/proc')/str(process.pid)
 status_path=root/'status';status_text=status_path.read_text()
 assert status_path.stat().st_size==0 and len(status_text)>0
 status=dict(line.split(':',1) for line in status_text.splitlines() if ':' in line)
 assert int(status['Pid'])==process.pid and int(status['PPid'])==os.getpid()
 raw=(root/'stat').read_text();pid_text,rest=raw.split(' (',1);comm,fields=rest.rsplit(') ',1)
 fields=fields.split();assert int(pid_text)==process.pid and comm=='worker ) sample'
 assert int(fields[1])==os.getpid() and int(fields[19])>0
 assert raw.split()[2] not in ('R','S','D','T','Z')
 print('proc files can have size zero but readable data; stat comm can contain spaces and a closing parenthesis')
 args=(root/'cmdline').read_bytes().rstrip(b'\0').split(b'\0')
 environment=(root/'environ').read_bytes().rstrip(b'\0').split(b'\0')
 assert args[-1]==b'argument with spaces' and b'XB_PROC_TEST=fixture-value' in environment
 assert metadata['current_environment']=='changed-after-start'
 assert os.readlink('/proc/self')==str(os.getpid())
 print('cmdline and environ use NUL separators; environ retains the initial value here; self resolves for the reader')
 entry=root/'fd'/str(metadata['fd']);target=os.readlink(entry);actual=entry.stat()
 assert target==str(path)+' (deleted)' and not path.exists()
 assert (actual.st_dev,actual.st_ino)==(metadata['device'],metadata['inode'])
 assert entry.read_bytes()==b'fixture-data'
 print('a deleted pathname remains inspectable through an open descriptor subject to access checks')
 output,error=process.communicate('done\n',timeout=5)
 assert process.returncode==0,error
 assert not root.exists()
 print('after the controlled child exits and is reaped, its proc directory disappears')
 finally:
 if process.poll() is None:process.kill();process.wait()

The child sets its short process name to worker ) sample, opens a temporary file, and unlinks its pathname while keeping the descriptor open. It then reports only fixture metadata and waits. This gives the controller a known live process whose name deliberately exercises the stat-format edge case.

The controller finds that status has reported size zero but nonempty readable contents. It checks labeled PID and parent PID fields. For stat, it separates the PID and parenthesized name before interpreting the remaining positional fields, confirming the parent PID and a positive start-time field. Naive whitespace splitting would mistake part of the name for a later field.

The final command-line argument contains spaces and remains one argument after NUL-based parsing. The initial environment contains XB_PROC_TEST=fixture-value, even though the child has changed its application environment value after startup. Both observations use harmless fixture data. They demonstrate why whitespace parsing and assuming a live environment map can produce incorrect conclusions.

Reading /proc/self in the controller resolves to the controller’s PID. Reading the child’s descriptor link returns the removed pathname with the deleted suffix, but following that proc descriptor entry still exposes the expected device/inode pair and fixture bytes under the experiment’s permissions. The missing ordinary pathname and the surviving open object are distinct facts.

After receiving the controller’s message, the child closes its handle and exits. The controller waits for completion and confirms that this child’s proc directory disappears. The example establishes a controlled before-and-after lifecycle observation; it does not promise atomic consistency among arbitrary proc reads while a busy application changes its state.

Match the view to the question

Use the relevant entry’s documented parser and units. A stat field number should be interpreted only after handling the command-name field correctly; a status label should not be assigned an assumed line number. If correlating samples, include process start time and boot context as appropriate so a reused numeric PID is not silently treated as the same instance.

Access and namespace context matter. Proc mount options, credentials, and ptrace-related checks can restrict visibility. /proc/net reflects the relevant network namespace, and a container’s visible proc data is not automatically its configured resource budget. Confirm which namespace and resource interface answer the operational question before labeling an observation as host-wide or container-specific.

Some proc entries are writable controls, but their scope differs. /proc/sys exposes tunables, while /proc/PID/oom_score_adj adjusts a particular process’s OOM preference; writing integers to status is not the interface for that task. Keep observation separate from a deliberate configuration change, and inspect the actual target and documented permissions before applying any write.

Sources

proc(5), procfs structure, per-process visibility, and mount options, checked 2026-10-15.
proc_pid_stat(5), field layout, parenthesized command name, and start time, checked 2026-10-15.
proc_pid_environ(5), initial environment region and NUL-separated entries, checked 2026-10-15.
proc_pid_fd(5), descriptor links, referenced objects, and permission checks, checked 2026-10-15.
proc_pid_cmdline(5), argument separators and process-controlled command-line contents, checked 2026-10-15.

Trace sampling reduces volume without fixing a cost ceiling

Sat, 23 Oct 2027 00:00:00 +0000

Trace sampling selects which recorded work reaches the tracing pipeline. It can reduce export traffic and storage, but a configured percentage is not a strict ceiling on bytes, processor time, or money. Requests produce different numbers of spans, attributes vary in size, and traffic itself changes. A one-percent policy applied to a sudden tenfold traffic increase can still produce roughly ten times as much sampled data.

Head sampling makes a decision near the start of a trace. A trace-identifier ratio sampler is useful when identifiers have the randomness its algorithm requires. Its probability describes selection across suitable traffic; it does not promise exactly half of every sixteen requests, or equal coverage of every route. Small populations and rare failures can be poorly represented even when the overall percentage looks reasonable.

Counts reproduce the article’s deliberately arranged identifier fixture, not a production percentage guarantee. Remote-parent counts are a separate phase. The historical revised/passed status is retained without rerunning examples or adding runtime credit.

Keep a decision consistent across boundaries

Downstream instrumentation should follow the accepted parent decision when that is the intended policy. In the Python SDK, ParentBased delegates root decisions to its configured root sampler and uses separate policies for sampled and unsampled parents. Wrapping TraceIdRatioBased(0.5) therefore differs from treating every downstream span as an independent new coin flip. Context propagation and the receiving sampler configuration must agree. At an untrusted boundary, decide whether to accept the caller’s sampling flag: W3C treats it as guidance, not authority to consume unlimited tracing resources.

An unsampled trace still involves application work, context handling, and potentially instrumentation overhead. Dropping export does not make the business request free. Conversely, preserving a parent decision does not guarantee that every child will be visible: a library might be uninstrumented, a service might lose context, or an exporter might fail. Sampling consistency addresses one source of missing data, not the whole delivery path.

When it breaks

A request can look ordinary at its start and fail much later. A head sampler that already dropped it cannot recover the missing spans merely because application code subsequently sets an error status. Tail sampling can inspect more of the completed trace before deciding, but requires buffering, routing related spans together, and managing delayed or missing data. It also cannot reconstruct spans discarded upstream before they reached it.

Another failure is measuring request reliability directly from a biased trace collection. Policies that deliberately retain slow requests or errors change the population being observed. Keep request outcome metrics with a clear denominator, and describe the trace selection policy alongside investigations. For cost control, observe actual exported spans and bytes, queue behavior, collector resources, and retention; a sampling percentage alone is not evidence that a spending limit holds.

Check it quickly

In a Python virtual environment, install opentelemetry-sdk==1.35.0. Save this as trace-sampling.py and run it using that environment; the original article reports a run with Python 3.12.14 and that pinned SDK. That run’s exact execution artifact is not present in the retained audit, and the SDK is not installed in this review environment, so the full example remains unrun here. The diagram’s historical pass label repeats the original report; it is not independently recovered execution proof. The custom identifier generator is exclusively a deterministic test fixture. Its sixteen identifiers deliberately straddle this SDK version’s ratio threshold. Do not copy that sequential identifier scheme into production instrumentation, where the sampler relies on appropriate randomness.

import json
from collections import Counter

from opentelemetry.context import Context
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.id_generator import IdGenerator
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
from opentelemetry.trace import Status, StatusCode
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator


class TestIds(IdGenerator):
 # Controlled test inputs only; production IDs need suitable randomness.
 def __init__(self):
 self.trace_index = 0
 self.span_index = 0

 def generate_trace_id(self):
 index = self.trace_index
 self.trace_index += 1
 return ((index + 1) << 64) | (index * (1 << 60) + 1)

 def generate_span_id(self):
 self.span_index += 1
 return self.span_index


exporter = InMemorySpanExporter()
provider = TracerProvider(sampler=ParentBased(TraceIdRatioBased(0.5)),
 id_generator=TestIds())
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("sampling-lab")
executed_children = 0
late_error_recording = None
for index in range(16):
 with tracer.start_as_current_span(f"root-{index}", context=Context()) as root:
 for child in range(2):
 with tracer.start_as_current_span(f"child-{index}-{child}"):
 executed_children += 1
 if index == 15:
 late_error_recording = root.is_recording()
 root.set_status(Status(StatusCode.ERROR, "error discovered after start"))
assert provider.force_flush()
spans = exporter.get_finished_spans()
counts = Counter(span.context.trace_id for span in spans)
assert len(counts) == 8 and set(counts.values()) == {3} and len(spans) == 24
assert {s.name for s in spans if s.parent is None} == {f"root-{i}" for i in range(8)}
assert executed_children == 32 and late_error_recording is False
assert not any(span.name == "root-15" for span in spans)

exporter.clear()
propagator = TraceContextTextMapPropagator()
for flag, label in (("01", "sampled"), ("00", "unsampled")):
 # High trace-ID bits do not override an accepted remote parent's flag.
 header = f"00-1111111111111111ffffffffffffffff-2222222222222222-{flag}"
 context = propagator.extract({"traceparent": header}, context=Context())
 with tracer.start_as_current_span(label, context=context):
 with tracer.start_as_current_span(label + "-child"):
 pass
assert provider.force_flush()
remote_spans = exporter.get_finished_spans()
assert {span.name for span in remote_spans} == {"sampled", "sampled-child"}
assert next(span for span in remote_spans if span.name == "sampled").parent.is_remote
print(json.dumps({"controlled_roots": 16, "exported_traces": len(counts),
 "exported_spans": len(spans), "executed_child_operations": executed_children,
 "late_error_was_recording": late_error_recording,
 "remote_parent_exported_spans": len(remote_spans)}, indent=2))
provider.shutdown()

The sixteen root operations each start two children. Exactly eight traces, containing twenty-four spans, are exported because of the chosen test inputs; all thirty-two child operations still execute. Root fifteen is deliberately outside the retained set. Its later error status does not cause it to become recording or appear in the exporter. These assertions test behavior at known inputs rather than estimating a sampling percentage statistically.

The second phase extracts two remote parent headers with the same high trace identifier and different sampled flags. Only the sampled parent’s child tree is exported, producing two spans despite that identifier being above the root ratio threshold. The unsampled parent produces none. No collector or tail sampler runs in this experiment. When applying the policy to a real service, test accepted remote context separately from new roots, then validate the resulting volume and diagnostic coverage against the workload that actually matters.

Sources

OpenTelemetry sampling, head and tail decisions and the risk of losing rare failures, checked 2026-10-15.
OpenTelemetry Python sampling reference, ParentBased and TraceIdRatioBased sampler behavior, checked 2026-10-15.
W3C Trace Context, propagated sampling flags and remote parent context, checked 2026-10-15.

Prometheus pulls metrics and records scrape outcomes

Mon, 30 Aug 2027 00:00:00 +0000

Prometheus collects metrics by requesting an HTTP endpoint on each configured target. The target exposes observations; the collector chooses when to scrape, parses the response, and stores samples. This arrangement makes collection failures observable from the collector’s perspective. It does not guarantee that the endpoint is reachable, that every expected target was discovered, or that the application’s user journeys work.

Prometheus generates up for configured instances: 1 for scrape success, 0 for scrape failure. A removed target can later be absent instead. Collection success does not establish application health or the presence of every business metric. The source fixture was not rerun for this artwork.

A job contains independently scraped instances

A job groups targets with a shared purpose, such as several API replicas. Each target is an instance with its own scrape outcome. The usual job and instance labels identify that context, although relabeling can change the final labels. Service discovery supplies candidates; scrape configuration determines their endpoint, protocol, interval, timeout, and authentication. Discovery and collection therefore have separate failure modes.

The collector must be able to connect to the target. An exporter needs an inbound path from authorized collectors, even though it does not initiate metric uploads to them. Network policy, TLS identity, credentials, and routing remain deployment responsibilities. A pull design changes who initiates the connection; it does not remove network boundaries or make arbitrary exporters safe to expose publicly.

Read the generated metric precisely

For a configured instance, Prometheus generates up, with value one for a successful scrape and zero for a failed scrape. It is a metric, not an up label. Scrape duration and sample counts provide additional collection evidence. An exporter can return a valid response while the database behind it is unavailable, or while an important business metric is missing. Such a scrape can still succeed.

Use application outcomes and exporter-specific dependency metrics to answer those further questions. Conversely, up=0 describes a collection failure, which might be caused by the monitoring path while users remain unaffected. Keep that distinction in alert wording so responders investigate the correct surface instead of assuming every collection problem is a customer outage.

Metric relabeling is not applied to automatically generated series such as up. Filtering all business samples can therefore leave a successful scrape indicator behind. Include expected metric families or sample-volume checks in the acceptance criteria when changing filters, and distinguish intentionally discarded observations from exporter failure.

When it breaks

A connection refusal, timeout, certificate error, or invalid exposition can prevent a scrape from succeeding. Inspect the target’s last scrape error before changing the interval. A slower interval does not repair a wrong port, and a larger timeout can conceal a slow exporter without removing its cost. Review emitted series and response size when collection becomes expensive. Keep scrape timeouts within the configured interval, and evaluate exporter load before shortening the schedule. Faster polling can multiply database queries performed by an exporter without making those queries cheaper.

A target removed from discovery is different from a configured target returning zero. Its absence can eventually remove the corresponding series from instant queries. Therefore, a fleet-wide sum of up cannot prove that every expected instance still exists. Compare discovery inventory with the intended deployment when completeness matters. Likewise, a successful /metrics request is not a substitute for testing a checkout or login operation.

Check it quickly

Save this as pull.py and run python3 pull.py with Docker available. It uses prom/prometheus:v3.5.1; an uncached image requires an initial registry download. One disposable container scrapes its own loopback endpoint and an unused loopback port. No host port is published, and the container has no external network access.

import json
import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
# All HTTP traffic stays on loopback inside one disposable container.
script = r"""set -eu
cat > /tmp/prom.yml <<'CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
scrape_configs:
 - job_name: self
 static_configs:
 - targets: ['127.0.0.1:9090']
 - job_name: refused
 static_configs:
 - targets: ['127.0.0.1:1']
CONFIG
/bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data \
 --web.listen-address=127.0.0.1:9090 > /tmp/prom.log 2>&1 &
for attempt in $(seq 1 30); do
 if wget -qO /tmp/ready.json 'http://127.0.0.1:9090/api/v1/query?query=count%28up%29%20%3D%3D%202%20and%20sum%28up%29%20%3D%3D%201'; then
 if grep -q '"status":"success"' /tmp/ready.json && ! grep -q '"result":\[\]' /tmp/ready.json; then
 wget -qO- 'http://127.0.0.1:9090/api/v1/query?query=up'
 exit 0
 fi
 fi
 sleep 1
done
cat /tmp/prom.log >&2
exit 1
"""
name = "xbcatm-prom-live-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if result.returncode:
 raise RuntimeError(result.stderr or result.stdout)
payload = json.loads(result.stdout)
assert payload["status"] == "success"
values = {item["metric"]["job"]: float(item["value"][1])
 for item in payload["data"]["result"]}
assert values == {"self": 1.0, "refused": 0.0}, values
print(json.dumps({"up_by_job": values, "scrape_reachability_only": True}, indent=2))

The result must contain self: 1.0 and refused: 0.0. Assertions check actual query results after both targets have been evaluated; the startup loop avoids interpreting an initially empty database as a failure. The final cleanup removes only the uniquely named test container. This verifies collection outcomes, not business availability or discovery completeness.

Turn the outcome into a useful investigation

In a real incident, identify the failing instance, read its scrape error, and test connectivity from the collector’s network context. Compare the response with the expected exporter format and check its dependency status. Keep secrets out of copied diagnostic output. Once collection recovers, confirm both up and the specific business series used by dashboards. Restoring one does not automatically establish the other.

The operational fixture is unchanged here and was not rerun in this review. Its expected outputs and earlier observations remain distinct from current execution evidence.

Sources

Prometheus: jobs and instances, generated scrape metrics and target identity, checked 2026-10-15.
Prometheus: configuration, scrape intervals, timeouts and target configuration, checked 2026-10-15.
Prometheus: HTTP API, instant query response format, checked 2026-10-15.

Dashboards summarize; they do not magnify

Sun, 18 Jul 2027 00:00:00 +0000

A useful dashboard helps a reader decide what to inspect next. It reduces a large collection of measurements into a small set of questions: are users succeeding, where is the impact, and is capacity becoming constrained? Reduction must preserve important differences. A green global percentage can conceal a failing region, and a blank panel can mean missing telemetry rather than a quiet, healthy service.

Give the page a reading order

Start with the service outcome and observation coverage, then show traffic, latency, errors, and relevant saturation. Provide drilldowns for regions, operations, or dependencies when those distinctions change the next action. Put deployment annotations and links to diagnostic material near the relevant views. Correlated lines can suggest where to look, but their timing alone does not prove that a deployment or dependency caused the incident.

Each panel should answer a clear question, not necessarily contain exactly one metric. Comparing queue depth with its configured limit can be useful when both share a unit. Mixing unrelated units on a complicated dual-axis chart makes apparent relationships easy to invent. State the unit, aggregation window, population, and threshold meaning. A reader should not need to remember whether an axis represents milliseconds, seconds, fractions, or percentages.

Preserve weighting and coverage

Compute a service error fraction from the total bad events divided by total eligible events over matching windows. An unweighted mean of regional percentages gives a quiet region the same influence as a busy one. That is a different statistic, not a harmless display choice. Even a correctly weighted result should have a useful breakdown when small populations matter to the service commitment.

Freshness is another dimension of scope. Display the last successful observation or an explicit coverage signal, and define when stale data becomes unknown. A service-specific freshness threshold is separate from the storage system’s staleness rules. Filling gaps with zero can turn an unavailable exporter into an apparently perfect error rate or an empty queue. Missing data should remain visible, with a next step to investigate collection or the service itself.

When it breaks

Automatically chosen axis ranges can make a tiny variation look like a major incident, while an overly broad fixed range can conceal a meaningful threshold crossing. Select scales according to the operational question and keep comparable panels consistent. An average latency also hides a slow minority; use appropriate distributions and counts without averaging precomputed percentiles across instances.

An attractive dashboard can still become a collection of unrelated snapshots. Different time zones, filters, refresh intervals, and aggregation windows prevent a coherent comparison. Shared variables should have explicit defaults and visible scope, especially for production environments. Test the page with a regional failure, sparse traffic, and absent measurements. If those situations all appear green, the page is compressing away information needed for decisions.

The weighted 1% overview, eastern 50% error fraction and unknown southern coverage must remain visible together. These are synthetic aggregation values, not a validated dashboard screen.

Check it quickly

Save this as dashboard.py and run Python 3. It computes panel-ready values from synthetic counts for one completed sixty-second interval. Two regions have fresh observations; a third has missing values and an older observation time. The ninety-second freshness threshold is an example dashboard policy, not a claim about Prometheus defaults. No Grafana installation is required for this aggregation exercise.

import json
from fractions import Fraction

# Counts cover the same completed 60-second interval.
rows = [
 {"region": "west", "requests": 9800, "errors": 0, "age_seconds": 10},
 {"region": "east", "requests": 200, "errors": 100, "age_seconds": 10},
 {"region": "south", "requests": None, "errors": None, "age_seconds": 240},
]
fresh = [r for r in rows if r["age_seconds"] <= 90
 and r["requests"] is not None and r["errors"] is not None]
missing = [r["region"] for r in rows if r not in fresh]
requests = sum(r["requests"] for r in fresh)
errors = sum(r["errors"] for r in fresh)
regional = {r["region"]: Fraction(r["errors"], r["requests"]) for r in fresh}
known_ratio = Fraction(errors, requests)
unweighted = sum(regional.values()) / len(regional)

assert known_ratio == Fraction(1, 100)
assert regional["east"] == Fraction(1, 2) and unweighted == Fraction(1, 4)
assert missing == ["south"]
print(json.dumps({
 "coverage": "incomplete",
 "fresh_requests_per_second": round(requests / 60, 3),
 "fresh_error_percent": float(known_ratio * 100),
 "regional_error_percent": {k: float(v * 100) for k, v in regional.items()},
 "wrong_unweighted_percent": float(unweighted * 100),
 "unknown_regions": missing,
}, indent=2))

Fresh observations contain 10,000 requests and 100 errors, giving 166.667 requests per second and a weighted 1% error fraction. The small eastern region is nevertheless failing half its requests. Averaging the two regional percentages would incorrectly label the combined request fraction as 25%. Show the 1% overview with its regional breakdown instead of choosing one statistic to hide the other.

The output also declares coverage incomplete and lists the southern region as unknown. Neither its requests nor its errors are invented as zero. This fixture checks the arithmetic and missing-data meaning behind the panels; it does not validate an actual dashboard query or screen. Carry the same definitions into the rendered dashboard and review it against known incident and telemetry-loss cases.

Sources

Grafana: Dashboard Best Practices, goals, focused questions, and a coherent reading order, checked 2026-09-13.
Google SRE: Monitoring Distributed Systems, service-oriented dashboards and signals, checked 2026-09-13.
Prometheus: Metric and Label Naming, units and meaningful aggregation, checked 2026-09-13.

Structured logs make their query contract explicit

Tue, 09 Mar 2027 00:00:00 +0000

Structured logging assigns stable meaning to fields instead of asking each query to infer everything from message text. A record might contain an event name, request identifier, numeric duration, severity, and a human-readable message. Free text remains useful and can also be searched or parsed. The practical gain is that routine questions can use a documented schema whose meaning survives changes to explanatory wording.

That schema needs more than curly braces. A duration stored as a number supports numeric comparisons, while a duration stored as a string may be indexed or compared differently. Milliseconds and seconds must not share an ambiguous field name. An event name should describe what happened consistently across versions. A schema version can identify deliberate format changes, but it does not automatically transform old records or keep every consumer compatible.

Article fixture: two start events and two completion events, two records per request. Synchronous task-local capture and line-safe JSON are the illustrated boundary. Scheduling order is schematic; no new runtime, queue, collector, indexing or redaction evidence is claimed.

Put context on the correct record

A request identifier is useful only if it belongs to the operation that emitted the record. Concurrent requests make a process-global mutable variable unsuitable for that purpose. Python context variables can hold task-local state across asynchronous work, provided values are set and reset at the intended scope. This does not mean every thread or custom execution mechanism inherits context in the same way; verify the actual boundary.

Capture context while it is available. This example formats each record synchronously in the emitting task, so its formatter can read the current context variable. A queue-based logging pipeline needs a different placement: attach the required fields before handing the record to another thread or process. Reading the listener’s current request context would associate the record with the listener, rather than reliably recovering the original request.

When it breaks

Hand-concatenating JSON can turn quotes or line breaks from a message into invalid records. A line-oriented collector may then split one application event into several apparent events. Use a JSON serializer and one complete JSON object per physical line. The message can retain a logical newline inside its string value while the serialized line contains an escaped representation. Rendering that message later should still keep its record boundary clear.

Structured output does not automatically create indexes, reduce query bills, or guarantee delivery. Unbounded fields, accidental type changes, missing identifiers, and sensitive values can make a JSON stream difficult or inappropriate to use. Select fields according to real queries, keep the schema stable, and verify the collector’s parser and retention rules. Searchable structure is a contract between producers and consumers, not a property established by the file extension.

Check it quickly

Save this as structured-logging.py and run it with Python; the complete unchanged program passed on 2026-10-19 with Python 3.12.14. It uses only standard-library modules. Two asynchronous requests each log a start event, wait at a shared gate, then log a completion event containing a deliberately troublesome message. The durations are fixed synthetic values for checking JSON types, not measurements of request performance. Output is captured in memory for assertions.

import asyncio
import io
import json
import logging
from collections import Counter
from contextvars import ContextVar

request_id = ContextVar("request_id", default=None)
stream = io.StringIO()


class JsonFormatter(logging.Formatter):
 def format(self, record):
 return json.dumps({"schema_version": 1, "event": record.event,
 "request_id": request_id.get(), "worker": record.worker,
 "duration_ms": record.duration_ms,
 "level": record.levelname, "message": record.getMessage()},
 ensure_ascii=False, separators=(",", ":"))


logger = logging.getLogger("structured-log-lab")
logger.setLevel(logging.INFO)
logger.propagate = False
handler = logging.StreamHandler(stream)
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)


async def request(name, ready, release):
 token = request_id.set(name)
 try:
 logger.info("starting %s", name, extra={"event": "request.started",
 "worker": name, "duration_ms": 0})
 ready.set()
 await release.wait()
 await asyncio.sleep(0)
 logger.info('untrusted line\n{"event":"forged"}',
 extra={"event": "request.completed", "worker": name, "duration_ms": 4})
 finally:
 request_id.reset(token)


async def main():
 ready = [asyncio.Event(), asyncio.Event()]
 release = asyncio.Event()
 tasks = [asyncio.create_task(request(name, signal, release))
 for name, signal in zip(("req-a", "req-b"), ready)]
 await asyncio.gather(*(signal.wait() for signal in ready))
 release.set()
 await asyncio.gather(*tasks)
 assert request_id.get() is None


try:
 asyncio.run(main())
finally:
 logger.removeHandler(handler)
 handler.close()
lines = stream.getvalue().splitlines()
records = [json.loads(line) for line in lines]
assert len(lines) == 4
assert Counter(record["request_id"] for record in records) == {"req-a": 2, "req-b": 2}
assert all(record["request_id"] == record["worker"] for record in records)
assert all(type(record["duration_ms"]) is int for record in records)
completed = [record for record in records if record["event"] == "request.completed"]
assert len(completed) == 2 and all("\n" in record["message"] for record in completed)
assert not any(record["event"] == "forged" for record in records)
print(json.dumps({"physical_lines": len(lines), "parsed_records": len(records),
 "requests": 2, "context_mixing": False,
 "numeric_duration_preserved": True, "forged_records": 0}, indent=2))

The assertions require four physical lines and four parsed records, with two records for each request identifier. Each record’s identifier must match the worker that emitted it, even though the requests overlap. Duration values remain JSON numbers. The embedded newline survives inside each decoded completion message, while no additional record named forged appears. Each task resets its token in a finally block. The outer-context assertion checks isolation from the child tasks; by itself it does not independently prove restoration inside each child.

This verifies task isolation, synchronous formatting, and line-safe serialization. It does not exercise a background queue, collector retry, storage indexing, or secret redaction. When integrating it, check the complete path with representative fields and the actual concurrency model. Inspect what the backend stores after parsing, and test the queries that motivated those fields. A successful producer-side serialization check is one useful boundary test, while a dependable logging system also needs evidence that the intended records reach their consumers with their meaning intact.

Sources

Python logging cookbook, contextual records and logging across concurrent work, checked 2026-10-15.
Python contextvars, task-local context and token reset behavior, checked 2026-10-15.
Python json, serialization, escaped characters, and JSON value types, checked 2026-10-15.
JSON Lines, one complete JSON value per physical line, checked 2026-10-15.

An access log separates client outcomes from upstream attempts

Mon, 01 Mar 2027 00:00:00 +0000

An nginx access log records request information when logging runs at the end of processing. The selected fields can distinguish what the client requested, which URI ultimately handled it, what response was sent, and what happened during upstream attempts. Keep these observations separate: one status, duration, or path cannot explain the whole request.

Historical nginx 1.28.0 observations, not a fresh execution. Attempt outcomes, addresses and timings retain their correspondence. Missing timing is not zero; response byte counters are not complete network-overhead accounting. Collection and retention need separate evidence.

When it breaks

The final client status can conceal an earlier upstream failure. If nginx retries another server successfully, the client may receive 200 while an attempt list records a failed connection first. Conversely, a locally generated response has no upstream operation to time. Treating every request as exactly one backend response loses the information needed to explain retries and local handling.

Original and current URI fields also differ after internal redirects. $request retains the request line, and $request_uri retains the original URI and query. $uri can reflect a later normalized or rewritten path. An investigator who logs only the final URI may not know which public address triggered it, while logging only the original path can hide the final handler’s context.

Timing fields are not interchangeable latency components that can always be subtracted into a precise diagnosis. Request time includes processing through the client-facing completion point used by nginx logging. Upstream timing fields describe their own phases and can contain several attempt values. Their resolution, missing values, and boundaries matter; a zero-looking millisecond value does not establish that no work occurred.

Check it quickly

The integration results below are historical observations; this revision did not rerun the fixture.

Use host Python and a disposable local Docker daemon with build/run permission. The historical run reported nginx 1.28.0, backend Python 3.11.16 and Docker 29.3.1 on Linux arm64. Use platform-compatible pinned images; build --network none does not prevent registry resolution. The script configures a backend and unreachable container port, validates nginx, and publishes a loopback proxy port. Cleanup is attempted, not verified; check its result. Never target a remote or production daemon:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True);target.write_text(content)
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'

backend=r'''from http.server import BaseHTTPRequestHandler,HTTPServer
import json
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=self.rfile.read(int(self.headers.get('Content-Length','0')))
 document={'path':self.path,'method':self.command,'body':body.decode(),'headers':{key:self.headers.get(key) for key in ['Host','X-Forwarded-For','X-Real-IP','X-Forwarded-Proto','X-Forwarded-Host']}}
 data=json.dumps(document).encode()
 self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.send_header('X-Backend-Info','fixture-node');self.end_headers();self.wfile.write(data)
 do_POST=do_GET
 def log_message(self,*args):pass
HTTPServer(('0.0.0.0',8081),Handler).serve_forever()
'''

fields={'request':'$request','request_uri':'$request_uri','uri':'$uri','status':'$status','upstream_status':'$upstream_status','upstream_addr':'$upstream_addr','request_time':'$request_time','upstream_connect_time':'$upstream_connect_time','upstream_header_time':'$upstream_header_time','upstream_response_time':'$upstream_response_time','body_bytes_sent':'$body_bytes_sent','bytes_sent':'$bytes_sent'}
extra="log_format details escape=json '"+json.dumps(fields,separators=(',',':'))+"';\nupstream service_pool { server 127.0.0.1:1 max_fails=1; server backend:8081 backup; }"
locations=r'''
access_log /tmp/access.json details;
location = /local { return 204; }
location = /rewrite { rewrite ^ /normalized last; }
location = /normalized { return 200 "normalized"; }
location /proxy {
 proxy_next_upstream error timeout;
 proxy_next_upstream_tries 2;
 proxy_pass http://service_pool;
}
'''
with lab(config(locations,extra),backend=backend) as (container,request):
 assert request('/local')[0]==204
 assert request('/rewrite?x=1')[2]==b'normalized'
 status,headers,body=request('/proxy?x=1');assert status==200
 def lines():return docker('exec',container,'cat','/tmp/access.json').stdout.splitlines()
 wait(lambda:len(lines())>=3)
 records={record['request_uri']:record for record in map(json.loads,lines())}
 print(json.dumps(records,indent=2),flush=True)
 local=records['/local'];assert local['status']=='204' and local['upstream_status']=='' and local['upstream_response_time']==''
 assert local['body_bytes_sent']=='0' and int(local['bytes_sent'])>0
 changed=records['/rewrite?x=1'];assert changed['request']=='GET /rewrite?x=1 HTTP/1.1' and changed['uri']=='/normalized'
 proxy=records['/proxy?x=1'];assert proxy['status']=='200' and proxy['upstream_status']=='502, 200'
 assert len(proxy['upstream_addr'].split(', '))==2 and int(proxy['body_bytes_sent'])==len(body)
 assert float(proxy['request_time'])>=0 and len(proxy['upstream_response_time'].split(', '))==2
 print('a local 204 has no upstream and no body bytes, but still sends response-header bytes')
 print('the final uri reflects the internal redirect while request and request_uri retain the original request')
 print('one client-visible 200 records upstream attempts 502, 200 and separate attempt-address/timing lists')

The /local request returns 204 without contacting an upstream. Its log has zero body bytes but a positive bytes_sent value because response headers were sent. In this JSON-escaped format, absent upstream fields are empty strings. This is different from a measured zero-second upstream operation and from the dash used by the tested version’s default escaping path.

The request /rewrite?x=1 internally becomes /normalized. The log retains GET /rewrite?x=1 HTTP/1.1 as the request line and the original URI with its query, while uri records /normalized. This verifies the difference at the actual log-write point, rather than inferring it only from the rewrite declaration or the response body.

The proxy request first tries loopback port one, where the disposable proxy container has no listener. nginx records that failed attempt with status 502 and then reaches the configured backup server. The final client response is 200, while upstream_status is the string 502, 200. The first value is nginx’s recorded failure outcome, not evidence that a backend actually returned an HTTP 502 response.

The same event contains two upstream addresses and two response-time entries. Connect and header timings can contain a dash for an attempt that never reached those phases, even though the overall variable exists. That differs from an entirely absent upstream variable. Preserve these relationships when parsing instead of replacing every unusual value with an indistinguishable numeric zero.

The logged body byte count equals the actual response-body length read by the client in this uncompressed fixture. bytes_sent includes more HTTP response bytes, but neither field is presented as a complete accounting of Ethernet, TCP, or TLS overhead. The example uses the counters to distinguish body content from header traffic, not to claim a physical-network billing measurement.

The script waits until the three events are present before parsing them. Logging occurs after request handling, and configured buffering can delay visibility further. A successful HTTP response does not guarantee that a remote log collector has already received or retained the corresponding event.

Keep diagnostic meaning in the schema

Choose fields that answer concrete operational questions: original target, final route, client status, upstream attempt outcomes, and relevant durations. Keep attempt lists intact until a parser with the documented grammar separates them. Multiple upstream groups or internal transitions can introduce additional structure beyond the simple two-attempt case shown here.

Avoid treating client-controlled forwarding headers as verified caller identity. Also consider whether query strings or other request fields contain sensitive values before retaining them. Richer logs are useful only when the information has a clear purpose and appropriate access and retention controls.

Test format changes by producing representative local responses, retries, rewrites, and failures, then parsing actual records. Validate rotation and collection separately. A trustworthy access log connects each field to its measured event and preserves missing or multi-attempt information instead of flattening it into a misleading summary.

Sources

nginx HTTP log module, request-completion timing, byte counters, and log-format fields, checked 2026-10-15.
nginx upstream variables, per-attempt addresses, statuses, and timing lists, checked 2026-10-15.
nginx core request variables, original request text and current normalized URI, checked 2026-10-15.
nginx 1.28.0 log implementation, JSON-escaped missing-variable representation, checked 2026-10-15.

Traces connect the recorded parts of a request

Tue, 15 Dec 2026 00:00:00 +0000

A distributed trace connects recorded operations through a shared trace identifier and explicit relationships. A span represents one operation, with start and end timestamps, attributes, events, and status. This makes a trace useful when a checkout request crosses an application, an inventory service, and a payment service. The resulting picture depends on which operations were instrumented, retained, and exported; it is not an automatic inventory of every action the request performed.

The trace identifier groups related spans. A span identifier distinguishes one operation within that trace, while the parent identifier connects a child to its immediate predecessor. Across HTTP, W3C traceparent carries the trace identifier, the sending span identifier, and trace flags. The receiving instrumentation extracts that context and creates a new span. Copying a request identifier into unrelated log messages does not create these span relationships. For work with several causal inputs, span links can express additional relationships without assigning multiple parents to one span.

Trace A and B are editorial names for the two source-described trace groups, not actual identifiers. The source example is represented as inert code and narrative: historical review remains read and execution blocked, with no rerun or new runtime credit. Arrow length is not duration.

Read relationships before adding durations

A server span can contain database spans and several overlapping outbound requests. Adding all those durations counts overlapping time repeatedly. Even subtracting the sum of child durations from the parent can produce nonsense when siblings run concurrently. Read the timeline and identify intervals on the critical path. Cross-machine clock differences and missing instrumentation also limit how precisely apparent gaps can be interpreted.

Span status needs its own interpretation. An HTTP response code, a recorded exception event, and an OpenTelemetry error status are separate pieces of data. Instrumentation may connect them according to its conventions; manually recording an exception does not justify assuming that every relevant status field changed. Service resource attributes identify the emitter, whereas an individual span describes an operation performed there.

When it breaks

A proxy can drop propagation headers, or a service can fail to extract them. The downstream work then appears as a separate trace even though the business request reached it successfully. A shared trace identifier with the wrong parent is another failure: related work is grouped together, but its causal position is wrong. Test both identifiers at a boundary instead of checking only that some spans arrived.

An absent span has several possible causes, including sampling, missing instrumentation, asynchronous context loss, and export failure. It does not prove that the operation never ran. Likewise, a green trace view does not establish that every downstream dependency succeeded. Correlate it with request outcome metrics and relevant logs, and check collector or exporter health before interpreting an incomplete picture as complete evidence.

Check it quickly

Create a Python virtual environment and install opentelemetry-sdk==1.35.0, which installs the matching API dependency. Save this as trace-flow.py and run it with that environment’s Python. The original article reports a Python 3.14.4 run. This revision retains the exercise without rerunning it; the current execution check remains blocked. The script starts a loopback HTTP server, uses separate caller and worker tracer providers, and exports to memory. It closes the server after the requests; no collector is required. Before an authorized run, ensure proxy settings bypass 127.0.0.1: urllib can otherwise consult a configured proxy. Creating this local listener and making HTTP requests require a suitable disposable environment.

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread
from urllib.error import HTTPError
from urllib.request import Request, urlopen

from opentelemetry.context import Context
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import SpanKind, Status, StatusCode
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

exporter = InMemorySpanExporter()
providers = [TracerProvider(resource=Resource({"service.name": name}))
 for name in ("caller", "worker")]
for provider in providers:
 provider.add_span_processor(SimpleSpanProcessor(exporter))
caller, worker = [p.get_tracer("trace-lab") for p in providers]
propagator = TraceContextTextMapPropagator()


class Handler(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def do_GET(self):
 carrier = {key.lower(): value for key, value in self.headers.items()}
 parent = propagator.extract(carrier, context=Context())
 with worker.start_as_current_span("handle" + self.path,
 context=parent, kind=SpanKind.SERVER) as span:
 status = 200
 if self.path == "/fail":
 try:
 raise ValueError("synthetic inventory failure")
 except ValueError as error:
 span.record_exception(error)
 span.set_status(Status(StatusCode.ERROR, str(error)))
 status = 503
 body = b"lab response\n"
 self.send_response(status)
 self.send_header("Content-Length", str(len(body)))
 self.end_headers()
 self.wfile.write(body)


server = HTTPServer(("127.0.0.1", 0), Handler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
base = f"http://127.0.0.1:{server.server_port}"
try:
 with caller.start_as_current_span("request", context=Context()):
 with caller.start_as_current_span("outbound", kind=SpanKind.CLIENT):
 headers = {}
 propagator.inject(headers)
 with urlopen(Request(base + "/ok", headers=headers), timeout=5) as response:
 assert response.status == 200
 response.read()
 try:
 with urlopen(base + "/fail", timeout=5) as response:
 raise AssertionError("Expected HTTP 503")
 except HTTPError as response:
 with response:
 assert response.code == 503
 response.read()
finally:
 server.shutdown()
 thread.join(timeout=5)
 server.server_close()

for provider in providers:
 assert provider.force_flush()
spans = {span.name: span for span in exporter.get_finished_spans()}
assert len(spans) == 4
root, client, received, orphan = [spans[name]
 for name in ("request", "outbound", "handle/ok", "handle/fail")]
assert root.parent is None
assert root.context.trace_id == client.context.trace_id == received.context.trace_id
assert client.parent.span_id == root.context.span_id
assert received.parent.span_id == client.context.span_id and received.parent.is_remote
assert orphan.parent is None and orphan.context.trace_id != root.context.trace_id
assert orphan.status.status_code == StatusCode.ERROR
assert [event.name for event in orphan.events] == ["exception"]
assert received.resource.attributes["service.name"] == "worker"
assert client.resource.attributes["service.name"] == "caller"
print(json.dumps({"exported_spans": len(spans), "linked_spans": 3,
 "unpropagated_request_started_new_trace": True,
 "failed_span_status": orphan.status.status_code.name,
 "exception_events": len(orphan.events)}, indent=2))
for provider in providers:
 provider.shutdown()

The first request creates three linked spans: request, outbound, and handle/ok. The assertions verify one trace identifier, two correct parent links, a remote parent on the server span, and different service names on the two providers. The second request deliberately sends no propagation header. It returns HTTP 503, creates a separate root span, and explicitly records both an exception event and an ERROR status. Four spans are exported altogether.

These are two providers in one Python process communicating over real local HTTP. A successful full run would verify propagation and recording behavior, not deployment-wide clock synchronization, collector reliability, or concurrent critical-path analysis; those expected assertions have not been rerun here. In a deployed service, repeat the boundary checks through the actual ingress and client libraries, then confirm export health and sampling configuration. A useful trace is one whose recorded relationships and limitations are understood before its timeline is used to explain an incident.

Sources

OpenTelemetry traces, span structure, relationships, and trace identity, checked 2026-09-13.
W3C Trace Context, traceparent fields and propagation across process boundaries, checked 2026-09-13.
OpenTelemetry Python instrumentation, manual spans, processors, and explicit exception recording, checked 2026-09-13.

Put the unit in the metric name

Fri, 30 Oct 2026 00:00:00 +0000

A metric named request_duration leaves readers guessing whether 250 means seconds, milliseconds, or something else. A Prometheus-facing name such as http_request_duration_seconds carries that contract into queries, recording rules, and alerts. The contract still needs a correct implementation. Appending _seconds to measurements collected in milliseconds changes the spelling, not the magnitude, and can create a thousandfold monitoring error.

This follows the article’s three Decimal inputs and threshold. The historical promtool parser and naming-lint result remains unchanged. Correct spelling does not prove correct instrumentation; check a known input through collection, export, query and display.

Convert at a defined boundary

Choose one quantity and one unit for each metric family. Prometheus recommends base units such as seconds and bytes, with a unit suffix where applicable. A duration and an absolute timestamp can both use seconds, but they are different quantities: last_success_timestamp_seconds should not be combined with request_duration_seconds. Describe the measurement and boundary in its help text so that a useful name has an equally clear definition.

If a library reports milliseconds, convert once before observation or export. Preserve sufficient precision instead of truncating a short duration to zero. Configure the dashboard to display the base unit appropriately; a formatter may show 0.250 seconds as 250 milliseconds without changing stored samples. Do not add another conversion in a recording rule because the panel happens to use milliseconds on its axis.

Unit and type answer different questions

An accumulating byte count conventionally ends in _bytes_total; a current memory usage gauge ends in _bytes. Both measure bytes, but only the former represents accumulated work and is suitable for a counter rate. Counts of requests conventionally use _total without inventing a physical unit. Histogram families add their own component suffixes, such as _seconds_bucket, _seconds_sum, and _seconds_count, to the duration family name.

A ratio commonly represents a fraction between zero and one; percentages use a different numeric scale. State which one a panel expects. A counter may reset when its process restarts, and Prometheus counter-rate functions account for decreases within the observed series. An additional _created series is not required for that behavior. Apply rate calculations before aggregating independent counters so that their individual resets remain visible.

When it breaks

A correctly named metric can still contain incorrect values. Mixing seconds and milliseconds under the same identity contaminates history, so changing only a label or help string cannot repair old samples. Treat a unit migration as a contract change: update the producer, queries, alerts, and dashboards together, and keep historical comparisons explicit. A new identity may be appropriate when the old series has an incompatible meaning.

The suffix also cannot enforce counter behavior. A gauge named _total remains a bad input to a counter-rate interpretation if it moves down during ordinary operation. Conversely, dividing an already calculated per-second rate by the interval again invents a different quantity. These are dimensional and semantic errors, not display problems. Other telemetry conventions may carry units in metadata rather than names; follow the documented exporter mapping instead of renaming every upstream instrument blindly.

Check it quickly

Save the following as units.py and run Python 3. It converts three synthetic latency measurements and emits a small Prometheus text exposition with help and type declarations. Decimal arithmetic makes the millisecond-to-second conversion explicit. The comparison demonstrates why a renamed but unconverted producer cannot share the same latency threshold.

from decimal import Decimal

latency_ms = [Decimal('120'), Decimal('250'), Decimal('630')]
latency_seconds = [value / Decimal('1000') for value in latency_ms]
threshold_seconds = Decimal('0.300')
correct_good = sum(value <= threshold_seconds for value in latency_seconds)
rename_only_good = sum(value <= threshold_seconds for value in latency_ms)
transferred_bytes = 1024 * 1024
used_ratio = Decimal('25') / Decimal('100')

assert correct_good == 2 and rename_only_good == 0
assert sum(latency_seconds) == Decimal('1.000')
assert used_ratio == Decimal('0.25')
print(f"# within 0.300s: converted={correct_good}/3, rename_only={rename_only_good}/3")
print('# HELP demo_last_request_duration_seconds Duration of the most recent completed request.')
print('# TYPE demo_last_request_duration_seconds gauge')
print(f'demo_last_request_duration_seconds {latency_seconds[-1]:.3f}')
print('# HELP demo_transferred_bytes_total Bytes transferred since this process started.')
print('# TYPE demo_transferred_bytes_total counter')
print(f'demo_transferred_bytes_total {transferred_bytes}')
print('# HELP demo_capacity_used_ratio Fraction of configured capacity currently used.')
print('# TYPE demo_capacity_used_ratio gauge')
print(f'demo_capacity_used_ratio {used_ratio}')

The first comment reports two of three observations within 0.300 seconds after conversion, compared with zero of three when millisecond values are merely treated as seconds. The exported last-request gauge is 0.630 seconds, the byte counter is 1,048,576, and the capacity ratio is 0.25, corresponding to 25% in a percentage display.

The recorded earlier review passed the complete Python output through promtool check metrics using promtool 3.14.0; that historical parser result is not a new engine run. That parser check establishes valid exposition and naming lint for this sample; it cannot infer whether real instrumentation measured the promised quantity. Verify a known input through collection, export, query, and display before trusting a renamed production metric.

Sources

Prometheus: Metric and Label Naming, base units, suffixes, and conventions at the Prometheus boundary, checked 2026-09-13.
Prometheus: Data Model, metric identity and labels, checked 2026-09-13.
Prometheus: Query Functions, counter rates and reset handling, checked 2026-09-13.

Readable evidence before dashboards

Tue, 18 Aug 2026 00:00:00 +0000

Monitoring dashboards are convenient, but convenience is not the same as evidence. This essay argues that for a small static service, the most auditable form of “is the site healthy?” is a set of plain-text facts an operator can read, re-run, and paste into a report — not an aggregated panel.

The distinction matters because the two answer different questions. A dashboard is a lens over many samples, tuned to alert you to trends. A probe is a single, reproducible observation of the public edge. When the job is to prove a release or diagnose a specific request, the probe is the higher-fidelity artifact.

A response supports a scoped claim. Repeated observations support trends when the underlying records and their provenance remain inspectable.

The dashboard’s aggregation is the risk

A dashboard turns thousands of raw observations into a series of aggregated panels: average latency, error rate, uptime percentage. Aggregation is where the audit trail thins. An “uptime 99.98%” figure does not tell you which request failed, which header was wrong, or whether today’s release is the one being served. It is a summary that erases the evidence that produced it unless you keep the raw log elsewhere.

For a confidence property like “the edge serves the tree I staged”, a dashboard is the wrong tool. A response-body comparison can answer that question for one URL, representation, network vantage point, and instant. A homepage header fetch cannot establish the identity of a whole tree or every CDN node.

A probe you can read

The repository ships a small read-only helper, static/downloads/check-http-headers.sh, also published at /downloads/check-http-headers.sh. It prints response headers and the final status/URL. The following is an illustrative excerpt, not a new production measurement:

HTTP/2 200
content-type: text/html; charset=utf-8
content-security-policy: default-src 'self'; …; script-src 'none'; …

A real capture can directly show the response status, declared media type, and delivered CSP. The abbreviated example above cannot establish a complete no-script policy: inspect the full enforcing header, including any script-src-elem or script-src-attr directives that take precedence for their categories. A Report-Only policy does not block execution. A panel that says “edge OK” should expose the same underlying capture and acceptance criteria. Both still depend on the probe implementation, TLS verification, routing, and retention of the original output. Reading certificate notAfter adds an expiry observation; it does not establish hostname matching or trust-chain validity.

Replace signal loss with a checkable statement

The mechanism the site favors instead of a dashboard is a contract enforced during build and a set of repeatable collection steps for the release window. Roughly:

Offline checks validate the artifact (every internal link, every SVG, the no-script boundary, the checksums).
Separate public-edge checks record status/headers, response bytes, and TLS identity/expiry. The header helper does not print certificate dates or automatically reject every unexpected HTTP status.
A byte manifest describes the staged tree; retrieving selected URLs and comparing their bodies establishes sampled delivery parity. A complete claim requires checking all published mappings and representation transformations, not merely hashing the server disk.

Each of these is a sentence an operator can paste into an issue or a changelog and a future auditor can re-run. None of them requires a panel to be understood.

Keep enough context to repeat the observation

Record UTC time, requested and final URL, method, selected encoding, resolved remote IP, client version, exit status, HTTP status, relevant headers, and body digest. Preserve failures as well as successes. Redact credentials, cookies, and personal data before sharing logs. A proxy error page returning 200 can pass a status-only check; comparing a known body or release marker catches a different failure mode.

A useful release report distinguishes “this response matched at this time” from “the service met its availability objective over a month.” Retain the raw observations behind aggregate metrics, and make a dashboard link back to them where possible. Neither display format creates trustworthy measurements by itself.

When you genuinely need a dashboard

None of this says observability is pointless. If you operate many instances, aggregate capacity, or long trends, a dashboard is the right view to see the forest. The claim is narrower: when the question is “is this one release healthy at the public edge”, a reproducible terminal read is a good starting point. Repeated probes, several vantage points, error budgets, and alerting add evidence about intermittent or regional failures that a single successful request cannot supply. A well-designed dashboard can expose those observations without hiding their provenance.

The decision heuristic is simple. If a claim can be checked with one command, preserve that command’s output together with its scope and acceptance criteria. Use dashboards where aggregation helps, while retaining access to the underlying evidence. That keeps the site’s own operational story as auditable as the site’s content.

Sources

Curl documentation, reading response headers from a single invocation, checked 2026-10-15.
Twelve-Factor: Logs, treating logs as event streams an operator can read, checked 2026-10-15.
OpenTelemetry: Observability primer, the complementary roles of logs, metrics, and traces, checked 2026-10-15.
W3C: Content Security Policy Level 3, enforcing versus Report-Only delivery and script directive precedence (working draft), checked 2026-10-18.

Observability · XB Field Notes

Baggage carries context across an explicit boundary

Choose what crosses and what gets recorded

When it breaks

Check it quickly

Sources

Annotations explain an alert without changing its identity

Separate identity from the current explanation

A group does not necessarily share its summary

When it breaks

Check it quickly

Sources

pg_stat_statements ranks query work within a collection window

Grouping is useful and bounded

Rank according to the question

When it breaks

Check it quickly

Sources

Read procfs as documented views of a changing process

When it breaks

Check it quickly

Match the view to the question

Sources

Trace sampling reduces volume without fixing a cost ceiling

Keep a decision consistent across boundaries

When it breaks

Check it quickly

Sources

Prometheus pulls metrics and records scrape outcomes

A job contains independently scraped instances

Read the generated metric precisely

When it breaks

Check it quickly

Turn the outcome into a useful investigation

Sources

Dashboards summarize; they do not magnify

Give the page a reading order

Preserve weighting and coverage

When it breaks

Check it quickly

Sources

Structured logs make their query contract explicit

Put context on the correct record

When it breaks

Check it quickly

Further reading

Sources

An access log separates client outcomes from upstream attempts

When it breaks

Check it quickly

Keep diagnostic meaning in the schema

Sources

Traces connect the recorded parts of a request

Read relationships before adding durations

When it breaks

Check it quickly

Sources

Put the unit in the metric name

Convert at a defined boundary

Unit and type answer different questions

When it breaks

Check it quickly

Sources

Readable evidence before dashboards

The dashboard’s aggregation is the risk

A probe you can read

Replace signal loss with a checkable statement

Keep enough context to repeat the observation

When you genuinely need a dashboard

Sources