A journal file moves from active to archived on rotation, then an archived file can be removed by vacuum.

The systemd journal needs an explicit retention contract

The systemd journal stores messages with structured metadata in journal files. That makes service output searchable without first inventing a parser for every application’s text format. Keeping those messages for a useful period is a separate requirement. Storage location, size policy, rotation, and forwarding determine which evidence remains available when an incident is investigated later. ...

October 26, 2027 · 5 min · XB Field Notes
Journal record bars pass through an explicit predicate into a smaller matched set.

Query the journal with explicit field and priority rules

A useful journal query selects the records that answer a specific incident question. Unit identity, invocation, time, priority, and application fields provide different dimensions. Combining them correctly matters more than finding a familiar word somewhere in the machine’s history. journald can attach process metadata, but it does not automatically collect every file an application writes or infer the business meaning of each message. ...

July 30, 2027 · 5 min · XB Field Notes
Four historical nginx log-format rows show three failing choices and one successful JSON-escaped format with quoted optional timing.

Structured nginx logs need escaping and a value-type contract

A log_format that resembles JSON is not necessarily a reliable structured event format. nginx substitutes variable values into the text, and those values can contain quotes, backslashes, control characters, or no data at all. Correct string escaping and correct value types are separate requirements. Testing emitted records with a real parser catches failures that a visually convincing configuration or successful nginx -t cannot establish. ...

July 15, 2027 · 6 min · XB Field Notes
Two request lanes retain req-a and req-b through synchronous formatting into four separate JSON records. Article fixture illustration, not a new run.

Structured logs make their query contract explicit

Structured logging assigns stable meaning to fields instead of asking each query to infer everything from message text. A record might contain an event name, request identifier, numeric duration, severity, and a human-readable message. Free text remains useful and can also be searched or parsed. The practical gain is that routine questions can use a documented schema whose meaning survives changes to explanatory wording. ...

March 9, 2027 · 5 min · XB Field Notes
Historical nginx 1.28.0 fixture: failed connection recorded as 502, successful backup attempt 200, and one final client response 200.

An access log separates client outcomes from upstream attempts

An nginx access log records request information when logging runs at the end of processing. The selected fields can distinguish what the client requested, which URI ultimately handled it, what response was sent, and what happened during upstream attempts. Keep these observations separate: one status, duration, or path cannot explain the whole request. ...

March 1, 2027 · 6 min · XB Field Notes
Los Angeles 2026 transition examples: repeated 01:30 branches to 08:30Z and 09:30Z; spring 02:30 points to a missing-time mark.

Timestamps need explicit time semantics

A wall-clock label such as 2026-11-01 01:30 does not identify one instant by itself. Its interpretation depends on a timezone and sometimes on which occurrence was intended during a clock transition. Conversely, an explicitly defined Unix timestamp already carries epoch and unit semantics without displaying a zone name. Decide which kind of value your system is storing before choosing a format. ...

November 29, 2026 · 5 min · XB Field Notes
Each logrotate invocation can advance the eligible active log into generation one and the previous generation into compressed generation two. Two retained generations and a size threshold do not continuously cap disk usage.

Logrotate applies a rotation policy when it runs

Logrotate evaluates configuration when invoked, rotates eligible logs, and applies the selected retention and compression actions. A timer or another scheduler determines when that evaluation happens. The tool does not continuously cap file size, and renaming a log does not automatically redirect an application’s open descriptor. A working setup combines evaluation frequency, file policy, and the writer’s actual reopening behavior. ...

November 21, 2026 · 8 min · XB Field Notes
A complete synthetic population has 100 requests and four failures, shown as a 10-by-10 grid with four red cells. A deliberate selection keeps five records, including two failures: 4/100 is 4%, while 2/5 is 40%. The selected fraction is not service-wide.

Metrics and logs need an explicit population

Metrics and logs are different representations of observed behavior, not competing definitions of truth. A metric usually summarizes measurements under a set of dimensions. A log record describes an event and can carry detailed context. Both can answer aggregate questions if their coverage and semantics support the calculation. Neither becomes authoritative merely because its dashboard looks cleaner or its records contain more fields. ...

October 16, 2026 · 5 min · XB Field Notes