Linux · XB Field Notes

LVM snapshots capture a block device in time

Mon, 10 Apr 2028 00:00:00 +0000

A classic LVM snapshot uses a copy-on-write exception store attached to an origin logical volume. lvcreate -s -L 5G -n snap vg/data requests a 5 GiB store, subject to allocation granularity and available volume-group space. It avoids copying the whole origin at creation, but does not guarantee zero work or instant completion. Thin snapshots use a different shared-pool mechanism.

The exception store must cover distinct changed chunks and metadata for the entire retention window, including writes during backup. A burst can consume it earlier than a quiet-workload estimate predicts. If a classic store fills, the snapshot becomes invalid and reads fail. Monitor validity and growth early enough to extend or finish the backup; a nominal percentage alone is not a recovery plan.

An exception store, not a copy

The origin uses the snapshot-origin target, while snapshot combines unchanged origin chunks with preserved exceptions. Writes to the snapshot itself go to its COW device. Repeated changes to an already-excepted chunk do not require another original copy for that snapshot. The earlier million-write, sixteen-chunk, fifteen-exception result has no recovered trace; touching all sixteen distinct chunks would require sixteen data exceptions, plus metadata.

lvs -a -o lv_name,lv_attr,origin,data_percent,lv_size exposes type, state and usage. swi-a-s--- is illustrative, not one universal attribute string; the fifth character I means invalid snapshot. Alert thresholds must leave room for observed write rates and response time. lvextend needs available VG extents. lvchange --monitor y enables monitoring, not guaranteed extension: dmeventd, configured autoextend policy and free capacity are also required.

Exhaustion is invalidation

Kernel documentation says an exhausted classic COW store is disabled and returns errors. The lvcreate manual’s twenty-percent-of-origin command is an example, not a general sizing recommendation. Account for chunk granularity, metadata, workload bursts and the full backup duration. Increasing capacity before exhaustion preserves a valid snapshot; an invalid recovery point must not be treated as complete merely because its LV still exists.

Origin writes normally continue when a classic snapshot overflows, but this does not guarantee that unrelated origin or storage failures cannot occur. An invalid snapshot is unusable as the recovery point; it did exist and may have supplied an earlier verified backup. Do not confuse current snapshot validity with the separate integrity of a copy already completed elsewhere.

Merging, and why thin changes the failure

lvconvert --merge vg/snap merges a persistent COW snapshot back into its origin and consumes the snapshot. It starts immediately when both LVs are closed; otherwise LVM defers it until an activation when both are closed. The origin may serve I/O after merging begins, with affected writes deferred as required. This is not a guarantee of interruption-free rollback, and newer origin data can be discarded.

A thin snapshot shares data and metadata capacity with other thin LVs instead of owning a separately sized COW store. Data exhaustion may queue writes or return errors according to policy; metadata exhaustion is another failure mode. Such failures can affect multiple volumes and filesystems, but do not imply instant destruction of every volume. Monitor both pool percentages, policy and extension capacity.

When it breaks

A block snapshot is not automatically an application-consistent backup. Filesystem and database buffers, logging and data spread across volumes require coordinated handling and recovery. During copying, check snapshot validity and propagate read failures; verify the resulting contents and a restore. A successful command exit or mount alone does not establish completeness, and this review performed no LVM creation, exhaustion, extension or merge experiment.

Before a merge, identify open users of both origin and snapshot and plan the application outage or activation boundary. Rehearse on disposable volumes and keep an independent backup. Size for all distinct changes across the retention window, not just its busiest hour. Confirm the actual resulting volume and restored application state; do not infer success solely from declining allocation counters.

LVM keeps changed chunks — mechanism diagram.

Check it quickly

The unchanged lines below remain held. They contain environment-specific names and require authorized LVM access; lvextend changes allocation, so this is not a read-only diagnostic block. No device-mapper, kernel, volume or mount operation was executed:

lvs -a -o lv_name,vg_name,lv_attr,origin,data_percent,lv_size
lvs -a -o lv_name,data_percent,metadata_percent vg/thinpool
dmsetup status vg-snap
lvextend -L +2G vg/snap

Classic snapshot Data% and its validity state differ from a thin pool’s Data% and Meta%. dmsetup status output is target-specific: normal snapshot allocation need not fall, while snapshot-merge allocation can decrease toward its metadata remainder. The final line neither proves free VG capacity nor revives an invalid snapshot. A separately held complete exercise must bind targets and verify each claimed transition.

Sources

lvcreate(8), documents -s/--snapshot, COW versus thin snapshots and illustrative sizing commands, checked 2026-09-16.
lvs(8), defines the lv_attr state characters including invalid snapshot, checked 2026-09-16.
Device-mapper snapshot support, the exception store, invalidation and lvconvert --merge behaviour, checked 2026-09-16.
lvmthin(7), thin snapshots, pool exhaustion and automatic extension, checked 2026-09-16.
lvconvert(8), immediate versus deferred merge and open-volume conditions, checked 2026-10-18.
PostgreSQL filesystem backup, consistent snapshot scope, WAL and multi-volume recovery requirements, checked 2026-10-18.

The SYN backlog holds incomplete handshakes

Fri, 24 Mar 2028 00:00:00 +0000

A listening TCP socket separates incomplete handshakes from established connections waiting for accept(). The real setting is tcp_max_syn_backlog; this article retains its historical URL spelling. It concerns remembered incomplete requests, whereas the listen() backlog concerns the accept queue. Diagnose which stage is under pressure before choosing a remedy.

SYN cookies provide a fallback when normal handshake state cannot be retained. They do not make servers immune to overload, guarantee delivery, or remove the need to understand the SYN queue. Packet loss, filtering, allocation failures and accept-side pressure can still prevent connections with cookies enabled.

The SYN queue holds incomplete handshakes; the accept queue holds established connections awaiting accept(). With listen backlog configured as 8192 and somaxconn as 4096, the accept-side cap is 4096. Buffering more waiting work does not itself increase processing rate. The source article remains held/pending. Whole article remains held/pending; this diagram does not endorse its disputed claims.

The backlog argument is the accept queue

Linux changed the listen() backlog meaning in 2.2. Its configured value is capped by somaxconn; with arguments 8192 and 4096 respectively, that cap is 4096. This is a configuration example, not a measurement of exact admission count. Read the actual listener and namespace rather than assuming defaults or inferring capacity from a short burst.

Kernel documentation describes tcp_max_syn_backlog as a per-listener limit on remembered requests in SYN_RECV. Its initial value depends on memory and kernel implementation. Old man-page memory thresholds should not be treated as the current host’s configuration. Neither queue length alone identifies the application’s processing rate.

SYN cookies remain a fallback

With kernel support, tcp_syncookies=1 enables the overflow fallback; value 2 is a testing mode that sends cookies unconditionally. The listen manual’s statement that cookies remove the logical bound describes stateless admission beyond remembered requests. It does not mean unlimited memory, CPU, bandwidth or established sockets. The kernel documentation still recommends considering SYN-backlog tuning for legitimate overload, alongside the application’s ability to drain work.

Do not promise that every cookie connection loses SACK and window scaling. The pinned Linux v6.12 implementation can encode these options in timestamp bits when timestamps are negotiated. Which options survive depends on implementation and negotiation. That qualified behavior does not turn an emergency mechanism into a capacity plan, and disabling cookies is not a general fix.

The accept queue is where the pressure lands

Accept-side pressure can coexist with SYN-side pressure. In Linux v6.12’s ordinary handshake path, failure to create a child socket can reach an overflow branch: with tcp_abort_on_overflow disabled, the server can retain the request rather than immediately reset it. A client may already consider the connection established while application traffic stalls. This is a possible state mismatch, not a guarantee that every connect() succeeds.

A larger queue absorbs a finite burst and may increase waiting time. It does not establish higher steady-state service capacity. Measure arrival rate, accept latency and application completion together; there is no measured 20,000-connections-per-second result in this article.

When it breaks

Rising overflow counters justify investigation, but do not uniquely blame a worker thread. Accept-loop scheduling, resource exhaustion, other listeners in the same network namespace and a burst between samples can change the interpretation. Historical nonzero totals are not evidence of a current incident.

SYN loss also occurs with cookies enabled. Client retry timing depends on kernel version, retry settings, socket behavior and application deadlines; 67 or 131 seconds is not a universal timeout contract. A successful TCP connection likewise does not prove that the application request completed.

Check it quickly

Use the following observation exercise only on an authorized Linux system with an existing listener on port 8443, in its network namespace. It requires ss, nstat and sysctl already installed. Do not start listeners, generate overload, change settings or elevate privileges for this exercise. These commands were not run in this review.

ss -lnt 'sport = :8443'
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPReqQFullDrop TcpExtTCPReqQFullDoCookies
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_syncookies net.ipv4.tcp_abort_on_overflow

Take two snapshots around an already-approved ordinary workload. For a listening TCP socket, Recv-Q reports queued established connections and Send-Q the effective backlog limit. Empty output means no matching visible listener, not a zero-length queue. A changed sysctl does not retroactively prove how an existing listener was configured.

nstat -az shows absolute counters, including zeros, and may update its local history; compare deltas over the interval. The named counters separate useful overflow, request-drop and cookie-fallback signals, but they are namespace-wide and are not mutually exclusive per-listener verdicts. Record kernel version, listener context and request outcomes before proposing tuning. Missing tools, access or an isolated test environment leave the runtime conclusion unresolved.

Sources

Kernel ip-sysctl documentation, SYN request limits, cookie modes and overload guidance; earlier check retained 2026-09-16, checked 2026-10-19.
Linux listen(2), accept backlog semantics and the configured somaxconn cap; earlier check retained 2026-09-16, checked 2026-10-19.
Linux tcp(7), version-sensitive settings and retry behavior; earlier check retained 2026-09-16, checked 2026-10-19.
Linux v6.12 cookie implementation, timestamp encoding of SACK and window scaling, checked 2026-10-19.
Linux v6.12 handshake implementation, conditional listen_overflow behavior, checked 2026-10-19.
iproute2 nstat manual, absolute-counter and zero-value options, checked 2026-10-19.

iotop shows which process is burning the disks

Wed, 15 Mar 2028 00:00:00 +0000

iotop connects task-level kernel accounting with a system-wide storage investigation. It can identify tasks reporting substantial IO or waiting, but those are different observations. A task blocked behind other work may be a victim rather than the source of contention. Kernel counters do not provide perfect attribution for every filesystem, cache, or storage layer.

Use iotop beside device and application measurements. Neither an active writer nor a high wait value alone proves a storage bottleneck. Compare the same time window before deciding which component deserves closer inspection.

Per-process counters

Check the installed implementation and version. In the cited iotop-c, main-process rows already include thread activity; -P hides separate thread rows. Other variants need their own manual. The -o option filters for observed IO activity; it does not classify harmful work. Read/write byte rates and IO-wait percentages have different units and should not be ranked as interchangeable measures.

Linux read_bytes accounts storage-layer reads; cached reads need not produce them. write_bytes is accounted when pages are dirtied, not necessarily when the device completes a write. Cancelled writes and later writeback can separate task totals from device throughput. Delay accounting records task waiting, including synchronous block-IO delay; CPU iowait is a separate aggregate and is not a reliable per-task blame measure.

See whatever sweeps the disk

Choose an explicit sort column and observe long enough to distinguish sustained work from a burst. A high write rate can be an expected backup, index build, or logging workload. Correlate process ownership, application role, and current work before changing scheduling or limits. A high rank identifies a candidate, not the correct remedy.

Resolve the task in the relevant PID namespace and verify that it still refers to the intended process. Command lines, parent relationships, and open files provide context, but an open file is not proof that all observed writes went there. PIDs can be reused between measurements; a stale number can point at unrelated work.

Watch before you act

Observation has costs and access requirements. Additional syscall tracing may perturb the process and expose arguments, so choose a separately authorized, scoped investigation when needed. An append flag belongs to opening a file; it is not a flag on each write syscall. Avoid presenting an imagined combination of outputs as a measured diagnosis.

Record collection conditions for a useful baseline: implementation, interval, process/thread mode, accumulation mode, and comparable workload. With the documented default one-second delay, -b -n 60 is roughly a minute, not five minutes. Device await includes queueing and service time; near-100-percent utilization does not establish saturation on every parallel SSD or RAID device.

When it breaks

Missing permissions, kernel accounting options, or enabled task delay accounting can cause failure or unavailable fields. An empty display does not exonerate either your processes or other users’ processes. Record the actual limitation and required environment; do not silently change capabilities or kernel settings to make the tool work. Some delay-accounting changes affect only tasks started afterward.

Bursty activity can disappear in averages, and short-lived tasks may escape sampling. Varying intervals changes what is visible. Accumulated mode summarizes observed accounting within its applicable lifetime; it does not recover an unrecorded previous hour or guarantee complete coverage of exited tasks. State the observation window with every conclusion.

Compare task bytes, task waiting and device observations over the same interval. Capture and identity checks remain unexecuted; a fresh snapshot cannot prove identity at an earlier sample.

Check it quickly

This unexecuted proposal targets the cited iotop-c interface and sysstat iostat. Confirm the installed implementations first. It captures full reports, stderr and both statuses without selecting a process from text. A fresh configuration directory prevents saved iotop preferences from silently changing the view.

(
 capture=$(mktemp -d) || exit 1
 printf 'Evidence directory: %s\n' "$capture"
 iotop --version || exit $?
 iostat -V || exit $?
 LC_ALL=C iostat -x -y -t 2 5 > "$capture/iostat.txt" 2> "$capture/iostat.err" &
 stat_pid=$!
 if LC_ALL=C XDG_CONFIG_HOME="$capture" iotop -b -P -k -t -d 2 -n 5 \
 --no-only --no-accumulated --no-accum-bw \
 > "$capture/iotop.txt" 2> "$capture/iotop.err"; then
 task_status=0
 else
 task_status=$?
 fi
 if wait "$stat_pid"; then stat_status=0; else stat_status=$?; fi
 printf 'iotop_status=%s iostat_status=%s\n' "$task_status" "$stat_status"
 cat "$capture/iotop.err" "$capture/iostat.err"
 [ "$task_status" -eq 0 ] && [ "$stat_status" -eq 0 ] || exit 1
 cat "$capture/iotop.txt" "$capture/iostat.txt"
)

In iotop-c, -P hides separate thread rows; their activity is already included in process totals. Fixed kilobyte units and non-accumulated mode support comparisons. This batch capture does not promise descending write order. Compare the labelled DISK WRITE values within each report; for an interactive ranking, explicitly select that column and descending direction. Never treat the first printed row as the largest writer.

Compare overlapping timestamped intervals; launch times are not perfectly synchronized. -y excludes iostat’s since-boot report. Missing counters, nonzero status or incomplete windows invalidate the comparison.

Save this helper as inspect_io_process.py. Select a process PID manually from the full report, in the same PID namespace. Python 3.9+, Linux pidfd support and readable proc entries are required:

"""Read-only Linux identity snapshot; requires Python 3.9+ and pidfd support."""
import json
import os
import re
import select
import sys


def snapshot(token, proc_root="/proc", ops=os, poll_factory=select.poll):
 if not re.fullmatch(r"[1-9][0-9]*", token):
 raise ValueError("PID must be a positive decimal process ID")
 pid = int(token)
 pidfd = ops.pidfd_open(pid, 0)
 directory = None
 try:
 directory = ops.open(f"{proc_root}/{pid}", os.O_RDONLY | os.O_DIRECTORY)
 watcher = poll_factory()
 watcher.register(pidfd, select.POLLIN)

 def read(name, limit):
 fd = ops.open(name, os.O_RDONLY, dir_fd=directory)
 with ops.fdopen(fd, "rb") as source:
 data = source.read(limit + 1)
 if len(data) > limit:
 raise ValueError(f"{name} exceeds the snapshot limit")
 return data

 def identity():
 head, tail = read("stat", 65536).rsplit(b")", 1)
 if int(head.split(b" ", 1)[0]) != pid:
 raise ValueError("stat PID mismatch")
 fields = tail.split() # field 3 is index 0; field 22 is index 19
 if fields[0] in (b"Z", b"X", b"x"):
 raise ProcessLookupError("process has exited")
 return int(fields[19])

 before = identity()
 executable = ops.readlink("exe", dir_fd=directory)
 raw_command = read("cmdline", 65536)
 # Remove one final NUL; empty input means no argument data.
 command = raw_command[:-1] if raw_command.endswith(b"\0") else raw_command
 command = command.split(b"\0") if raw_command else []
 after = identity()
 if before != after or watcher.poll(0):
 raise ProcessLookupError("identity changed or process exited")
 return {"pid": pid, "starttime_ticks": before, "executable": executable,
 "argv": [arg.decode("utf-8", "backslashreplace") for arg in command]}
 finally:
 if directory is not None:
 ops.close(directory)
 ops.close(pidfd)


if __name__ == "__main__":
 try:
 if len(sys.argv) != 2:
 raise ValueError("usage: inspect_io_process.py PID")
 print(json.dumps(snapshot(sys.argv[1]), ensure_ascii=True))
 except (OSError, ValueError, IndexError, AttributeError) as error:
 sys.exit(f"Identity snapshot failed: {error}")

python3 inspect_io_process.py "${PID:?Set PID from the reviewed process row}"

A successful snapshot reports an executable, arguments and stable start-time value while checking a pidfd for exit. Permission, missing-process or unavailable-pidfd errors stop the check. PID reuse before acquiring the handle cannot be excluded; this snapshot does not prove identity at the earlier sample. Proc fields can change and are not atomic. Recheck before any later action; nothing is killed or reprioritized here.

Sources

iotop man page, rendered iotop-c manual; verify the installed variant, checked 2026-10-18.
vmstat man page, documents system-wide CPU and IO-wait reporting, checked 2026-09-05.
Linux delay accounting, taskstats delay scope and enablement prerequisites, checked 2026-10-18.
Linux proc filesystem, storage byte accounting, cancelled writes, and CPU iowait limits, checked 2026-10-18.
iostat manual, first report, await, and parallel-device utilization limits, checked 2026-10-18.
iotop-c upstream manual, process aggregation, batch options, configuration and interactive sorting, checked 2026-10-19.
sysstat upstream iostat manual, -y omits the since-boot report and -t timestamps interval reports, checked 2026-10-19.
pidfd_open manual, bounded identity or controlled-operation contract, checked 2026-10-19.
Python os interfaces, bounded identity or controlled-operation contract, checked 2026-10-19.
proc_pid_stat manual, bounded identity or controlled-operation contract, checked 2026-10-19.

Network namespaces isolate networking; their names do not own every process

Sat, 19 Feb 2028 00:00:00 +0000

A Linux network namespace separates networking resources such as interfaces, routes, neighbor tables, socket ports, and packet-filter rules. It shares the kernel with other namespaces; it is not a separate operating system. A process normally inherits its parent’s network namespace, and tools can execute a process in another one. The resulting isolation covers networking resources, not every aspect of process identity or filesystem access.

When it breaks

Running ip netns exec does not automatically provide the full isolation of a container. PID namespaces, user namespaces, and hostname isolation are separate mechanisms. An application may see different routes while still sharing ordinary files and a hostname with its parent environment. Treating one network namespace as a complete sandbox can therefore assign it guarantees that were never configured.

A new namespace also does not automatically receive useful external connectivity. Interfaces need deliberate placement and configuration, and local loopback may need to be brought up. Moving an interface changes which namespace manages it; the old context will no longer list that device as its own. A configuration command executed from the wrong namespace can report a missing interface even though the intended device still exists elsewhere.

Namespace names have their own lifecycle. ip netns conventionally exposes handles beneath /run/netns, commonly also reachable through /var/run/netns. Deleting a name removes that handle, but a process or another reference can keep the namespace alive. Cleanup that equates disappearance from the named list with termination of every service can leave networking resources and a running process behind.

Networking scopes differ while selected context stays shared. Name deletion can leave an owned process reference alive; current runtime remains held.

Check it quickly

These are historical article reports; no raw integration logs were recovered and no privileged operations were rerun.

Save as check-network-namespace.py and run python3 check-network-namespace.py only in an authorized disposable VM with Docker and enabled Python assertions. Pinned Python 3.11.16/iproute2 6.1.0 packages must remain available. NET_ADMIN and SYS_ADMIN are prerequisites, not guarantees against other security restrictions. Do not weaken host protections. A temporary proc mount supports namespace-local settings; host interfaces and published ports are untouched:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found

def server(stack,name,address,port=18080,label='server'):
 code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
 address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
 def server_bind(self):
 if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
 super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
 assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
 code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
 client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
 assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
 return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

with lab(['blue','green']) as (root,stack,setting):
 probe='import json,os,socket;print(json.dumps({"net":os.readlink("/proc/self/ns/net"),"pid":os.readlink("/proc/self/ns/pid"),"hostname":socket.gethostname()}))'
 parent=json.loads(run('python3','-c',probe).stdout)
 views={name:json.loads(ns(name,'python3','-c',probe).stdout) for name in ['blue','green']}
 assert len({parent['net'],views['blue']['net'],views['green']['net']})==3
 assert all(v['pid']==parent['pid'] and v['hostname']==parent['hostname'] for v in views.values())
 for name in ['blue','green']:
 ip(name,'link','add','dummy0','type','dummy');ip(name,'addr','add','192.0.2.10/24','dev','dummy0');ip(name,'link','set','dummy0','up')
 ip('blue','route','add','203.0.113.0/24','dev','dummy0')
 assert json.loads(ip('blue','-j','route','show','203.0.113.0/24').stdout)
 assert not json.loads(ip('green','-j','route','show','203.0.113.0/24').stdout)
 setting('blue','net.ipv4.ip_forward',0);setting('green','net.ipv4.ip_forward',1)
 forwarding={name:ns(name,'sysctl','-n','net.ipv4.ip_forward').stdout.strip() for name in ['blue','green']};assert forwarding=={'blue':'0','green':'1'}
 marker=root/'shared.txt';marker.write_text('shared filesystem')
 assert ns('blue','cat',str(marker)).stdout==ns('green','cat',str(marker)).stdout=='shared filesystem'
 daemon=server(stack,'blue','192.0.2.10',label='blue')
 named=request('blue','192.0.2.10');assert named['ok'] and named['body']['server']=='blue'
 run('ip','netns','delete','blue');assert not Path('/run/netns/blue').exists()
 assert daemon.poll() is None and os.readlink(f'/proc/{daemon.pid}/ns/net')==views['blue']['net']
 by_pid=run('nsenter','-t',str(daemon.pid),'-n','python3','-c',
 'import http.client,json; c=http.client.HTTPConnection("192.0.2.10",18080,timeout=1);c.request("GET","/");r=c.getresponse();print(json.dumps({"status":r.status,"body":json.loads(r.read())}));c.close()')
 surviving=json.loads(by_pid.stdout);assert surviving['status']==200 and surviving['body']['server']=='blue'
 print(json.dumps({'parent':parent,'namespace_views':views,'blue_only_route':True,'forwarding':forwarding,
 'same_file_visible':True,'before_name_deleted':named,'name_deleted_process_alive':True,'after_name_deleted':surviving}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The parent, blue, and green report three different network namespace identifiers. Their PID namespace identifiers and hostnames remain equal. A marker file created by the parent is also readable in both children. Those positive observations establish the intended scope: networking has changed, while these particular process and filesystem properties remain shared in the fixture.

Each child creates its own dummy0 with address 192.0.2.10/24. The identical names and addresses coexist because they belong to different networking contexts. Only blue receives a route for 203.0.113.0/24. Querying green’s routes confirms that it has no corresponding entry. The example verifies separate effective tables rather than merely printing the two namespace names.

The program explicitly sets IPv4 forwarding to zero in blue and one in green, then reads the values back through each namespace. This is a setting-isolation check, not a packet-forwarding demonstration. The local dummy interfaces have no physical peer. Configured addresses and routes can be inspected without implying that either namespace can reach an external service.

A real HTTP server starts in blue on 192.0.2.10. A normal named-namespace request receives status 200 and identity blue. The script then deletes the blue name and confirms its handle is absent, while the server process and its network namespace identifier remain. The process has not been killed by removing the naming reference.

Using that known server PID, nsenter executes another client in the surviving network namespace. It again receives status 200 and identity blue. This establishes continued application operation after name deletion, not just the presence of a proc entry. Cleanup attempts to stop its process and remove named resources; interruption can leave work unfinished.

Keep naming, process ownership, and connectivity separate

Record both a service’s process ownership and the networking context used for management. Namespace identifiers help compare currently live contexts; do not turn an ephemeral identifier into a permanent inventory identity. If a name disappears unexpectedly, inspect the relevant owned process before concluding that its networking state has vanished or creating a replacement with assumed empty state.

For ordinary configuration files, ip netns exec can create a mount namespace and bind namespace-specific files into their conventional locations. This supports arrangements such as per-namespace resolver configuration; it does not establish a private copy of every filesystem object. Document those additional choices when reproducing a client’s environment so that DNS settings and route isolation are not confused.

Build connectivity through explicit virtual links, routes, forwarding, and filtering as required. A network namespace supplies a boundary for those resources, while their configuration determines communication. During teardown, stop the processes and release references owned by the service before expecting the namespace to disappear completely. Name deletion is one management operation within that lifecycle, not a universal process-termination command.

Sources

Linux network_namespaces(7), network resources isolated by a namespace, checked 2026-09-12.
Linux ip-netns(8), named handles, execution and deletion while processes remain, checked 2026-09-12.
Linux ip-link(8), interface creation and namespace placement, checked 2026-09-12.

Strace explains selected system-call boundaries

Mon, 14 Feb 2028 00:00:00 +0000

Strace observes system calls and signals at the boundary between a selected Linux process and the kernel. It can reveal attempted paths, arguments, return values, and errors without rebuilding the program. That evidence is useful when application logs omit an operation, but it does not expose every user-space decision or automatically identify which failed call caused the application’s final outcome.

These are the source fixture’s two controlled outcomes, not newly captured traces. The same error can have different control-flow significance. Only selected process/syscall boundaries are shown, not all application work or a complete causal trace. Current audit remains revised, execution blocked; no tracing was attempted.

When it breaks

A parent can launch a child that performs the relevant work. Tracing only the parent may show waiting and exit behavior while missing the child’s failed file open. Following created processes with -f expands coverage, but does not automatically trace every preexisting sibling or external service involved in the request. The selected process set is part of the evidence.

An error return is not necessarily the root cause. Programs routinely probe optional files, search several locations, and recover from expected failures. Finding the first ENOENT, or the last unsuccessful call before exit, cannot replace understanding application control flow. The meaningful question is how that result affected the operation being investigated.

Filters and timing also shape interpretation. Restricting output to file opens excludes network and other activity by design. Tracing can change scheduling and latency, especially for workloads making many system calls; there is no universal slowdown factor. A clean filtered trace does not prove that excluded operations succeeded or that the same timing behavior occurs without a tracer.

Check it quickly

The current review did not repeat the tracing integration. The recorded outcomes describe the earlier environment, not verified ptrace permission in every container.

Save this as straceprobe.py and run python3 straceprobe.py in a disposable Linux environment with strace installed. The recorded run used Linux 6.12.76, Python 3.11.2, and strace 6.1. The fixture starts its own harmless program, uses temporary configuration paths, and requires permission to trace the children it launches:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,sys

child=r'''
from pathlib import Path
import sys
try:Path(sys.argv[1]).read_bytes()
except FileNotFoundError:pass
try:content=Path(sys.argv[2]).read_bytes()
except FileNotFoundError:sys.exit(7)
assert content==b'configured'
'''
parent='import subprocess,sys\nchild='+repr(child)+'\nsys.exit(subprocess.run([sys.executable,"-c",child,*sys.argv[1:]]).returncode)'
with TemporaryDirectory(prefix='xbcatm-strace-') as directory:
 root=Path(directory);optional=root/'optional.conf';required=root/'required.conf'
 command=[sys.executable,'-c',parent,str(optional),str(required)]
 def trace(name,follow):
 output=root/(name+'.trace')
 args=['strace']+(['-f'] if follow else [])+['-tt','-T','-yy','-s','200','-e','trace=openat,close','-o',str(output),'--',*command]
 result=subprocess.run(args,capture_output=True,text=True,timeout=10)
 return result.returncode,output.read_text(),result.stderr
 code,unfollowed,error=trace('parent-only',False)
 assert code==7 and str(required) not in unfollowed,(code,error)
 code,followed,error=trace('with-children',True)
 assert code==7,(code,error)
 def calls(text,path):return [line for line in text.splitlines() if 'openat(' in line and '"'+str(path)+'"' in line]
 for path in [optional,required]:
 lines=calls(followed,path);assert len(lines)==1 and 'ENOENT' in lines[0],lines
 print('failing run:',lines[0])
 print('without -f the parent trace misses the child opens; following the child exposes both missing paths')
 required.write_bytes(b'configured')
 code,recovered,error=trace('recovered',True);assert code==0,(code,error)
 assert 'ENOENT' in calls(recovered,optional)[0]
 assert ' = -1 ' not in calls(recovered,required)[0]
 print('successful run:',calls(recovered,optional)[0])
 print('successful run:',calls(recovered,required)[0])
 print('a successful application can contain ENOENT; the fixture distinguishes optional lookup from required configuration')
 summary=root/'summary'
 result=subprocess.run(['strace','-f','-c','-e','trace=openat','-o',str(summary),'--',*command],capture_output=True,text=True,timeout=10)
 assert result.returncode==0,result.stderr
 text=summary.read_text();assert 'openat' in text and 'total' in text
 print('syscall summary:\n'+text)

The application child tries an optional configuration file and tolerates its absence. It then requires a second configuration file, exiting with status seven when that file is missing. The parent returns the child’s status. This produces a complete failure whose cause is known from the fixture code, rather than relying on an unavailable placeholder application.

The first trace omits -f. The command returns seven, but the required path does not appear among the traced parent opens. Repeating with -f records the child’s opens for both optional and required paths, each returning ENOENT. The comparison demonstrates why process selection can hide the relevant operation even when the overall command’s failure is visible.

The fixture then creates only the required file with the expected content. The next run exits zero, and its required open succeeds. The optional lookup still returns ENOENT. This successful run is a deliberate counterexample to the claim that any error line proves application failure; the optional and required results have different roles in the program’s logic.

The trace includes -tt timestamps, -T elapsed durations for traced calls, and -yy descriptor context. A bounded string length keeps the controlled paths visible, while -o separates trace output from application output. The script asserts actual return codes and targeted open results instead of treating the existence of a trace file as sufficient verification.

A final -c run summarizes the selected openat calls, including their errors, for the recovered application. Its count includes runtime startup as well as fixture work. These totals are not the number of business requests, and default summary timing should not be equated with total application wall time. Exact counts and durations may vary with the runtime and environment.

Follow a hypothesis through to an outcome

Start from a concrete operation, identify the processes performing it, and choose a filter that preserves the evidence needed to test the hypothesis. File, network, and process tracing answer different questions. Expand the selection when a dependency falls outside the current view, while keeping the captured workload small enough to interpret.

Attaching with -p depends on process identity and tracing permissions. Credentials, security policy, namespace relationships, and container configuration can prevent access even when the PID exists. This fixture traces newly launched children without extra container capabilities; that result does not establish permission to attach to arbitrary existing services on another system.

Keep captured arguments and data appropriate to the investigation. Trace output can contain paths, request data, or credentials, so use controlled inputs and limit what is retained. Correlate the trace with application behavior, reproduce the proposed fix, and check the final outcome. Here, successful recovery with the optional error still present provides that causal comparison.

Sources

strace(1), follow and filter options, return values, timing, summaries, and decoding, checked 2026-09-12.
ptrace(2), tracing relationships, stops, and access restrictions, checked 2026-09-12.

Cron system and user tables have different contracts

Wed, 12 Jan 2028 00:00:00 +0000

User crontabs belong to one account; system tables can name an account per entry. Copying lines between formats without adjusting fields changes the command boundary. Identify the format before debugging the calendar expression.

In Debian user tables the command starts after five fields; system tables insert a username before it. Unescaped percent signs are handled before shell quoting and split off standard input. The figure is schematic; privileged examples were not executed.

The account column changes the parser boundary

User entries contain minute, hour, day of month, month, day of week, then command text; crontab installs them for the selected user. Debian’s /etc/crontab and /etc/cron.d files insert a username after those five fields. It is not an optional command annotation.

Month remains required. When day-of-month and day-of-week are both restricted, Debian/Vixie permits either day match, not month versus weekday. Check dialects before transferring expressions to schedulers with seconds fields or different day rules; similar spacing proves nothing.

System drop-ins require suitable ownership, permissions, filenames, and final newlines. Debian accepts root-owned files without group/other write permission and appropriate names, not arbitrary backup suffixes. Periodic directories such as /etc/cron.daily follow distribution scheduling, commonly through run-parts; scripts are not individually parsed as crontabs. Inspect the installed arrangement for timing.

When it breaks

Putting root after five fields in root’s user crontab asks the shell to execute a command named root; successful installation does not prove it exists. Omitting a system entry’s account instead puts command text in the username position. Check boundaries and effective identity, especially for protected data or writes.

Do not assume interactive aliases, shell startup files, or daemon-parent variables are available. Explicitly set required values, PATH, and interpreter. Nor should cron.d settings be assumed to inherit /etc/crontab’s environment. Test permissions and outputs as the execution account.

Before shell processing, unescaped percent signs delimit standard input; subsequent percents become newlines. Shell quoting cannot prevent this. Put complex commands in scripts and escape literal percents in entries. Check scheduling syntax, command execution, and application outcomes separately.

Check it quickly

Repair draft checked 2026-10-19: this revision was checked statically and with offline command/file-state doubles only. No daemon, account or task-table changes were executed. Technical and operational verification remain blocked. The version-specific outcomes below describe the original fixture.

A future run needs a separately authorized, freshly provisioned disposable Debian 12 lab with Python 3.11+, cron 3.0pl1-162, procps and passwd, an executable /tmp for the guarded shell script, no host mounts, no cron daemon or root table, no xbtime account or test drop-in, no enabled unrelated schedules in /etc/crontab, /etc/cron.d or any user’s spool (including periodic-directory dispatch), and no concurrent administration. Save as cron-tables.py; use python3 cron-tables.py --isolated-disposable-lab only after independently establishing those conditions. The flag and root identity do not prove isolation. This still modifies lab-wide account files and cron state; discard the whole lab afterward.

The temporary system account uses existing nogroup and the private work directory as its home, with no new home or private group and no lastlog initialization; its UID is discovered, not assumed to be 1000. Account creation, lookup, table installation and daemon startup are protected from the first attempted mutation. Both test tables disable mail with MAILTO. The exact pinned absence diagnostic is distinguished from permission or command failure. Workers publish JSON by same-directory replacement; the deliberately invalid entry still starts with root after its five fields. Only that entry selects a small SHELL wrapper that checks the permit under the shared lock, preserves the lock descriptor across exec and passes the unchanged command to /bin/sh. Its diagnostic is published by rename. A positive run permit and shared lifetime lock bound admitted worker cleanup. Every cleanup action is attempted independently, with bounded daemon and worker waits; errors preserve the original failure and skip private-file deletion; a failed deletion can leave only partial files. Revoking the permit prevents late workers from treating a deleted stop file as permission. The barrier does not prove every cron wrapper exited, so lab disposal remains required.

from pathlib import Path
from tempfile import mkdtemp
import fcntl
import json
import os
import pwd
import shutil
import subprocess
import sys
import time


def command(arguments, **options):
 return subprocess.run(arguments, capture_output=True, text=True, timeout=5,
 env={**os.environ, 'LC_ALL': 'C'}, **options)


def table_absent(result):
 # Debian cron 3.0pl1-162: other listing failures must abort.
 return (result.returncode == 1 and result.stdout == ''
 and result.stderr == 'no crontab for root\n')


def remove_table():
 current = command(['crontab', '-l'])
 if table_absent(current):
 return
 current.check_returncode()
 command(['crontab', '-r']).check_returncode()
 if not table_absent(command(['crontab', '-l'])):
 raise RuntimeError('could not verify table removal')


def attempt(errors, name, operation):
 try:
 operation()
 except Exception as error:
 errors.append(f'{name}: {error!r}')


def stop_daemon(daemon, errors):
 if daemon is None:
 return
 attempt(errors, 'terminate cron', daemon.terminate)
 try:
 daemon.wait(timeout=5)
 except subprocess.TimeoutExpired:
 attempt(errors, 'kill owned cron', daemon.kill)
 attempt(errors, 'reap cron', lambda: daemon.wait(timeout=5))
 except Exception as error:
 errors.append(f'wait cron: {error!r}')


def remove_account(username):
 try:
 pwd.getpwnam(username)
 except KeyError:
 return
 command(['userdel', username]).check_returncode()
 try:
 pwd.getpwnam(username)
 except KeyError:
 return
 raise RuntimeError('account removal not verified')


def wait_workers(root):
 with (root / 'lifetime.lock').open() as gate:
 deadline = time.monotonic() + 5
 while True:
 try:
 fcntl.flock(gate, fcntl.LOCK_EX | fcntl.LOCK_NB)
 return
 except BlockingIOError:
 if time.monotonic() >= deadline:
 raise TimeoutError('workers did not release lifetime lock')
 time.sleep(0.1)


worker_source = r'''from pathlib import Path
import fcntl,json,os,sys
root=Path(__file__).parent
mode=sys.argv[1]
with (root/'lifetime.lock').open() as gate:
 fcntl.flock(gate,fcntl.LOCK_SH)
 if not (root/'run-permit').exists(): raise SystemExit(0)
 data={'uid':os.getuid(),'table':os.environ.get('LAB_TABLE'),
 'path':os.environ.get('PATH'),'shell':os.environ.get('SHELL'),
 'parent_only':os.environ.get('PARENT_ONLY')}
 if mode=='percent': data['stdin']=sys.stdin.read()
 destination=root/(mode+'.json')
 temporary=root/f'{mode}-{os.getpid()}.tmp'
 temporary.write_text(json.dumps(data))
 temporary.replace(destination)
'''


shell_source = r'''#!/usr/bin/python3
from pathlib import Path
import fcntl,os,sys
root=Path(__file__).parent
with (root/'lifetime.lock').open() as gate:
 fcntl.flock(gate,fcntl.LOCK_SH)
 if not (root/'run-permit').exists(): raise SystemExit(0)
 os.set_inheritable(gate.fileno(),True)
 os.execv('/bin/sh',['/bin/sh',*sys.argv[1:]])
'''


def main():
 if sys.argv[1:] != ['--isolated-disposable-lab']:
 raise SystemExit('requires a separately verified disposable Debian 12 lab')
 if os.geteuid() != 0:
 raise RuntimeError('root is required, but does not prove isolation')
 probe = command(['pgrep', '-x', 'cron'])
 if probe.returncode != 1 or probe.stdout or probe.stderr:
 raise RuntimeError(f'cron absence not established: {probe!r}')
 if not table_absent(command(['crontab', '-l'])):
 raise RuntimeError('expected absent root table; existing table or listing error')
 if shutil.which('root', path='/usr/bin:/bin') is not None:
 raise RuntimeError('the deliberately invalid command exists')
 username = 'xbtime'
 try:
 pwd.getpwnam(username)
 except KeyError:
 pass
 else:
 raise RuntimeError('test account already exists')
 system_table = Path('/etc/cron.d/xbcatm_time_test')
 if system_table.exists() or system_table.is_symlink():
 raise RuntimeError('system test table already exists')
 root = Path(mkdtemp(prefix='cron-tables-', dir='/tmp'))
 daemon = None
 account_attempted = account_created = table_attempted = system_owned = False
 try:
 (root / 'lifetime.lock').touch(mode=0o644)
 (root / 'run-permit').touch()
 account_attempted = True
 command(['useradd', '--system', '--no-create-home', '--no-user-group',
 '--gid', 'nogroup', '--home-dir', str(root), '--no-log-init', '--shell', '/bin/sh', username]).check_returncode()
 account_created = True
 account = pwd.getpwnam(username)
 os.chown(root, account.pw_uid, account.pw_gid)
 worker = root / 'worker.py'
 worker.write_text(worker_source)
 # The account must read both files even under a restrictive parent umask.
 worker.chmod(0o644)
 guarded_shell = root / 'guarded-shell'
 guarded_shell.write_text(shell_source)
 guarded_shell.chmod(0o755)
 (root / 'lifetime.lock').chmod(0o644)
 user_table = ('SHELL=/bin/sh\nPATH=/usr/bin:/bin\nLC_ALL=C\nMAILTO=""\nLAB_TABLE=user\n'
 f'* * * * * /usr/bin/python3 {worker} user\n'
 f'* * * * * /usr/bin/python3 {worker} percent%first%second\n'
 f'SHELL={guarded_shell}\n'
 f'* * * * * root /usr/bin/id >{root}/wrong-column.tmp 2>&1; '
 f'/bin/mv {root}/wrong-column.tmp {root}/wrong-column.err\n')
 table_attempted = True
 command(['crontab', '-'], input=user_table).check_returncode()
 invalid = command(['crontab', '-'], input='* * * * * /bin/true')
 assert invalid.returncode != 0
 listed = command(['crontab', '-l'])
 listed.check_returncode()
 assert listed.stdout == user_table
 with system_table.open('x') as handle:
 system_owned = True
 handle.write('SHELL=/bin/sh\nPATH=/usr/bin:/bin\nLC_ALL=C\nMAILTO=""\nLAB_TABLE=system\n'
 f'* * * * * {username} /usr/bin/python3 {worker} system\n')
 system_table.chmod(0o644)
 daemon = subprocess.Popen(['cron', '-f'], env={**os.environ, 'PARENT_ONLY': 'parent-demo'},
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
 deadline = time.monotonic() + 75
 names = ['user.json', 'system.json', 'percent.json', 'wrong-column.err']
 while time.monotonic() < deadline:
 if daemon.poll() is not None:
 raise RuntimeError('cron exited')
 if all((root / name).exists() for name in names):
 break
 time.sleep(0.2)
 else:
 raise AssertionError('scheduled table outputs were not produced')
 user = json.loads((root / 'user.json').read_text())
 system = json.loads((root / 'system.json').read_text())
 percent = json.loads((root / 'percent.json').read_text())
 wrong = (root / 'wrong-column.err').read_text()
 assert user['uid'] == 0 and user['table'] == 'user'
 assert system['uid'] == account.pw_uid and system['table'] == 'system'
 assert all(value['parent_only'] is None for value in (user, system))
 assert all(value['path'] == '/usr/bin:/bin' for value in (user, system))
 assert all(value['shell'] == '/bin/sh' for value in (user, system))
 assert percent['stdin'] == 'first\nsecond\n'
 assert 'root: not found' in wrong
 print(json.dumps({'user_table': user, 'system_table': system,
 'missing_final_newline_rejected': invalid.returncode,
 'previous_table_preserved': True, 'percent_stdin': percent['stdin'],
 'username_in_user_table_fails': wrong.strip()}), flush=True)
 finally:
 original = sys.exc_info()[1]
 errors = []
 attempt(errors, 'revoke run permit', lambda: (root / 'run-permit').unlink(missing_ok=True))
 if system_owned:
 attempt(errors, 'remove system table', lambda: system_table.unlink(missing_ok=True))
 if table_attempted:
 attempt(errors, 'remove root table', remove_table)
 stop_daemon(daemon, errors)
 attempt(errors, 'wait for workers', lambda: wait_workers(root))
 if account_attempted:
 attempt(errors, 'remove account', lambda: remove_account(username))
 if not account_created:
 errors.append('account creation failed partway; database state needs lab disposal')
 if not errors:
 attempt(errors, 'remove private files', lambda: shutil.rmtree(root))
 if errors:
 message = 'cleanup incomplete; discard lab: ' + '; '.join(errors)
 if original is not None:
 original.add_note(message)
 else:
 raise RuntimeError(message)


if __name__ == '__main__':
 main()

Historically, the user job ran as UID zero and the new system-table account as UID 1000; this revision discovers its system account’s UID instead. Both receive separate LAB_TABLE values and explicit PATH, without the daemon parent’s PARENT_ONLY. Percent parsing supplies first\nsecond\n on stdin. The misplaced username produces root: not found despite successful installation.

A replacement missing its final newline is rejected with exit one, preserving the previous table. The original fixture reported passing assertions on cron 3.0pl1-162; this revision has no real daemon result. Separate probes found malformed minute text accepted even in dry-run mode, so zero exit does not guarantee valid fields. Account, environment, parsing, and installation observations leave mail delivery, daylight-saving handling, and business success untested; verify these separately.

Sources

Debian 12: crontab(5), user/system table syntax, environment and percent handling, checked 2026-09-13.
Debian 12: cron(8), system drop-ins, ownership and execution environment, checked 2026-09-13.
Debian 12: crontab(1), installation, dry-run behavior and newline requirements, checked 2026-09-13.
Python 3.11 subprocess, timeout and child cleanup semantics, checked 2026-10-19.
Debian cron 3.0pl1-162 crontab.c, distinguish absent-table diagnostics from other failures, checked 2026-10-19.
Python 3.11 os.replace, atomic same-filesystem publication, checked 2026-10-19.
Debian cron 3.0pl1-162 do_command.c, configured shell receives -c and the command, checked 2026-10-19.
Python 3.11 os, descriptor inheritance and exec, checked 2026-10-19.
Linux flock(2), lock lifetime across exec, checked 2026-10-19.
Debian 12 useradd(8), system-account, group, home and log-initialization options, checked 2026-10-19.

Shell word splitting turns unquoted variables into several arguments

Fri, 31 Dec 2027 00:00:00 +0000

A shell variable holds a string, but an unquoted expansion can turn that string into several command arguments. In Bash, word splitting examines expansion results using IFS as the delimiter set. This is why a pathname that works when typed with quotes can fail after being stored in a variable. Debug the arguments the receiving command actually gets, including empty ones, before blaming the filesystem.

IFS controls splitting, while quoting also blocks filename expansion. Empty and absent arguments remain different.

Splitting depends on the context

With the default IFS, runs of spaces, tabs, and newlines separate fields. Leading and trailing IFS whitespace is discarded. An unquoted value containing two words therefore becomes two arguments; an empty unquoted value normally contributes none. In contrast, "$value" supplies one argument even when the string is empty. Empty and absent arguments can have completely different meanings to a program.

Splitting applies to results of unquoted parameter expansion, command substitution, and arithmetic expansion in suitable contexts. It is not a second parse of the entire command line. An ordinary assignment such as copy=$value stores the string without splitting or filename expansion. Later passing $copy unquoted to a command reintroduces both hazards. Bash conditional expressions using double brackets also have different expansion rules from ordinary command arguments.

IFS is a set of delimiter characters, not a multi-character separator or a regular expression. Non-whitespace delimiters behave differently from IFS whitespace: adjacent colons, for example, can produce an empty field. Setting IFS to an empty string disables splitting, but it does not turn an empty unquoted expansion into an explicit empty argument. Use a parser suited to the format when quoting, escaping, or structured records matter.

When it breaks

Disabling splitting does not disable the filename expansion that follows it. A variable containing a star can still become matching filenames when IFS is empty. The result can change merely because a file appeared in the working directory. Quoting the variable prevents both transformations; changing IFS alone only addresses one stage. Shell options controlling globbing are independent as well.

A scalar string is also a poor representation of an argument list. Adding quote characters inside the string does not make Bash treat those characters as shell quotation on expansion. The quotation that controls parsing must be in the command’s source. Avoid eval as a repair: it introduces another shell parse and can execute text that was previously data. For Bash commands assembled incrementally, use an array and expand it as "${args[@]}".

Some data is deliberately delimited, so splitting is not always a mistake. Make the format and delimiter explicit, keep changes to IFS local to the operation, and decide what empty fields mean. Do not use whitespace splitting to process arbitrary filenames or CSV records. These examples specifically target Bash; other shells, including zsh in its normal configuration, do not necessarily split scalar expansions the same way.

For example, a semicolon obtained from a variable does not by itself become a command separator during ordinary expansion. It remains argument data, even though splitting can still damage the surrounding value. This distinction separates incorrect argument construction from deliberately reparsing a string as code.

Check it quickly

This self-contained Bash experiment checks argument counts rather than using echo, which hides the distinction between one spaced argument and several arguments. It creates its own two matching files, tests an empty value, and checks the difference between whitespace and colon delimiters. The temporary directory is removed at exit.

bash --noprofile --norc <<'BASH'
set -e
set +f
unset GLOBIGNORE
shopt -u nullglob failglob dotglob
work=$(mktemp -d)
trap 'rm -rf -- "$work"' EXIT
cd "$work"
IFS=$' \t\n'
value='report final.md'
set -- $value
[[ $# -eq 2 ]]
set -- "$value"
[[ $# -eq 1 && $1 == 'report final.md' ]]
value=''
set -- $value
[[ $# -eq 0 ]]
set -- "$value"
[[ $# -eq 1 && $1 == '' ]]
touch a.txt 'two words.txt'
IFS=''
pattern='*.txt'
set -- $pattern
[[ $# -eq 2 ]]
set -- "$pattern"
[[ $# -eq 1 && $1 == '*.txt' ]]
IFS=:
record='red::blue'
set -- $record
[[ $# -eq 3 && $2 == '' ]]
printf 'colon field <%s>\n' "$@"
args=('report final.md' '' '*.txt')
set -- "${args[@]}"
[[ $# -eq 3 && $2 == '' && $3 == '*.txt' ]]
printf 'PASS: boundaries, empty values, globbing, delimiters\n'
BASH

Expect three colon-field lines, with an empty middle field, followed by PASS. The assertions before those lines distinguish zero arguments from one empty argument and prove that an empty IFS still permits globbing. In a failing script, put a small argument-counting function at the call boundary and print each argument separately with Bash printf’s %q format. That diagnostic shows boundaries and escaping; it does not turn a human-readable log into a safe command to execute later.

Sources

GNU Bash: Word Splitting, IFS and empty-field rules, checked 2026-09-06.
GNU Bash: Shell Expansions, splitting before filename expansion, checked 2026-09-06.
GNU Bash: Arrays, preserving a list of arguments, checked 2026-09-06.

Huge pages trade allocation flexibility for translation reach

Tue, 21 Dec 2027 00:00:00 +0000

Huge pages can increase the amount of memory covered by each cached address translation. They can reduce translation overhead for suitable workloads, but the result depends on access patterns and hardware. A large allocation alone does not prove frequent TLB misses, and fewer pages needed to describe a working set does not mean those translations all fit in the processor’s TLB.

Page size is an input

A 4 KiB base page is common on x86, but Linux page sizes depend on architecture and configuration. Query the running system rather than treating four kilobytes as universal. Supported huge-page sizes also vary; 2 MiB and 1 GiB are familiar examples, not a complete list for every processor.

For an idealized contiguous 8 GiB range, 4 KiB pages require 2,097,152 page-sized mappings, while 2 MiB pages require 4,096. That arithmetic describes coverage. It does not measure hardware TLB capacity, miss rate, page-walk latency, or the number of translations simultaneously resident. Different translation structures can also have different capacities for different page sizes.

Explicit pools and transparent allocation

Hugetlb provides explicitly requested huge pages backed by managed pools. Its persistent pages are unavailable for ordinary allocations and are not swapped out. Pools can be allocated at boot or adjusted later; surplus pages may also be allowed. Reserving at boot often makes contiguous allocation easier, but is not the only supported lifecycle.

Transparent Huge Pages use a different mechanism. Depending on kernel support and policy, larger anonymous-memory allocations can occur on page faults or through later collapse. THP is not merely a background process and does not require the administrator to provision a hugetlb pool. Modern kernels can support multiple THP sizes; do not equate every transparent allocation with one 2 MiB mapping.

The flexibility differs too. THP can fall back to smaller pages and can be split for operations such as reclaim. An explicit hugetlb mapping does not silently become an ordinary-page mapping; an application may separately choose a fallback allocation when its configuration permits. Distinguish the kernel mapping contract from the application’s optional behavior.

Budget capacity and placement

In /proc/meminfo, HugePages_Total describes the default-size pool, HugePages_Free counts currently unallocated pages, and HugePages_Rsvd counts commitments not yet allocated at fault time. Total multiplied by page size is pool capacity, not the amount promised by Rsvd. Reservations can account for part of the apparently free pool. Hugetlb covers all supported pool sizes, while per-size sysfs directories provide finer detail.

A sparse mapping can consume more resident memory than its useful data warrants. Pool capacity also competes with ordinary memory needs. Size from application requirements, verify actual mappings, and leave room for other allocations. Benefits should be demonstrated with representative throughput, latency, and translation measurements rather than inferred from a large configured pool.

When it breaks

Growing a pool can fail because the kernel cannot find suitably contiguous physical memory. Existing free huge pages are already contiguous, so a nonzero free counter is not itself evidence of fragmentation. Wrong page size, reservations, permissions, NUMA placement, or cgroup limits can instead explain why a particular application cannot use apparently available capacity.

NUMA policy can place pages away from the CPUs using them or restrict eligible nodes. Examine per-node allocation and access locality before blaming translation overhead. A translation-cache improvement can coexist with worse memory latency, and neither a global free counter nor a successful startup establishes good placement.

THP allocation, collapse, compaction, and splitting may affect latency differently. thp_collapse_alloc counts successful allocations during collapse; it is not a stall-duration counter. Correlate appropriate counter deltas with latency. Choosing madvise, always, or another supported per-size policy requires workload evidence, not a blanket rule that every database should use the same mode.

The 8 GiB page-count comparison is arithmetic; hugetlb reservations and THP behavior require separate interpretation.

Check it quickly

Save this as hugepage_check.py and run python3 hugepage_check.py. It reads counters and page size without allocating a huge-page workload or modifying kernel settings. Missing data produces an explicitly labeled arithmetic illustration. Parser and boundary assertions use synthetic strings.

import os
import re
from pathlib import Path

WORKING_SET = 8 * 1024**3
FIELDS = ("HugePages_Total", "HugePages_Free", "HugePages_Rsvd",
 "HugePages_Surp", "Hugepagesize")


def parse_meminfo(text):
 values = {}
 for key in FIELDS:
 unit = r"\s+kB" if key == "Hugepagesize" else ""
 match = re.search(r"^" + key + r":\s+(\d+)" + unit + r"\s*$", text, re.M)
 if match:
 values[key] = int(match.group(1))
 return values


def pages_for(size, page):
 if size < 0 or page <= 0:
 raise ValueError("invalid byte count or page size")
 return (size + page - 1) // page


assert pages_for(4097, 4096) == 2
assert parse_meminfo("HugePages_Total: 8\nHugepagesize: 2048 kB\n")["Hugepagesize"] == 2048
assert parse_meminfo("HugePages_Total: bad\nHugepagesize: 2 MB\n") == {}
try:
 pages_for(1, 0)
except ValueError:
 pass
else:
 raise AssertionError("zero page size accepted")

try:
 values = parse_meminfo(Path("/proc/meminfo").read_text())
except OSError as error:
 values = {}
 print("SKIP live hugetlb counters:", type(error).__name__)
try:
 base = os.sysconf("SC_PAGE_SIZE")
 assert base > 0
 print("Observed base page size:", base, "bytes")
except (AttributeError, ValueError, OSError):
 base = 4096
 print("Assumed illustration base page size:", base, "bytes")

if all(key in values for key in FIELDS) and values["Hugepagesize"] > 0:
 huge = values["Hugepagesize"] * 1024
 print("Default-size hugetlb pool counters:", values)
 print("Total pool capacity:", values["HugePages_Total"] * huge, "bytes")
 print("Reserved but not allocated:", values["HugePages_Rsvd"] * huge, "bytes")
else:
 huge = 2 * 1024**2
 print("SKIP complete pool report; assuming 2 MiB for arithmetic only")
print("8 GiB page-count illustration:", pages_for(WORKING_SET, base),
 "base pages versus", pages_for(WORKING_SET, huge), "large pages")
print("No TLB miss rate, THP coverage, allocation success, or latency was measured")

With 4 KiB base pages and a 2 MiB default huge-page size, expect the two page counts above. Zero pool totals are valid observations, not proof that the kernel lacks support. This revised script passed with Python 3.12.14 on Linux. It measures neither THP coverage nor performance; those require a separate, workload-specific experiment.

Sources

Kernel hugepages documentation, defines hugetlb settings and behavior, checked 2026-06-01.
Kernel transparent hugepages documentation, defines THP modes and compaction behavior, checked 2026-06-01.
proc_meminfo: huge-page counters, checks commitment and huge-page counters with units, checked 2026-10-15.

Linux overcommit budgets virtual-memory commitments

Sat, 18 Dec 2027 00:00:00 +0000

Linux overcommit policy controls admission of certain virtual-memory commitments. It is separate from allocating resident physical pages, choosing an OOM victim, and enforcing a container’s memory limit. A successful allocation can precede physical demand, but that does not make all allocations free or guarantee that later accesses will succeed under every resource constraint.

Virtual commitment differs from residence

An allocator such as malloc may obtain memory through brk or mmap and may reuse an existing arena without making a new system call. Anonymous pages are commonly populated lazily. Address-space metadata, page tables, and actual page faults still consume resources; the cost depends on what the application requests and touches.

Committed_AS is an accounting quantity, not the process resident-set sum and not a measurement of current pressure. Mapping type matters: private writable mappings can require commitment even before every page becomes resident, while ordinary shared or read-only file-backed mappings have different accounting. A large virtual-size figure cannot by itself identify the application responsible for a physical-memory shortage.

Three admission policies

Mode 0 is the default heuristic policy. It rejects obvious overcommits while allowing allocations that would exceed a simple resident-memory budget. Its decision is not a promise that every accepted byte has dedicated RAM. Mode 1 skips the overcommit admission check, which can support sparse workloads, but address-space limits, mapping limits, and other allocation failures still apply.

Mode 2 enforces stricter commitment accounting. The nominal CommitLimit is derived from swap plus a configured amount of RAM: ratio-based calculation excludes hugetlb-reserved memory, while a nonzero overcommit_kbytes supplies an absolute allowance instead of the ratio. Kernel reserves and the calling context can further affect admission. The displayed difference from Committed_AS is therefore useful context rather than a universal maximum guaranteed allocation.

A chargeable allocation may fail with ENOMEM when it cannot fit that budget. This can expose capacity problems earlier, including commitments needed by fork and later permission changes. Fork accounting follows applicable mappings and copy-on-write obligations; it is not simply twice the parent’s resident set. Some applications deliberately reserve sparse address space, so changing policy can change their startup behavior.

Strict accounting is not OOM immunity

Mode 2 reduces one class of overcommit risk, but does not override cgroup memory limits, NUMA constraints, other kernel allocations, or all later memory demands. A task can still encounter allocation failure or an OOM event. Do not promise that choosing mode 2 protects a particular worker from being killed.

OOM victim selection is a separate policy using the relevant scope and eligibility rules, not random punishment and not a decision made by the CPU scheduler. The killed process need not be the task that most recently requested memory. Inspect the event’s host or cgroup context before attributing responsibility from the victim’s name alone.

Containers share the host kernel and should not be assumed to own an independent vm.overcommit_memory policy. Their memory controls add another boundary. Virtual machines have their own guest kernel accounting, while host backing and ballooning introduce additional constraints. Record which layer each metric belongs to before comparing budgets.

When it breaks

A workload can successfully reserve memory and then encounter reclaim, swap activity, or OOM when demand grows. Neither mode 0 nor mode 1 guarantees that the failure will be slow, that swap is available, or that a particular process dies. Compare demand and latency with actual reclaim and memory-pressure evidence.

Conversely, mode 2 can reject a request despite apparently free RAM because the commitment budget is already occupied. That is one possible cause of ENOMEM, not a diagnosis from the error alone. Process resource limits, excessive mapping counts, and application configuration deserve separate checks.

Large nominal headroom does not prove the ratio is too conservative, and small headroom does not automatically justify increasing it. Budget changes can admit demand that later competes for real resources. Plan from workload requirements and recovery margin, preserve configuration context, and test changes outside production before rollout.

Overcommit controls commitment admission; accepted requests still face later physical and cgroup constraints.

Check it quickly

Save this as overcommit_probe.py and run python3 overcommit_probe.py. It reads only the named proc files, prints the observed mode and accounting fields, and performs synthetic parser checks. It does not reserve memory, trigger OOM, or suggest automatic tuning.

from pathlib import Path
import re


def read_value(path):
 try:
 return Path(path).read_text(encoding="utf-8").strip()
 except OSError as error:
 print("SKIP", path, type(error).__name__)
 return None


def meminfo_kb(text):
 return {name: int(value) for name, value in re.findall(
 r"^(MemTotal|SwapTotal|CommitLimit|Committed_AS):\s+(\d+)\s+kB\s*$",
 text, re.M)}


def report(mode, info):
 names = {"0": "heuristic", "1": "always overcommit", "2": "strict accounting"}
 lines = ["mode: " + names.get(mode, "unknown or unreadable")]
 limit, committed = info.get("CommitLimit"), info.get("Committed_AS")
 if limit is not None and committed is not None:
 lines.append("nominal commit headroom: %s kB" % (limit - committed))
 else:
 lines.append("commit headroom unavailable")
 if mode == "2":
 lines.append("Commit accounting can reject chargeable requests; other limits still apply")
 else:
 lines.append("CommitLimit is not the enforced mode-2 ceiling in this mode")
 lines.append("This snapshot is not available RAM, a cgroup budget, or OOM immunity")
 return lines


fixture = meminfo_kb("CommitLimit: 100 kB\nCommitted_AS: 0 kB\n")
assert "nominal commit headroom: 100 kB" in report("2", fixture)
assert meminfo_kb("CommitLimit: bad kB\nCommitted_AS: 7 MB\n") == {}
assert "commit headroom unavailable" in report("2", {})
assert report("9", {})[0] == "mode: unknown or unreadable"
for mode in ("0", "1", "2"):
 assert "OOM immunity" in report(mode, fixture)[-1]

mode = read_value("/proc/sys/vm/overcommit_memory")
ratio = read_value("/proc/sys/vm/overcommit_ratio")
kbytes = read_value("/proc/sys/vm/overcommit_kbytes")
text = read_value("/proc/meminfo")
if mode is None or text is None:
 print("SKIP live overcommit report; synthetic checks passed")
else:
 info = meminfo_kb(text)
 print("overcommit_memory=%s overcommit_ratio=%s overcommit_kbytes=%s" % (mode, ratio, kbytes))
 print("Observed kB fields:", info)
 for line in report(mode, info):
 print(line)

Expect a live snapshot or an explicit SKIP message. Zero committed memory remains a valid value. The revised example passed with Python 3.12.14 on Linux; synthetic checks cover malformed input, absent fields, and all three modes. Reading these counters does not test real allocation rejection or recovery.

Sources

Kernel vm sysctl documentation, defines the overcommit_memory modes and defaults, checked 2026-06-01.
proc manual page, documents /proc/meminfo CommitLimit and Committed_AS accounting, checked 2026-06-01.
Kernel overcommit accounting rules, checks policy modes and charged mappings, checked 2026-10-15.
proc_meminfo: CommitLimit and Committed_AS, checks commitment and huge-page counters with units, checked 2026-10-15.
Kernel cgroup v2 memory controls, checks local memory limits and OOM boundaries, checked 2026-10-15.

systemd-tmpfiles applies creation and age policies

Fri, 10 Dec 2027 00:00:00 +0000

systemd-tmpfiles describes filesystem objects and housekeeping rules in configuration files. A rule can create a directory with a chosen mode, while an age rule can make old entries eligible for cleanup. The command must actually run the relevant operation for either effect to occur. This is useful lifecycle policy, but it is not a continuous watcher or a hard storage quota.

This tree describes the article’s specific d and x age-cleanup fixture. An x exclusion does not override explicit r or R deletion. A cleanup invocation, pathname deletion and storage release are different events; no cleanup was run for this artwork.

When it breaks

An age is not simply “time since the file was created” or “time since its contents last changed.” The documented default considers several timestamps, including access and modification, status change for files, and creation time where available. A recently changed qualifying timestamp can preserve an entry even when its mtime looks old. Rewriting timestamps for a test can therefore change the result in an unexpected direction.

--create and --clean perform different operations. Creating the configured directory does not prove that old contents have been removed. The d rule used below keeps its directory while making eligible contents subject to age cleanup. Other types have different removal semantics; do not substitute a recursive removal rule simply because both actions sound like cleaning.

An x exclusion preserves a matching directory and its contents during age-based cleanup, whereas X excludes the directory itself without the same recursive protection. These exclusions do not override explicit r or R removal rules. Read the effective configuration and operation together before deciding which paths a maintenance run can delete.

Check it quickly

Run this Python script as an administrator in a disposable Linux environment with systemd-tmpfiles. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It passes one explicit configuration file and a temporary --root, so all created objects and deletion targets belong to the exercise. The live system’s ordinary tmpfiles configuration is not used:

from pathlib import Path
from tempfile import TemporaryDirectory
import os,stat,subprocess,time

with TemporaryDirectory(prefix='xbcatm-tmpfiles-') as directory:
 root=Path(directory);config=root/'policy.conf'
 definition='d /work 0750 - - 1s -\nx /work/keep - - - - -\n'
 config.write_text(definition)
 def apply(action):
 result=subprocess.run(['systemd-tmpfiles','--root='+str(root),action,str(config)],capture_output=True,text=True,timeout=10)
 assert result.returncode==0,result.stderr
 apply('--create');work=root/'work'
 assert stat.S_IMODE(work.stat().st_mode)==0o750
 old=work/'old';old.write_text('fixture');past=time.time()-60;os.utime(old,(past,past))
 apply('--clean');assert old.exists()
 print('old atime and mtime alone do not satisfy the default age rule')
 config.write_text(definition.replace('1s','m:1s'));apply('--clean')
 assert not old.exists()
 print('explicit mtime-only age removes the same recently created file')
 config.write_text(definition);old.write_text('aged fixture')
 keep=work/'keep';keep.mkdir();(keep/'preserved').write_text('excluded fixture')
 outside=root/'outside';outside.write_text('outside cleanup tree')
 link=work/'link';link.symlink_to(outside)
 with old.open('rb') as retained:
 time.sleep(1.2)
 fresh=work/'fresh';fresh.write_text('recent fixture')
 apply('--clean')
 assert not old.exists() and not link.is_symlink()
 assert retained.read()==b'aged fixture'
 print('unlinked file remains readable through its already open descriptor')
 assert fresh.read_text()=='recent fixture'
 assert (keep/'preserved').read_text()=='excluded fixture'
 assert outside.read_text()=='outside cleanup tree'
 assert work.is_dir()
 print('aged entries removed; recent file and recursively excluded subtree remain')
 print('symlink removed without following it; target outside the cleaned tree survives')
print('all creation and cleanup occurred in a temporary alternate root')

The create operation produces /work inside the temporary root with mode 0750. A new file then receives deliberately old access and modification timestamps. Default cleanup leaves it present because other relevant timestamps are still recent. Merely backdating mtime and atime is not enough to make this file old under the complete default rule.

Changing only the age specification to m:1s makes modification time the selected file timestamp. Cleanup now removes that same recently created file. This proves that timestamp selection changes policy, not just display formatting. The script restores the default rule afterward; mtime-only aging is an intentional alternative to assess against the application’s access pattern, not an automatic improvement.

The next phase creates an ordinary file, an excluded subtree, and a symbolic link to another harmless file outside the cleaned tree. After waiting long enough for their timestamps to age, it creates a fresh file and runs cleanup. The aged ordinary entry and symlink disappear, while the recent file and excluded subtree remain. The symlink’s target is still intact, and the configured /work directory still exists.

The ordinary file is held open across cleanup. Although its pathname is gone, its existing descriptor can still read the original bytes until it is closed. Removing a directory entry and releasing the storage backing every open reference are different events. A process holding deleted files can consequently explain why a cleanup removed names without reclaiming all the expected space.

Assertions compare actual modes, path existence, and file contents. No test treats a zero exit status alone as proof that the intended deletion happened. The entire alternate root is removed afterward, leaving the host’s existing temporary files and normal policy untouched.

Match housekeeping to the producer

Inspect the distribution’s tmpfiles setup and cleanup units to establish when these rules are invoked. Eligibility at one second does not mean deletion occurs exactly one second later; the cleanup cadence and current workload matter. An inactive timer or failed cleanup command can leave eligible files in place indefinitely.

Age policies also cannot bound how many bytes a producer creates between cleanup runs. High write volume, recent accesses, exclusions, and open descriptors all affect retained space. Use appropriate storage limits and disk monitoring when capacity must be bounded, and diagnose the producing application instead of shortening retention until useful evidence disappears.

Keep service-owned runtime, cache, and persistent state conceptually separate. Data needed after restart should not accidentally inherit a temporary-file deletion policy. When managed service directories already express the required ownership and lifetime, reuse that mechanism. Add tmpfiles rules for a concrete lifecycle requirement and verify both eligible deletion and required preservation.

Sources

systemd 252 tmpfiles.d, creation types, timestamp-based aging, exclusions, and symbolic links, checked 2026-09-12.
systemd 252 systemd-tmpfiles, separate create/clean operations, alternate roots, and scheduled integration, checked 2026-09-12.

A systemd oneshot completes one activation

Sun, 28 Nov 2027 00:00:00 +0000

A Type=oneshot service represents finite work whose startup completes when its start commands finish successfully. That makes it useful for setup steps and scheduled jobs whose completion matters to a dependent unit. The word oneshot describes one activation’s process model. It does not create a permanent record that prevents the task from running again after a restart, another activation, or a reboot.

A oneshot completes one activation. Its earlier side effects survive later-command failure, and retained active/exited state can have no live main process. The three-command and PID 0 values illustrate the article’s fixture; no service was executed for this artwork.

When it breaks

A successful oneshot normally becomes inactive when its commands finish. That is expected behavior for a completed task, not evidence that a daemon crashed. With RemainAfterExit=yes, the manager instead keeps the unit active after the commands exit. An active (exited) state can therefore have no main process at all. Monitoring that assumes every active service needs a live PID will misclassify this service type.

Multiple ExecStart entries run in sequence for a oneshot. Under ordinary success rules, a failing command prevents later start commands from running and marks the activation failed. Earlier commands may already have changed files or databases. systemd does not roll those effects back, so retrying the activation can repeat work that succeeded before the failure.

RemainAfterExit is manager state, not a durable exactly-once guarantee. Starting an already active unit does not rerun it, but restarting it does. A timer targeting an active oneshot also does not turn each elapsed interval into another execution. Choose the retained state only when it represents a real ongoing condition, and keep repeatable scheduled work inactive between runs.

Check it quickly

Run this Python script as an administrator in a disposable Linux environment managed by systemd. It creates one runtime unit and a temporary step program. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. Ensure the example unit filename is unused. Each step records its name before the second step deliberately fails:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess

unit='xbcatm-oneshot-lab.service';path=Path('/run/systemd/system')/unit
assert not path.exists()
def ctl(*args,success=True):
 result=subprocess.run(['systemctl',*args],text=True,capture_output=True,timeout=10)
 assert (result.returncode==0)==success,(args,result.stderr)
 return result.stdout.strip()
def value(key):return ctl('show',unit,'-p',key,'--value')
with TemporaryDirectory(prefix='xbcatm-oneshot-',dir='/run') as directory:
 root=Path(directory);worker=root/'step.py';calls=root/'calls'
 worker.write_text("""from pathlib import Path
import sys
root=Path(__file__).parent
with (root/'calls').open('a') as output: output.write(sys.argv[1]+'\\n')
if sys.argv[1]=='second' and not (root/'healthy').exists(): sys.exit(7)
""")
 definition=f'''[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutStartSec=3s
ExecStart=/usr/bin/python3 {worker} first
ExecStart=/usr/bin/python3 {worker} second
ExecStart=/usr/bin/python3 {worker} third
'''
 path.write_text(definition)
 try:
 subprocess.run(['systemd-analyze','verify',str(path)],check=True)
 ctl('daemon-reload');ctl('start',unit,success=False)
 assert calls.read_text().splitlines()==['first','second']
 assert value('Result')=='exit-code' and value('ExecMainStatus')=='7'
 print('second command fails; third command does not run')
 (root/'healthy').touch();ctl('reset-failed',unit);ctl('start',unit)
 assert calls.read_text().splitlines()==['first','second','first','second','third']
 assert (value('ActiveState'),value('SubState'),value('MainPID'))==('active','exited','0')
 before=calls.read_text();ctl('start',unit);assert calls.read_text()==before
 print('active/exited with PID 0; repeated start does not rerun')
 ctl('restart',unit)
 assert calls.read_text()==before+'first\nsecond\nthird\n'
 print('restart executes all commands again')
 ctl('stop',unit);path.write_text(definition.replace('RemainAfterExit=yes','RemainAfterExit=no'))
 ctl('daemon-reload');before=calls.read_text()
 for _ in range(2):
 ctl('start',unit)
 assert value('ActiveState')=='inactive' and value('Result')=='success'
 assert calls.read_text()==before+'first\nsecond\nthird\n'*2
 print('without RemainAfterExit: two starts, two successful executions')
 finally:
 ctl('stop',unit)
 if value('ActiveState')=='failed':ctl('reset-failed',unit)
 path.unlink();ctl('daemon-reload')

The initial activation records only first and second, returns a failing start command, and exposes exit status seven. The third command does not run. The existing log entries remain, which makes the partial effects visible instead of pretending that a failed activation left no work behind.

After the healthy marker is created, the retry records all three steps. The complete file now contains the two earlier entries followed by a fresh sequence of three. State becomes active and exited, with main PID zero. That combination means the manager retains a successful activation, not that the program remains running in the background.

A second start while active leaves the file unchanged. Restarting the same unit appends another complete sequence. This is the direct counterexample to interpreting oneshot as “execute this migration once for the lifetime of the machine.” The manager is doing exactly what the new activation requests.

The script then stops the unit, changes RemainAfterExit to no, reloads the manager, and starts it twice. Both executions succeed and both end inactive. The log gains two full sequences. This behavior fits a job that should run whenever triggered and complete before another trigger can start it again.

The unit has an explicit three-second startup timeout. Oneshot startup timeouts are disabled by default in the documented version, so a stuck command otherwise may hold an activation open indefinitely. Choose a realistic bound for actual work rather than copying this small laboratory timeout into a long migration.

Give the task its own correctness guarantees

Keep each start command finite and in the foreground. If the task launches a long-running worker, describe that worker with an appropriate service lifecycle instead of inferring continued operation from the parent’s successful exit. Unit type, command completion, and the business result should describe the same event.

For a migration, use the database’s migration ledger, transaction behavior, and concurrency controls to decide whether a change has already been applied. A retained active state in one manager cannot coordinate multiple hosts or survive every operational transition. Make retries safe at the layer where side effects occur, and preserve the result needed to diagnose partial completion.

Inspect the service’s recorded result and application output, not just whether a timer or dependency triggered it. A timer can be active while its last job failed. Separate installation, scheduling, and execution guarantees, and verify a failed path as well as the successful one before depending on the task during boot or release.

Sources

systemd 252 service types, oneshot command ordering, retained state, and startup timeouts, checked 2026-09-12.
systemd 252 systemctl, start, restart, and manager reload behavior, checked 2026-09-12.
systemd 252 timers, activation of an already active target service, checked 2026-09-12.

A veth pair carries frames between two endpoints; routing supplies their meaning

Sun, 21 Nov 2027 00:00:00 +0000

A veth pair consists of two interconnected virtual Ethernet devices. A frame transmitted through one is delivered to the other, where the receiving network stack or bridge handles it. The two ends can reside in separate Linux network namespaces, making the pair useful for container networking. Creating the pair supplies a link; addresses, routes, bridge membership, forwarding, and access policy are additional configuration.

When it breaks

An interface marked administratively UP can still lack carrier because its veth peer is down. Reading only the UP flag can therefore make an incomplete setup look ready. Inspect both endpoints and distinguish administrative state from LOWER_UP. A route may already name the interface even while the paired link cannot carry useful traffic, so a route listing alone is not a connectivity check.

The pair does not restrict IP payloads to the peer interface’s own address. Routing chooses which link should carry a packet, and Ethernet forwarding or the receiving stack decides what happens next. Traffic destined beyond a container’s immediate neighbor can traverse the pair toward a router or bridge. Saying that every packet not addressed to the peer stays local confuses link delivery with IP destination selection.

Endpoint lifetime is coupled too. Deleting one veth endpoint removes its peer, even if the peer is managed from another namespace. A cleanup action in the host-side context can therefore remove a container’s interface. Stopping traffic, removing routes, and deleting the pair need an intentional order when they affect a live workload; namespace separation does not make the two devices independent resources.

veth: one pair, two namespaces: conceptual mechanisms and expected checks, not new runtime evidence.

Check it quickly

Save as check-veth.py for an authorized disposable Linux lab using a verified local Docker daemon and enabled Python assertions. The recipe names Python 3.11.16 and iproute2 6.1.0; image architecture, packages and kernel need verification. NET_ADMIN/SYS_ADMIN are insufficient when mandatory policy denies mounts, as Docker’s default AppArmor profile does. Use a compatible approved lab, not weakened host protections. The nested namespaces and proc mount require actual privileged kernel operations. Runtime network isolation does not make image pulls or builds offline; cleanup commands are attempted, not checked.

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found

def server(stack,name,address,port=18080,label='server'):
 code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
 address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
 def server_bind(self):
 if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
 super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
 assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
 code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
 client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
 assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
 return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

with lab(['left','right']) as (root,stack,setting):
 run('ip','link','add','left0','type','veth','peer','name','right0')
 run('ip','link','set','left0','netns','left');run('ip','link','set','right0','netns','right')
 ip('left','addr','add','192.0.2.1/30','dev','left0');ip('right','addr','add','192.0.2.2/30','dev','right0')
 ip('left','link','set','left0','up')
 before=json.loads(ip('left','-j','link','show','left0').stdout)[0]
 assert 'LOWER_UP' not in before['flags']
 failed=ns('left','ping','-n','-c','1','-W','1','192.0.2.2',check=False);assert failed.returncode!=0
 ip('right','link','set','right0','up')
 after={name:json.loads(ip(name,'-j','link','show',interface).stdout)[0] for name,interface in [('left','left0'),('right','right0')]}
 assert all('LOWER_UP' in v['flags'] for v in after.values())
 ip('left','neigh','flush','dev','left0');ip('right','neigh','flush','dev','right0')
 successful=ns('left','ping','-n','-c','1','-W','1','192.0.2.2');assert successful.returncode==0
 daemon=server(stack,'right','192.0.2.2',label='right');response=request('left','192.0.2.2');assert response['ok']
 ip('right','addr','add','203.0.113.9/32','dev','lo');ip('left','route','add','203.0.113.9/32','via','192.0.2.2')
 server(stack,'right','203.0.113.9',label='beyond-link-address');routed_response=request('left','203.0.113.9');assert routed_response['ok']
 ip('left','link','delete','left0');gone=ip('right','link','show','right0',check=False);assert gone.returncode!=0 and 'does not exist' in gone.stderr
 print(json.dumps({'one_end_up_flags':before['flags'],'one_end_up_ping_exit':failed.returncode,
 'both_up_flags':{n:v['flags'] for n,v in after.items()},'both_up_ping_exit':successful.returncode,'http':response,'other_ip_destination':routed_response,
 'deleting_one_removes_peer':True,'peer_absent_diagnostic':gone.stderr.strip()}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The expected sequence starts with only left0 up: UP without LOWER_UP, a down right0, and failed ping. Then both ends should report LOWER_UP. These are intended real Linux observations, not model outputs or measurements made in this review.

After bringing right0 up, the script clears neighbor entries left by the failed attempt before testing ping and HTTP to 192.0.2.2. It prints the returned identity but asserts only the client’s ok flag, not the exact identity field. Clearing stale state does not promise instantaneous recovery for requests already waiting.

Next, the right namespace adds 203.0.113.9/32 on its loopback interface. The left namespace installs a host route to that address through 192.0.2.2. A second HTTP listener bound to 203.0.113.9 responds successfully across the same veth pair. The packet’s IP destination is not the peer’s link address, yet its selected path still uses that virtual Ethernet link.

This second destination is local to the right namespace, so the fixture does not claim to forward onward through another router. It isolates the distinction between a link’s neighboring address and a packet’s final IP destination. A larger topology can connect the receiving endpoint to a bridge or routed network, but those additional components need their own configuration and verification.

Finally, deleting left0 makes ip link show right0 fail in the right namespace with a device-not-found diagnostic. The peer has been removed, not merely placed administratively down. Both namespaces themselves still exist until cleanup. This separates the paired device lifecycle from the lifetime of the namespace that happened to contain one endpoint.

Diagnose the layers in order

Confirm which namespace owns each end and which names the runtime assigned. Then inspect administrative and carrier state, address configuration, neighbor resolution, and the route for the failing destination. A stale neighbor entry can delay recovery after a link change, while an absent route can fail before any frame is transmitted. These conditions require different repairs even when a high-level client reports only a connection timeout.

A pair is not a multiport switch. A bridge can join several such links at Ethernet level, while a router forwards between IP networks. Enabling an endpoint does not automatically add either behavior or an internet route. Document the actual attachment and filtering points, especially when a container runtime creates the devices and another component manages routes or firewall policy.

Treat deletion as affecting both endpoints and verify the workload afterward. Exact runtime evidence for this fence was not recovered; no packet exchange or privileged setup ran in this review. Its expected sequence cannot establish throughput, latency, offload behavior or a complete container network. Any future execution needs source binding and verified cleanup.

Sources

Linux veth(4), paired transmission, carrier state and namespace use, checked 2026-09-12.
Linux ip-link(8), veth creation, link state, movement and deletion, checked 2026-09-12.
Linux network_namespaces(7), network boundaries and virtual-device lifetime, checked 2026-09-12.
Moby default AppArmor template, explicit mount denial in the default profile, checked 2026-10-18.
Docker AppArmor, mandatory policy and mount restrictions, checked 2026-10-18.
Docker run reference, capabilities and capability-aware default seccomp policy, checked 2026-10-18.
Python 3.14: assert statements, optimization removes assertion checks, checked 2026-10-18.

A page fault is not always a storage read

Sat, 13 Nov 2027 00:00:00 +0000

A page fault means the processor could not complete an access using the current mapping or permissions. It is not synonymous with a storage read, and some faults represent invalid accesses rather than recoverable demand paging. Linux’s minor and major counters distinguish whether handling required page-in I/O. Interpret those counters alongside the mapping type and workload; neither is a direct measurement of all application storage traffic.

Minor does not mean free

A minor fault can map an already resident file page, allocate and zero anonymous memory, or copy a page for a copy-on-write mapping. The kernel may need memory allocation, page-table work, copying, and synchronization. Describing it as only adding a pointer misses those costs. A sufficiently intense minor-fault workload can consume meaningful CPU time and affect latency even without storage reads.

The first application access does not necessarily produce a fresh fault for each base page. Prefaulting, existing mappings, readahead-related mapping behavior, and large pages can change the count. The operating system’s page size is not universally 4096 bytes. An example should obtain the local value and avoid claiming an exact fault count from the number of loop iterations.

Major describes a different path

A major fault indicates handling involving page-in I/O, often for file-backed data or swapped-out memory. Storage can include devices or remote filesystems; the label is not a physical-disk latency benchmark. One fault can initiate readahead that serves later accesses, so a count is not a one-to-one count of pages or disk requests. Conversely, ordinary file reads can cause I/O without producing corresponding major faults in the application.

File-cache residency can allow a new process to incur minor faults even when it has never touched that file. A process restart is not equivalent to dropping the host’s page cache. Anonymous first-write faults commonly allocate zero-filled memory without reading old payloads from storage, but pressure and concurrent activity can still affect measurements. “Anonymous” is not a universal promise that the entire process’s major counter remains unchanged.

When it breaks

A cold-start latency spike can correlate with major faults, but do not prescribe warming before establishing what is missing. Pre-touching unnecessary pages consumes memory and can displace useful cache. Staggered starts may help a measured shared bottleneck; another workload may instead need smaller mappings, a different access pattern, or more memory headroom. Fault counts alone do not choose among these remedies.

Sustained swapping can repeatedly return evicted pages to the working set and increase latency. Adding more workers can worsen that situation by increasing memory demand. Correlate faults with swap activity, reclaim, storage waits, and request latency. A rising major count is evidence to investigate, not a verdict that the storage device is faulty or that every cache miss was avoidable.

Per-process counters also include work outside the line of code being measured. Interpreter activity, startup, and other threads can contribute. A residency probe is only a snapshot, and access immediately afterward can encounter a different state. Avoid dropping global caches or increasing a test allocation until it threatens unrelated work just to obtain a dramatic graph.

Minor faults, major faults and invalid accesses follow different paths; the anonymous-memory example does not exercise every path.

Check it quickly

Save this as fault_probe.py and run python3 fault_probe.py on Linux with Python 3.9 or newer and sufficient spare memory. It writes one byte per native page through a private anonymous mapping of sixteen MiB, then repeats the pass. The mapping is closed automatically, and no file, global cache setting, or memory limit is changed.

import mmap
import resource
import sys
import time

if not sys.platform.startswith("linux"):
 raise SystemExit("This exercise requires Linux getrusage semantics")
PAGE = mmap.PAGESIZE
SIZE = 16 * 1024 * 1024

def counters():
 value = resource.getrusage(resource.RUSAGE_SELF)
 return value.ru_minflt, value.ru_majflt

def touch(region):
 for offset in range(0, len(region), PAGE):
 region[offset] = 1

with mmap.mmap(-1, SIZE, flags=mmap.MAP_PRIVATE | mmap.MAP_ANONYMOUS,
 prot=mmap.PROT_READ | mmap.PROT_WRITE) as region:
 before = counters()
 start = time.perf_counter()
 touch(region)
 first_seconds = time.perf_counter() - start
 middle = counters()
 start = time.perf_counter()
 touch(region)
 second_seconds = time.perf_counter() - start
 after = counters()
first = tuple(b - a for a, b in zip(before, middle))
second = tuple(b - a for a, b in zip(middle, after))
print(f"page_size={PAGE} mapping_bytes={SIZE}")
print(f"first: minor={first[0]} major={first[1]} seconds={first_seconds:.6f}")
print(f"second: minor={second[0]} major={second[1]} seconds={second_seconds:.6f}")
assert first[0] > 0, "expected demand-fault activity"
assert second[0] < first[0], "repeat pass did not show reduced minor faults"
assert first[1] == second[1] == 0, "observed major faults: this run is not a no-major example"
print("PASS: this run had fewer second-pass minor faults and no measured major faults")

The test expects demand-fault activity on the first pass, fewer minor faults on the second, and no measured majors in this particular run. Those assertions passed for both exact language examples during this review. Counts and durations are observations, not fixed targets. The script does not assert that the second pass is faster; timers and scheduling can vary.

A failed assertion means this run did not establish the expected contrast. Inspect its memory conditions and counters rather than silently deleting the check. The successful result shows one anonymous-memory case where fault activity occurred without measured page-in I/O. It does not demonstrate a file-backed major fault, establish universal fault latency, or replace measurements of the real application under representative pressure.

Sources

Linux getrusage, minor and major fault accounting, checked 2026-10-15.
Linux mmap, private anonymous mappings and population behavior, checked 2026-10-15.
Linux memory concepts, demand paging, page cache, reclaim and huge pages, checked 2026-10-15.
Linux process stat, process fault counters and their scope, checked 2026-10-15.

Socket activation passes a listener to a compatible service

Mon, 08 Nov 2027 00:00:00 +0000

Socket activation lets systemd open a listening endpoint before the application process starts. Activity on that endpoint can then trigger a compatible service, which receives descriptors for sockets the manager already owns. The useful boundary is listener ownership versus application lifetime. An open endpoint is not yet proof that the application has initialized, accepted a request, or produced a correct response.

The active socket unit precedes a service with MainPID 0. A first connection starts a worker that validates LISTEN_PID, LISTEN_FDS 1 and listening FD 3, handles a framed request and replies. The worker deliberately exits; the manager retains the listener. A later connection starts a different worker using FD 3. Rebinding would get EADDRINUSE. Stopping the socket removes the path only because this fixture sets RemoveOnStop=yes. Illustration of selected source mechanisms; no example executed in this round.

When it breaks

The application must understand the activation protocol. A daemon that unconditionally binds the same endpoint again can fail because the manager already occupies it. Activation moves binding responsibility into the socket unit; it does not eliminate address conflicts, missing interfaces, filesystem permission errors, or other reasons that initial bind can fail.

With Accept=no, the service receives the listening socket and handles acceptance itself. Accept=yes instead supports a per-connection service-instance model for suitable stream sockets. These are different application contracts. Neither mode implies that an arbitrary daemon will automatically become idle, exit after work, or restart only when doing so is safe.

Queued connections also have limits. Clients can establish transport state before application startup finishes, then time out waiting for a response. Socket activation does not make database dependencies ready or create an unlimited request queue. Keep explicit service dependencies, readiness behavior, and client timeouts appropriate to the real workload.

Check it quickly

Run this Python script as an administrator in a disposable Linux system whose PID 1 is systemd. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It creates a unique socket/service pair using a Unix stream socket in a temporary directory. No public TCP port is exposed. The worker intentionally serves one newline-terminated message and exits:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,socket,subprocess,time,uuid

name='xbcatm-socket-'+uuid.uuid4().hex[:8];service=name+'.service';unit=name+'.socket'
paths={n:Path('/run/systemd/system')/n for n in (service,unit)}
assert all(not p.exists() for p in paths.values())
def run(*args):
 result=subprocess.run(args,capture_output=True,text=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def value(key):return run('systemctl','show',service,'-p',key,'--value')
def wait_inactive():
 deadline=time.monotonic()+5
 while value('ActiveState')!='inactive':
 assert time.monotonic()<deadline,'service did not exit';time.sleep(0.02)
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);endpoint=root/'listener.sock';worker=root/'worker.py'
 worker.write_text(r'''from pathlib import Path
import errno,json,os,socket
root=Path(__file__).parent
assert int(os.environ['LISTEN_PID'])==os.getpid()
assert int(os.environ['LISTEN_FDS'])==1
with socket.socket(fileno=3) as listener:
 assert listener.getsockopt(socket.SOL_SOCKET,socket.SO_ACCEPTCONN)==1
 with socket.socket(socket.AF_UNIX,socket.SOCK_STREAM) as duplicate:
 try:duplicate.bind(str(root/'listener.sock'))
 except OSError as error:assert error.errno==errno.EADDRINUSE
 else:raise AssertionError('unexpected duplicate bind success')
 connection,_=listener.accept()
 with connection,connection.makefile('rb') as incoming:
 message=incoming.readline(256);assert message.endswith(b'\n')
 connection.sendall(b'ack '+message)
 (root/(str(os.getpid())+'.json')).write_text(json.dumps({'pid':os.getpid(),'fd':3,'message':message.decode().strip()}))
''')
 paths[unit].write_text(f'[Socket]\nListenStream={endpoint}\nAccept=no\nSocketMode=0600\nRemoveOnStop=yes\n')
 paths[service].write_text(f'[Service]\nType=exec\nExecStart=/usr/bin/python3 {worker}\n')
 try:
 run('systemd-analyze','verify',*[str(p) for p in paths.values()]);run('systemctl','daemon-reload')
 run('systemctl','start',unit)
 assert endpoint.is_socket() and value('ActiveState')=='inactive' and value('MainPID')=='0'
 print('socket listening while service has no process')
 for message in ('first','second'):
 with socket.socket(socket.AF_UNIX,socket.SOCK_STREAM) as client:
 client.settimeout(5);client.connect(str(endpoint));client.sendall((message+'\n').encode())
 with client.makefile('rb') as response:assert response.readline()==('ack '+message+'\n').encode()
 wait_inactive();assert endpoint.is_socket() and value('Result')=='success'
 records=[json.loads(p.read_text()) for p in root.glob('*.json')]
 assert len(records)==2 and len({row['pid'] for row in records})==2
 assert {row['message'] for row in records}=={'first','second'}
 assert all(row['fd']==3 for row in records)
 print('two requests: two deliberate one-request workers; each uses inherited listening fd 3')
 print('rebinding the occupied socket fails; manager retains listener between workers')
 run('systemctl','stop',unit);assert not endpoint.exists()
 print('stopping socket removes its endpoint with RemoveOnStop=yes')
 finally:
 run('systemctl','stop',unit,service)
 for name,path in paths.items():
 if run('systemctl','show',name,'-p','ActiveState','--value')=='failed':run('systemctl','reset-failed',name)
 path.unlink()
 run('systemctl','daemon-reload')

After starting only the socket unit, the endpoint exists while the corresponding service is inactive and has no main PID. A first client connection then causes the worker to execute. The client sends a complete framed message and checks an application reply, rather than treating a successful connect call as proof of usable service.

The worker verifies that LISTEN_PID names its own process and LISTEN_FDS reports one inherited descriptor. It wraps descriptor three and checks that it is already listening. The numbering and environment checks match the native activation contract. A production program handling multiple descriptors should also identify and validate the sockets it actually expects.

Attempting another bind to the same Unix socket path produces EADDRINUSE. The worker continues using the inherited listener and returns the expected reply. This controlled failure demonstrates why an application cannot simply retain its ordinary bind path while assuming activation will somehow avoid the conflict.

After the worker exits successfully, the manager still holds the listening endpoint. A second connection launches another worker, which handles the second message and exits. The recorded PIDs are different and both invocations used descriptor three. Two processes appear because this specific worker deliberately exits after one request; Accept=no did not itself require one process per connection.

Stopping the socket unit removes the endpoint because the fixture explicitly sets RemoveOnStop=yes. The script then stops any remaining test units and removes their runtime definitions and temporary files. This cleanup behavior is configured and asserted by the fixture, not a universal promise that every socket path disappears whenever any associated daemon exits.

Design the handoff and shutdown together

For a long-running daemon, accept and serve the inherited sockets using its documented integration, and let its normal shutdown behavior decide when descriptors close. For a daemon intended to leave when idle, verify the idle-exit logic and what happens to in-flight requests. systemd cannot infer when an application’s work is safely complete.

Stopping future activation and terminating current work are separate operational decisions. Inspect both socket and service units during maintenance. A service may be started explicitly through another dependency or by an operator, and a still-active listener may trigger it again. Test the deployed unit relationships rather than assuming that stopping one visible process permanently disables the endpoint.

Set appropriate endpoint permissions and service identity. Passing an already-open descriptor allows listener setup and application privileges to differ, but it does not make the service a sandbox. Verify real request handling, restart behavior, queued-client outcomes, and failure diagnostics. The practical advantage is a deliberate, testable handoff between listener and process, not automatic readiness or zero resource cost.

Sources

systemd 252 systemd.socket, listener ownership, Accept modes, activation, and endpoint removal, checked 2026-09-12.
systemd 252 sd_listen_fds, inherited descriptor numbering, count, and process-identity checks, checked 2026-09-12.
systemd 252 systemd.service, service startup types and process lifecycle, checked 2026-09-12.

systemd restart policies need explicit failure and rate rules

Sun, 07 Nov 2027 00:00:00 +0000

A restart policy decides whether a service manager should try another process after the current one ends. It can recover from a transient failure, but it cannot correct a missing configuration or a deterministic bug. The useful contract has three parts: which endings qualify, how long to wait, and how many starts are permitted within a time window. Those settings answer different questions.

First classify the ending, then apply the fixed delay and start-window limit. The 100 ms, ten-second and three-start values come from the article’s systemd 252.39 fixture and are schematic, not a fresh run or recommended production values. Read the failure and repair its cause before resetting the limiter.

When it breaks

Restart=on-failure does not mean restart after every signal. For service types other than oneshot, ordinary SIGHUP, SIGINT, SIGTERM, and SIGPIPE terminations are normally considered clean. A nonzero exit or an unclean signal such as SIGKILL qualifies under the default success rules. SuccessExitStatus and the restart-specific exception directives can change classification, so inspect the effective unit.

on-abnormal is not simply a stricter synonym for on-failure: it excludes ordinary nonzero process exit codes while covering abnormal signals and certain timeout or watchdog failures. Choose according to how the program reports recoverable and intentional endings. A process that remains alive but stops doing useful work needs a separately configured detection mechanism; exit-based restart logic cannot infer that condition.

RestartSec supplies a delay before restarting. In the systemd 252 exercise below, it is a fixed delay, not an exponentially increasing backoff. StartLimitIntervalSec and StartLimitBurst limit starts within an interval, including manual starts. They are not a lifetime retry counter, and reaching the limit does not schedule an automatic recovery attempt at the next interval boundary.

Check it quickly

Run this Python script as an administrator in a disposable Linux systemd environment. It creates a uniquely named runtime unit and a temporary worker, then removes them. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. The worker initially exits with status seven and records every actual invocation:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,time,uuid

unit='xbcatm-restart-lab-'+uuid.uuid4().hex[:8]+'.service'
path=Path('/run/systemd/system')/unit
assert not path.exists()
def ctl(*args):
 result=subprocess.run(['systemctl',*args],text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def value(key):return ctl('show',unit,'-p',key,'--value')
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'state transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix='xbcatm-restart-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py';calls=root/'calls';healthy=root/'healthy'
 worker.write_text("""from pathlib import Path
import sys,time
root=Path(__file__).parent
with (root/'calls').open('a') as output: output.write('started\\n')
if not (root/'healthy').exists(): sys.exit(7)
time.sleep(60)
""")
 path.write_text(f'''[Unit]
StartLimitIntervalSec=10s
StartLimitBurst=3
[Service]
Type=exec
ExecStart=/usr/bin/python3 {worker}
Restart=on-failure
RestartSec=100ms
''')
 try:
 ctl('daemon-reload');ctl('start',unit)
 wait_for(lambda: value('ActiveState')=='failed')
 assert calls.read_text().splitlines()==['started']*3
 assert value('Result')=='exit-code'
 subprocess.run(['journalctl','--sync'],check=True)
 logs=subprocess.run(['journalctl','-u',unit,'--no-pager','-o','cat'],capture_output=True,text=True,check=True).stdout
 assert 'Start request repeated too quickly' in logs
 print('three starts; failed/exit-code with start-rate-limit journal message')
 healthy.touch();ctl('reset-failed',unit);ctl('start',unit)
 wait_for(lambda: len(calls.read_text().splitlines())==4)
 first=value('MainPID');assert first!='0'
 ctl('kill','--kill-who=main','--signal=TERM',unit)
 wait_for(lambda: value('ActiveState')=='inactive')
 assert value('NRestarts')=='0'
 print('on-failure treats ordinary SIGTERM as clean; no restart')
 ctl('start',unit);first=value('MainPID')
 ctl('kill','--kill-who=main','--signal=KILL',unit)
 wait_for(lambda: value('MainPID') not in ('0',first))
 assert value('NRestarts')=='1'
 print('SIGKILL triggers one restart')
 ctl('stop',unit);time.sleep(0.2)
 assert value('ActiveState')=='inactive'
 print('explicit stop remains stopped')
 finally:
 ctl('stop',unit)
 if value('ActiveState')=='failed':ctl('reset-failed',unit)
 path.unlink();ctl('daemon-reload')

The first phase permits three starts in ten seconds with a short fixed restart delay. The worker’s own file contains exactly three entries, so the count represents actual process launches rather than a counter whose meaning was assumed. The manager then leaves the unit failed and records that the start request was repeated too quickly.

The retained fixture expects terminal Result=exit-code alongside a start-limit journal message. The v252.39 source preserves an earlier non-success result in service_enter_dead; this supports that possibility, but does not reproduce the reported run. Use state, application invocation evidence, and the corresponding journal message together rather than relying on one expected diagnostic label across versions and failure paths.

The script repairs the fixture by creating its healthy marker, clears the failed state and rate counter, and starts it again. This time the process remains alive. Sending an ordinary SIGTERM produces an inactive unit with no automatic restart under on-failure. A later SIGKILL does cause one replacement process.

Finally, an explicit systemctl stop leaves the service inactive after the restart delay. Manager-requested shutdown is not treated as a crash that policy should immediately reverse. The cleanup handles failed state only when present, since an inactive unreferenced unit can be unloaded by the manager.

Recover the service, not just its counter

Read the failure before resetting its evidence. Inspect the configured command, exit status, recent invocation logs, and any dependency errors. Correct the cause or deploy a known working version, then reset the limiter if an immediate retry is justified. Repeatedly clearing the counter without changing the failure simply converts a bounded crash loop into an operator-driven one.

Alert on sustained unavailability and excessive restarts using the monitoring system actually responsible for the service. The unit settings shown here do not create an alert destination. Retain enough diagnostic context to distinguish failed execution, an application exit, resource exhaustion, and a deliberate stop without logging sensitive configuration.

Select the delay and burst to match startup cost and dependency behavior. A rapid retry may recover quickly from one fault but amplify load during a shared outage. Verify both recovery and termination in the deployed configuration, including any drop-ins. A policy is useful when the application resumes useful work and remains controllable, not merely when a PID repeatedly changes.

Sources

systemd 252 service restart rules, exit classification, fixed delay, and restart exceptions, checked 2026-09-12.
systemd 252 unit start limits, interval limits, manual starts, and counter reset, checked 2026-09-12.
systemd 252 systemctl, stop, state inspection, and reset-failed operations, checked 2026-09-12.
systemd-stable v252.39 service source, service_enter_dead retains an earlier non-success result; source evidence only, checked 2026-10-19.

Read procfs as documented views of a changing process

Fri, 29 Oct 2027 00:00:00 +0000

Linux procfs exposes process and kernel information through filesystem interfaces. Many entries generate their contents when read instead of storing ordinary disk-file bytes. Each entry has a documented meaning, permission model, and format. Reading several entries does not freeze the process or create a transactionally consistent snapshot, so useful observations must be tied to the right process instance and interpretation.

A successful readlink does not guarantee a successful open through that link. Linux separately checks the referenced inode’s permissions. Conversely, inspecting a descriptor does not consume its owner’s data merely by reading the link text. Distinguish inspecting metadata from opening and reading the referenced object, especially for pipes and other stateful streams.

Each procfs entry is a documented, permission-checked view. Read times, process identity, entry-specific parsing and object lifetime determine what the observation can establish.

When it breaks

Ordinary file assumptions can fail immediately. A proc entry can report size zero yet return useful content when read. A per-process directory can disappear as the process exits, and a numerical PID can eventually identify another instance. Treating a failed read as proof of a broken installation, or assuming a PID forever names one workload, can misdiagnose normal lifecycle changes.

Formats require entry-specific parsing. status contains labeled fields, while stat uses a positional layout with a parenthesized command name that can contain spaces and closing parentheses. Splitting the whole stat line on whitespace can shift the expected fields. cmdline and environ use NUL separators, so ordinary line or space splitting loses argument and value boundaries.

The views also have semantic limits. environ describes the environment region established at program execution, not a guaranteed live dictionary of every later environment change. Command-line contents can be changed by the process. A descriptor link ending in (deleted) does not mean the open object has vanished. These entries are evidence about specific state, not interchangeable proofs of identity or access.

Check it quickly

Save this as procview.py and run python3 procview.py in a disposable Linux environment. The 2026-10-15 review recorded this fixture passing on Linux 6.18.44 and Python 3.12.14; it was not rerun here. The script creates one controlled child with a small, explicitly supplied environment, reads only its own and that child’s proc entries, and performs no writes to kernel tuning interfaces:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,subprocess,sys

worker=r'''
from pathlib import Path
import ctypes,json,os,sys
assert ctypes.CDLL(None).prctl(15,b'worker ) sample',0,0,0)==0
os.environ['XB_PROC_TEST']='changed-after-start'
path=Path(sys.argv[1]);path.write_bytes(b'fixture-data')
with path.open('rb') as handle:
 info=os.fstat(handle.fileno());path.unlink()
 print(json.dumps({'fd':handle.fileno(),'device':info.st_dev,'inode':info.st_ino,'current_environment':os.environ['XB_PROC_TEST']}),flush=True)
 assert sys.stdin.readline()=='done\n'
'''
with TemporaryDirectory(prefix='xbcatm-proc-') as directory:
 path=Path(directory)/'data'
 process=subprocess.Popen([sys.executable,'-c',worker,str(path),'argument with spaces'],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True,env={'PATH':'/usr/bin:/bin','XB_PROC_TEST':'fixture-value'})
 try:
 metadata=json.loads(process.stdout.readline());root=Path('/proc')/str(process.pid)
 status_path=root/'status';status_text=status_path.read_text()
 assert status_path.stat().st_size==0 and len(status_text)>0
 status=dict(line.split(':',1) for line in status_text.splitlines() if ':' in line)
 assert int(status['Pid'])==process.pid and int(status['PPid'])==os.getpid()
 raw=(root/'stat').read_text();pid_text,rest=raw.split(' (',1);comm,fields=rest.rsplit(') ',1)
 fields=fields.split();assert int(pid_text)==process.pid and comm=='worker ) sample'
 assert int(fields[1])==os.getpid() and int(fields[19])>0
 assert raw.split()[2] not in ('R','S','D','T','Z')
 print('proc files can have size zero but readable data; stat comm can contain spaces and a closing parenthesis')
 args=(root/'cmdline').read_bytes().rstrip(b'\0').split(b'\0')
 environment=(root/'environ').read_bytes().rstrip(b'\0').split(b'\0')
 assert args[-1]==b'argument with spaces' and b'XB_PROC_TEST=fixture-value' in environment
 assert metadata['current_environment']=='changed-after-start'
 assert os.readlink('/proc/self')==str(os.getpid())
 print('cmdline and environ use NUL separators; environ retains the initial value here; self resolves for the reader')
 entry=root/'fd'/str(metadata['fd']);target=os.readlink(entry);actual=entry.stat()
 assert target==str(path)+' (deleted)' and not path.exists()
 assert (actual.st_dev,actual.st_ino)==(metadata['device'],metadata['inode'])
 assert entry.read_bytes()==b'fixture-data'
 print('a deleted pathname remains inspectable through an open descriptor subject to access checks')
 output,error=process.communicate('done\n',timeout=5)
 assert process.returncode==0,error
 assert not root.exists()
 print('after the controlled child exits and is reaped, its proc directory disappears')
 finally:
 if process.poll() is None:process.kill();process.wait()

The child sets its short process name to worker ) sample, opens a temporary file, and unlinks its pathname while keeping the descriptor open. It then reports only fixture metadata and waits. This gives the controller a known live process whose name deliberately exercises the stat-format edge case.

The controller finds that status has reported size zero but nonempty readable contents. It checks labeled PID and parent PID fields. For stat, it separates the PID and parenthesized name before interpreting the remaining positional fields, confirming the parent PID and a positive start-time field. Naive whitespace splitting would mistake part of the name for a later field.

The final command-line argument contains spaces and remains one argument after NUL-based parsing. The initial environment contains XB_PROC_TEST=fixture-value, even though the child has changed its application environment value after startup. Both observations use harmless fixture data. They demonstrate why whitespace parsing and assuming a live environment map can produce incorrect conclusions.

Reading /proc/self in the controller resolves to the controller’s PID. Reading the child’s descriptor link returns the removed pathname with the deleted suffix, but following that proc descriptor entry still exposes the expected device/inode pair and fixture bytes under the experiment’s permissions. The missing ordinary pathname and the surviving open object are distinct facts.

After receiving the controller’s message, the child closes its handle and exits. The controller waits for completion and confirms that this child’s proc directory disappears. The example establishes a controlled before-and-after lifecycle observation; it does not promise atomic consistency among arbitrary proc reads while a busy application changes its state.

Match the view to the question

Use the relevant entry’s documented parser and units. A stat field number should be interpreted only after handling the command-name field correctly; a status label should not be assigned an assumed line number. If correlating samples, include process start time and boot context as appropriate so a reused numeric PID is not silently treated as the same instance.

Access and namespace context matter. Proc mount options, credentials, and ptrace-related checks can restrict visibility. /proc/net reflects the relevant network namespace, and a container’s visible proc data is not automatically its configured resource budget. Confirm which namespace and resource interface answer the operational question before labeling an observation as host-wide or container-specific.

Some proc entries are writable controls, but their scope differs. /proc/sys exposes tunables, while /proc/PID/oom_score_adj adjusts a particular process’s OOM preference; writing integers to status is not the interface for that task. Keep observation separate from a deliberate configuration change, and inspect the actual target and documented permissions before applying any write.

Sources

proc(5), procfs structure, per-process visibility, and mount options, checked 2026-10-15.
proc_pid_stat(5), field layout, parenthesized command name, and start time, checked 2026-10-15.
proc_pid_environ(5), initial environment region and NUL-separated entries, checked 2026-10-15.
proc_pid_fd(5), descriptor links, referenced objects, and permission checks, checked 2026-10-15.
proc_pid_cmdline(5), argument separators and process-controlled command-line contents, checked 2026-10-15.

The systemd journal needs an explicit retention contract

Tue, 26 Oct 2027 00:00:00 +0000

The systemd journal stores messages with structured metadata in journal files. That makes service output searchable without first inventing a parser for every application’s text format. Keeping those messages for a useful period is a separate requirement. Storage location, size policy, rotation, and forwarding determine which evidence remains available when an incident is investigated later.

Storage path and retention are separate questions. Rotation closes the old file as an archive and opens a replacement; vacuum can delete archives, not active files. A journald service restart does not establish reboot or power-loss durability, and a byte target does not guarantee a fixed evidence window. This is a source model, not a new execution result.

When it breaks

The default namespace normally uses Storage=auto: persistent storage is selected when /var/log/journal exists, otherwise runtime storage is used. Storage=volatile writes beneath /run/log/journal; this is not simply the kernel’s text ring buffer. Those runtime files normally disappear across a machine reboot, but restarting the journald service is a different event and need not erase them.

Storage=persistent prefers /var/log/journal, with runtime storage used during early boot or when persistent storage cannot be written. A flush operation transfers runtime data under the documented conditions. Inspect actual files and effective configuration instead of inferring location solely from the presence of an option in one configuration fragment.

journald has default size limits and rotation behavior. It does not grow without bounds merely because no administrator wrote a rotation job. However, SystemMaxUse is not an exact hard quota that guarantees the filesystem can never fill. Runtime storage has separate settings, free-space constraints also apply, other programs consume space, and only archived journal files can be deleted during vacuuming.

Check it quickly

Run this Python script as an administrator in a disposable Linux system managed by systemd. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It creates a unique journal namespace, gives it an explicit persistent-storage configuration, emits two harmless messages through transient services, and removes only that namespace’s data afterward. The vacuum operations deliberately delete the experiment’s old message:

from pathlib import Path
import json,shutil,subprocess,uuid

name='xbcatm-journal-'+uuid.uuid4().hex[:8]
config=Path('/etc/systemd')/('journald@'+name+'.conf')
directory=Path('/var/log/journal')/(Path('/etc/machine-id').read_text().strip()+'.'+name)
assert not config.exists() and not directory.exists()
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=15)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout
def journal(*args):return run('journalctl','--namespace='+name,*args)
def emit(message):
 run('systemd-run','--quiet','--wait','--collect','--unit='+name+'-'+message,
 '--property=LogNamespace='+name,'--property=StandardOutput=journal',
 '--property=SyslogIdentifier='+name,'/usr/bin/printf',message+'\n')
 journal('--sync')
def messages():
 rows=journal('--no-pager','-o','json','SYSLOG_IDENTIFIER='+name)
 return [json.loads(line)['MESSAGE'] for line in rows.splitlines()]
config.write_text('[Journal]\nStorage=persistent\nSystemMaxUse=16M\nRuntimeMaxUse=8M\n')
try:
 emit('old');journal('--flush');journal('--sync')
 assert messages()==['old'] and list(directory.glob('*.journal'))
 run('systemctl','restart','systemd-journald@'+name+'.service')
 assert messages()==['old']
 print('persistent namespace: message survives journald service restart')
 journal('--vacuum-size=1')
 assert messages()==['old']
 print('vacuum cannot remove the active journal file to satisfy a one-byte target')
 journal('--rotate');emit('new')
 assert messages()==['old','new']
 assert list(directory.glob('*@*.journal'))
 journal('--vacuum-size=1')
 assert messages()==['new'] and not list(directory.glob('*@*.journal'))
 print('after rotation: archived old message removed; active new message remains')
 print('persistent files observed under /var/log/journal; no reboot durability claim')
finally:
 run('systemctl','stop','systemd-journald@'+name+'.service',
 'systemd-journald@'+name+'.socket','systemd-journald-varlink@'+name+'.socket')
 config.unlink(missing_ok=True)
 for path in (directory,Path('/run/log/journal')/directory.name,Path('/run/systemd')/('journal.'+name)):
 if path.exists():shutil.rmtree(path)

The first message is present in journal files beneath the namespace’s /var/log/journal directory. After restarting that namespace’s journald service, the message is still queryable. This verifies the selected storage path and survival across a daemon restart. It does not simulate a power failure, prove storage-device durability, or test a complete machine reboot.

Vacuuming to a one-byte target initially leaves the message visible because its file is still active. The requested size is too small, but vacuum is not permitted to delete that active file just to meet the number. This provides a concrete counterexample to treating a vacuum target as a guaranteed final disk-usage figure.

The script then rotates the journal and writes a new message. Rotation closes the earlier file as an archive and opens a replacement. Both messages are readable before the second vacuum. Afterward, the archived old message is gone while the new message in the active file remains. Rotation and deletion are separate operations with different effects on evidence.

All journal commands select the unique namespace, so they do not vacuum the system’s default journal. Namespace service and socket instances are stopped before their test directories are removed. A normal query without the namespace selector would look in the default namespace and could miss these records entirely, even though they were successfully stored.

Size the evidence window

Choose retention from the incident-response window and observed message volume. A size limit cannot promise a fixed number of days when traffic changes. A retention-age limit likewise does not reserve enough bytes to preserve every message until that age. Monitor actual disk use and the earliest available records, especially after an unusually noisy failure.

Log collection can also lose messages before retention becomes relevant. Rate limits, application output routing, permissions, and unavailable storage affect what reaches the journal. Preserve the distinction between “no matching stored record” and “the event never occurred.” Avoid putting credentials into messages merely because local journal access is restricted.

If records must survive loss of the machine, provide and verify an independent collection path. Forwarding configuration alone is not evidence that a remote destination received or retained the message. Check delivery, access control, and the remote retention window using an identifiable harmless event. Local searchable logs and durable external evidence answer related but different operational needs.

Sources

systemd 252 journald.conf, storage selection, size defaults, free-space limits, and retention, checked 2026-09-12.
systemd 252 journalctl, namespace selection, rotation, synchronization, and archive vacuuming, checked 2026-09-12.
systemd 252 systemd-journald.service, journal namespaces, transport, and persistent file locations, checked 2026-09-12.

Inotify reports changes, but it is not a complete history

Wed, 13 Oct 2027 00:00:00 +0000

Inotify delivers Linux filesystem notifications through an instance with registered watches and a readable event queue. Each record identifies a watch, event mask, and possibly a name and move cookie. This can avoid repeatedly checking unchanged paths, but the stream is not an unlimited audit log. Correct consumers interpret notifications as evidence that relevant state may need to be inspected again.

Notifications provide bounded evidence of change. Watch coverage, object replacement and queue loss all require checking current filesystem state; a rescan cannot recover missing event history.

When it breaks

Watching a directory does not recursively watch its descendants. It reports relevant activity on immediate entries, so a new subdirectory needs its own watch if deeper changes matter. A tree watcher must handle initial enumeration and directories arriving later. Assuming one root watch covers the entire tree silently misses changes below that root.

A file watch follows the watched object, not a pathname forever. An editor can finish writing a temporary file and rename it over the target. The parent directory reports the replacement, while the old file watch does not transfer to the new inode. Waiting for a later close-write event on the destination name can fail because the replacement was already closed before publication.

Successive identical unread events can coalesce, and a full queue drops events while reporting IN_Q_OVERFLOW. Event counts therefore do not reliably count writes. After overflow, a consumer cannot reconstruct every missing operation from the remaining records. It must reconcile the current watched state instead of continuing with an assumption that its event-derived model is complete.

Check it quickly

Save the script as inotify-events.py and run python3 inotify-events.py in a disposable Linux environment. The earlier record used Linux 6.12.76 and ext4; the 2026-10-15 review recorded both translations passing on Linux 6.18.44 with Python 3.12.14. That is historical execution, not a rerun here. It creates ASCII-named temporary entries and intentionally fills its own queue. The configured queue limit must be at most 32768; the script reads that limit without changing any sysctl:

from pathlib import Path
from tempfile import TemporaryDirectory
import ctypes,os,struct

MODIFY=0x2;ATTRIB=0x4;CLOSE_WRITE=0x8;MOVED_FROM=0x40;MOVED_TO=0x80
CREATE=0x100;DELETE=0x200;DELETE_SELF=0x400;MOVE_SELF=0x800;OVERFLOW=0x4000;IGNORED=0x8000
libc=ctypes.CDLL(None,use_errno=True)
libc.inotify_init1.argtypes=[ctypes.c_int];libc.inotify_init1.restype=ctypes.c_int
libc.inotify_add_watch.argtypes=[ctypes.c_int,ctypes.c_char_p,ctypes.c_uint32];libc.inotify_add_watch.restype=ctypes.c_int
header=struct.Struct('iIII')
with TemporaryDirectory(prefix='xbcatm-inotify-') as directory:
 root=Path(directory);fd=libc.inotify_init1(os.O_NONBLOCK|os.O_CLOEXEC);assert fd>=0,os.strerror(ctypes.get_errno())
 def watch(path,mask):
 wd=libc.inotify_add_watch(fd,os.fsencode(path),mask);assert wd>=0,os.strerror(ctypes.get_errno());return wd
 def drain():
 events=[]
 while True:
 try:data=os.read(fd,65536)
 except BlockingIOError:return events
 offset=0
 while offset<len(data):
 wd,mask,cookie,length=header.unpack_from(data,offset);offset+=header.size
 name=data[offset:offset+length].split(b'\0',1)[0].decode();offset+=length
 events.append((wd,mask,cookie,name))
 try:
 parent=watch(root,MODIFY|CLOSE_WRITE|CREATE|DELETE|MOVED_FROM|MOVED_TO)
 sub=root/'sub';sub.mkdir();drain();(sub/'unwatched').touch();assert drain()==[]
 nested=watch(sub,CREATE);(sub/'watched').touch();assert any(wd==nested and mask&CREATE and name=='watched' for wd,mask,_,name in drain())
 print('a directory watch is not recursive; adding a child-directory watch observes its new entries')
 counter=root/'counter';stream=os.open(counter,os.O_CREAT|os.O_WRONLY,0o600);drain()
 try:
 for _ in range(3):os.write(stream,b'x')
 finally:os.close(stream)
 events=drain();assert sum(wd==parent and bool(mask&MODIFY) and name=='counter' for wd,mask,_,name in events)==1
 moved=root/'moved';counter.rename(moved);events=drain()
 before=next(e for e in events if e[1]&MOVED_FROM and e[3]=='counter');after=next(e for e in events if e[1]&MOVED_TO and e[3]=='moved')
 assert before[2]==after[2]!=0
 print('three unread identical modifications coalesce; rename FROM/TO events share a nonzero cookie')
 target=root/'target';target.write_bytes(b'old');file_watch=watch(target,MODIFY|ATTRIB|DELETE_SELF|MOVE_SELF);drain()
 replacement=root/'replacement';replacement.write_bytes(b'new');drain();replacement.replace(target);events=drain()
 assert any(wd==file_watch and mask&IGNORED for wd,mask,_,_ in events)
 assert any(wd==parent and mask&MOVED_TO and name=='target' for wd,mask,_,name in events)
 assert not any(wd==parent and mask&CLOSE_WRITE and name=='target' for wd,mask,_,name in events)
 target.write_bytes(b'newer');events=drain()
 assert any(wd==parent and mask&MODIFY and name=='target' for wd,mask,_,name in events)
 assert all(wd!=file_watch for wd,_,_,_ in events)
 print('replacement invalidates the old file watch; the parent-directory watch continues to observe the pathname')
 burst=root/'burst';burst.mkdir();drain();watch(burst,CREATE)
 limit=int(Path('/proc/sys/fs/inotify/max_queued_events').read_text());assert 0<limit<=32768,'use a disposable environment with a bounded queue limit'
 for number in range(limit+8):(burst/str(number)).touch()
 events=drain();assert any(wd==-1 and mask&OVERFLOW for wd,mask,_,_ in events)
 current=len(list(burst.iterdir()));assert current==limit+8
 print('actual queue overflow:',len(events),'events retained for',current,'creates; rescan sees the current entries')
 finally:os.close(fd)

The first case watches the parent directory, creates a subdirectory, and drains the parent event. Creating a file inside that subdirectory produces no event for the parent watch. Adding an explicit subdirectory watch makes the next creation visible. This distinguishes direct-entry coverage from recursive monitoring using the same inotify instance.

Three writes to one file occur before the queue is read. The fixture observes one matching modify event, demonstrating coalescing rather than one notification per write. Renaming the file then produces FROM and TO events with the same nonzero cookie. The cookie associates these two records; it is not a durable identifier for the file’s entire future lifetime.

Next, the script watches a file and replaces its pathname with a previously written and closed file. The old watch receives an ignored event when its object is deleted, while the parent watch reports MOVED_TO for the target name. No destination close-write event accompanies that publication. A subsequent write to the replacement is visible through the parent watch, not the removed file watch.

For overflow, the fixture creates 16392 distinct entries with a configured limit of 16384 queued events. It receives 16385 records, including the overflow notification with watch descriptor minus one. A directory rescan finds all 16392 current entries. Those results show both missing event history and a concrete way to recover present-state knowledge.

All watches share one owned instance, which closes before temporary-tree cleanup. The experiment does not induce a system-wide watch shortage or modify other applications’ queues. Its deliberately bounded burst validates overflow behavior without requiring an application to remain permanently overloaded or relying on an assumed loss signal that was never observed.

Reconcile names and object lifetimes

Pair move cookies when both sides are visible, but allow for moves into or out of the observed set and records separated by other events. Recheck the relevant paths before acting. A name in an event describes the reported change; it does not guarantee that the same object will still occupy that name when the consumer later opens it.

For trees, establish watches and reconcile directory contents while accounting for activity during enumeration. On overflow, rebuild the affected state from the filesystem and restore the necessary watch coverage. An overflow record does not identify one missing filename, so handling it as an ordinary per-watch event is insufficient.

Use notifications to trigger work that tolerates repeated or combined changes, such as reloading a complete configuration after validating its current contents. If every transition must be preserved, use an application journal or another interface with the required history contract. Inotify is useful precisely when its coverage and loss rules match the consumer’s actual needs.

Sources

inotify(7), event formats, coalescing, overflow, renames, and nonrecursive watches, checked 2026-10-15.
inotify_add_watch(2), watch registration and object identity, checked 2026-10-15.
Linux 6.12 inotify UAPI, event structure and flag values used by the executable fixture, checked 2026-10-15.

Hard links share an inode until a name is replaced

Sun, 10 Oct 2027 00:00:00 +0000

A hard link adds another directory entry referring to an existing file. Neither name is intrinsically the original: both identify the same inode within its filesystem. File content, ownership, permissions, and link count belong to that shared object. A filename is a way to reach it, so changes to directory entries must be distinguished from changes to the file itself.

A name and an open reference have different lifetimes. Track the inode and the actual update method.

When it breaks

Treating a hard link as an independent backup creates a false recovery boundary. Writing or truncating the shared inode through either name changes what the other name reads. Changing its mode likewise changes the same object’s permissions. A second pathname can survive removal of the first, but it does not preserve an independent historical version against ordinary in-place modifications.

Many editors and deployment tools update differently: they create a replacement file and rename it over one pathname. That directory entry now reaches a new inode, while another hard link still reaches the old one. Whether both names show the update depends on the update method. The mere fact that they were linked yesterday does not establish their identity today.

Hard links also stay within the applicable filesystem mount boundary. link cannot join arbitrary files from different mounts, even when directories appear next to each other in one pathname tree. Ordinary directory hard links are prohibited on Linux. These restrictions prevent using hard links as a general replacement for symbolic links across a directory hierarchy.

Check it quickly

Save the script as hardlinks.py and run python3 hardlinks.py on Linux with permission to create temporary directories. It requires writable /tmp and /run on different filesystems; set CROSS_ROOT to another writable directory on a second filesystem when needed. No root privilege or mount creation is required. The recorded environment used separate tmpfs mounts, Linux 6.12.76, and Python 3.11.2. All created entries are confined to temporary directories:

from pathlib import Path
from tempfile import TemporaryDirectory
import errno,os

with TemporaryDirectory(prefix='xbcatm-hardlinks-',dir='/tmp') as directory, TemporaryDirectory(prefix='xbcatm-crosslink-',dir=os.environ.get('CROSS_ROOT','/run')) as other:
 root=Path(directory);source=root/'source';alias=root/'alias'
 source.write_bytes(b'original');os.link(source,alias)
 before=source.stat();linked=alias.stat()
 assert (before.st_dev,before.st_ino)==(linked.st_dev,linked.st_ino)
 assert before.st_nlink==linked.st_nlink==2 and before.st_size==linked.st_size==8
 alias.write_bytes(b'changed')
 assert source.read_bytes()==b'changed'
 print('two names share device/inode, size, and link count; in-place write changes both views')
 replacement=root/'replacement';replacement.write_bytes(b'new file');replacement.replace(source)
 assert source.stat().st_ino!=alias.stat().st_ino
 assert source.read_bytes()==b'new file' and alias.read_bytes()==b'changed'
 assert source.stat().st_nlink==alias.stat().st_nlink==1
 print('atomic replacement changes one name to a new inode; the other name retains the old file')
 with alias.open('rb') as opened:
 alias.unlink()
 assert os.fstat(opened.fileno()).st_nlink==0 and opened.read()==b'changed'
 print('last unlink removes the final name but an existing open descriptor still reads the old data')
 assert root.stat().st_dev!=Path(other).stat().st_dev,'exercise requires two filesystem mounts'
 try:os.link(source,Path(other)/'cross-filesystem')
 except OSError as error:assert error.errno==errno.EXDEV,error
 else:raise AssertionError('cross-filesystem hard link unexpectedly succeeded')
 child=root/'directory';child.mkdir()
 try:os.link(child,root/'directory-link')
 except OSError as error:assert error.errno==errno.EPERM,error
 else:raise AssertionError('directory hard link unexpectedly succeeded')
 print('cross-filesystem link fails with EXDEV; directory hard link fails with EPERM on Linux')

The fixture creates source, writes eight bytes, and adds alias with os.link. Both names report identical device/inode pairs, size eight, and link count two. Comparing the device as well as the inode matters because inode numbers are not globally unique across filesystems. Matching file contents alone would also be insufficient evidence of shared identity.

Writing changed through alias updates the existing inode, so reading source returns the same changed bytes. The operation uses an ordinary truncating write, not replacement. This establishes why the second name cannot serve as a separate copy protecting the prior contents from a writer with access to the shared file.

Next, a separate file containing new file replaces source by rename. The two names now have different inode numbers and each has link count one. source reads the replacement, while alias retains changed. Replacement affects the selected directory entry; it neither searches for other names nor rewrites every hard link to the old inode.

The fixture opens alias and removes its final directory entry. File status through the open descriptor reports link count zero, yet that descriptor still reads changed. The open reference preserves access until it is closed. A zero link count means no hard-link names remain, not that every process has immediately lost its existing reference.

Finally, linking into the second filesystem fails with EXDEV, and attempting to hard-link a directory fails with EPERM. The script checks the specific errors. If its mount precondition is absent, it fails explicitly rather than reporting that a cross-filesystem operation was tested when both directories actually used the same filesystem.

Inspect identity and update behavior

Before relying on shared storage, inspect the device/inode pair and link count of the actual paths. Link count tells how many hard-link directory entries refer to the inode; it does not list where those names are. Traversing directories to find matching inodes is a separate operation, affected by permissions and changes happening during the search.

For snapshot-like directory trees built with hard links, ensure updates replace files instead of modifying shared inodes, and control all writers that can reach them. Filesystem snapshots and independent copies have different contracts. A space-saving tree of links should be described by its actual update and retention behavior rather than simply being called a backup.

Unlinking and renaming concern namespace visibility. This experiment verifies identity and open-reference behavior, not persistence across power loss. If durable publication matters, separately verify the filesystem’s synchronization requirements and the application’s recovery procedure. Keeping another name or observing an atomic rename does not by itself prove the intended data will survive a crash.

Sources

link(2), shared inode identity, cross-filesystem restrictions, and directory-link errors, checked 2026-09-12.
unlink(2), link-count reduction and lifetime of an unlinked open file, checked 2026-09-12.
rename(2), replacement of one pathname without changing other hard links, checked 2026-09-12.

Cron repeats on the calendar

Thu, 30 Sep 2027 00:00:00 +0000

Cron launches commands at matching calendar times without inherently waiting for earlier invocations. Seventy-second work scheduled every minute can therefore overlap. Treat triggering, concurrency, and successful completion as separate responsibilities.

The illustrative lanes show the next calendar trigger arriving before a seventy-second invocation ends. A shared nonblocking lock makes the later invocation record a skip; it does not provide exactly-once effects. The source’s prior measurements were not rerun.

Read fields as calendar selections

A user crontab entry contains minute, hour, day of month, month, day of week, then command. Asterisks select all field values. A minute step such as */15 selects values within each hour, not fifteen minutes since completion. Steps that unevenly divide a field’s range can produce unequal boundary gaps.

In Debian/Vixie format, when day of month and day of week are both restricted, either matching day permits execution; minute, hour, and month constraints still apply. Month and weekday are not alternatives. Verify representative dates before installing the expression.

Timezone, daylight-saving changes, and downtime affect scheduling. Implementations handle skipped or repeated times differently; Debian documents special handling for certain small clock changes and fixed-time jobs. This experiment neither changes the clock nor tests transitions. Calendar triggers are not durable queues guaranteeing replay of missed runs.

When it breaks

Overlapping jobs can duplicate side effects, compete for resources, or overwrite files. Explicitly enforce any single-invocation policy: nonblocking advisory locks permit recorded skips; blocking locks can leave waiting processes. Neither provides exactly-once business effects or repairs partial operations.

A local file lock only coordinates participants using the same relevant locking facility. It does not automatically serialize jobs on unrelated hosts, and network filesystem semantics require separate examination. Keep the lock held for the work’s lifetime, and make skipped work visible. Silently dropping every later trigger while one worker hangs can look like a healthy schedule unless completion age is monitored.

Cron is not universally log-free. Debian logs job starts to syslog by default and offers additional logging controls. Command output may be mailed when a working mail path exists, but a minimal container often lacks that path. Record useful start, completion, failure, and skip outcomes through an intentional destination. A start message proves dispatch, not completion of the application’s task.

Check it quickly

Repair draft checked 2026-10-19: this revision was checked statically and with offline command/file-state doubles only. No cron, crontab, account, or signal operation was executed. Technical and operational verification remain blocked; the measurements below belong to the original fixture, not this revision.

Only a separately authorized, freshly provisioned disposable Debian 12 lab with Python 3.11+, cron 3.0pl1-162 and procps is suitable for a future run. It must have no host mounts, existing root table, cron daemon, enabled unrelated schedules in /etc/crontab, /etc/cron.d or any user’s spool (including periodic-directory dispatch), or concurrent administrators. Root is a privilege prerequisite, not isolation proof. Save as cron-overlap.py; the future command is python3 cron-overlap.py --isolated-disposable-lab. That flag acknowledges an externally verified prerequisite and cannot establish containment. Allow roughly two minutes for real calendar ticks, and discard the entire lab afterward.

The test table disables mail with MAILTO. Setup is inside the cleanup boundary, including a failed table installation or daemon startup. A failed listing is accepted as absence only for the pinned package’s exact English absent-table diagnostic; permission and command errors abort. Workers publish complete per-event JSON files by same-directory replacement. Cleanup revokes a positive run permit before removing schedules and stopping the owned daemon, then waits at most five seconds for the workers’ lifetime lock and completion records. Each cleanup step reports its own error without skipping later steps; private-file removal is skipped after earlier cleanup errors; deletion itself can fail partway. This checks admitted work, not the exit of every cron wrapper or late-start process. It is not a containment or crash-durability proof.

from pathlib import Path
from tempfile import mkdtemp
import fcntl
import json
import os
import shutil
import subprocess
import sys
import time


def command(arguments, **options):
 return subprocess.run(arguments, capture_output=True, text=True, timeout=5,
 env={**os.environ, 'LC_ALL': 'C'}, **options)


def table_absent(result):
 # Debian cron 3.0pl1-162: other listing failures must abort.
 return (result.returncode == 1 and result.stdout == ''
 and result.stderr == 'no crontab for root\n')


def remove_table():
 current = command(['crontab', '-l'])
 if table_absent(current):
 return
 current.check_returncode()
 command(['crontab', '-r']).check_returncode()
 if not table_absent(command(['crontab', '-l'])):
 raise RuntimeError('could not verify table removal')


def attempt(errors, name, operation):
 try:
 operation()
 except Exception as error:
 errors.append(f'{name}: {error!r}')


def stop_daemon(daemon, errors):
 if daemon is None:
 return
 attempt(errors, 'terminate cron', daemon.terminate)
 try:
 daemon.wait(timeout=5)
 except subprocess.TimeoutExpired:
 attempt(errors, 'kill owned cron', daemon.kill)
 attempt(errors, 'reap cron', lambda: daemon.wait(timeout=5))
 except Exception as error:
 errors.append(f'wait cron: {error!r}')


def read_events(root):
 return sorted((json.loads(path.read_text()) for path in root.glob('*.json')),
 key=lambda event: event['monotonic'])


def wait_workers(root):
 # Workers hold a shared lifetime lock before checking the run permit or publishing.
 with (root / 'lifetime.lock').open() as gate:
 deadline = time.monotonic() + 5
 while True:
 try:
 fcntl.flock(gate, fcntl.LOCK_EX | fcntl.LOCK_NB)
 break
 except BlockingIOError:
 if time.monotonic() >= deadline:
 raise TimeoutError('workers did not release lifetime lock')
 time.sleep(0.1)
 events = read_events(root)
 started = {e['pid'] for e in events if e['event'] == 'start'}
 ended = {e['pid'] for e in events if e['event'] == 'end'}
 if not started <= ended:
 raise RuntimeError('a started worker has no completion record')


worker_source = r'''from pathlib import Path
import fcntl,json,os,sys,time
root=Path(__file__).parent
kind=sys.argv[1]
def record(event):
 destination=root/f'{os.getpid()}-{event}.json'
 temporary=destination.with_suffix('.tmp')
 temporary.write_text(json.dumps({'event':event,'kind':kind,
 'pid':os.getpid(),'monotonic':time.monotonic()}))
 temporary.replace(destination)
def main():
 with (root/'lifetime.lock').open() as gate:
 fcntl.flock(gate,fcntl.LOCK_SH)
 if not (root/'run-permit').exists(): return
 lock=None
 try:
 if kind=='locked':
 lock=open(root/'job.lock','a')
 try: fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
 except BlockingIOError:
 record('skip')
 return
 record('start')
 try:
 until=time.monotonic()+70
 while time.monotonic()<until and (root/'run-permit').exists():
 time.sleep(0.1)
 finally: record('end')
 finally:
 if lock is not None: lock.close()
main()
'''


def main():
 # This flag is an operator prerequisite, NOT evidence of containment.
 if sys.argv[1:] != ['--isolated-disposable-lab']:
 raise SystemExit('requires a separately verified disposable Debian 12 lab')
 if os.geteuid() != 0:
 raise RuntimeError('root is required, but does not prove isolation')
 probe = command(['pgrep', '-x', 'cron'])
 if probe.returncode != 1 or probe.stdout or probe.stderr:
 raise RuntimeError(f'cron absence not established: {probe!r}')
 if not table_absent(command(['crontab', '-l'])):
 raise RuntimeError('expected absent root table; existing table or listing error')
 root = Path(mkdtemp(prefix='cron-overlap-', dir='/tmp'))
 daemon = None
 table_attempted = False
 try:
 (root / 'lifetime.lock').touch()
 (root / 'run-permit').touch()
 worker = root / 'worker.py'
 worker.write_text(worker_source)
 table = 'MAILTO=""\n' + ''.join(f'* * * * * /usr/bin/python3 {worker} {kind} >>{root}/worker.log 2>&1\n'
 for kind in ('plain', 'locked'))
 table_attempted = True
 command(['crontab', '-'], input=table).check_returncode()
 daemon = subprocess.Popen(['cron', '-f'], stdout=subprocess.DEVNULL,
 stderr=subprocess.DEVNULL)
 deadline = time.monotonic() + 140
 while time.monotonic() < deadline:
 if daemon.poll() is not None:
 raise RuntimeError('cron exited')
 events = read_events(root)
 plain = [e for e in events if (e['kind'], e['event']) == ('plain', 'start')]
 locked = [e for e in events if (e['kind'], e['event']) == ('locked', 'start')]
 skipped = [e for e in events if (e['kind'], e['event']) == ('locked', 'skip')]
 if len(plain) >= 2 and skipped:
 break
 time.sleep(0.2)
 else:
 raise AssertionError('two real cron ticks were not observed')
 assert len(plain) == 2 and len(locked) == 1 and len(skipped) == 1
 assert plain[1]['monotonic'] - plain[0]['monotonic'] < 70
 assert not any(e['event'] == 'end' and e['pid'] == plain[0]['pid'] for e in events)
 print(json.dumps({'actual_cron_starts': len(plain), 'overlap_observed': True,
 'seconds_between_starts': round(plain[1]['monotonic']-plain[0]['monotonic'], 3),
 'locked_starts': len(locked), 'locked_skips': len(skipped)}), flush=True)
 finally:
 original = sys.exc_info()[1]
 errors = []
 attempt(errors, 'revoke run permit', lambda: (root / 'run-permit').unlink(missing_ok=True))
 if table_attempted:
 attempt(errors, 'remove root table', remove_table)
 stop_daemon(daemon, errors)
 attempt(errors, 'wait for workers', lambda: wait_workers(root))
 # Retain private files after failed cleanup; discard the whole lab.
 if not errors:
 attempt(errors, 'remove private files', lambda: shutil.rmtree(root))
 if errors:
 message = 'cleanup incomplete; discard lab: ' + '; '.join(errors)
 if original is not None:
 original.add_note(message)
 else:
 raise RuntimeError(message)


if __name__ == '__main__':
 main()

Two every-minute entries run workers that would last seventy seconds. The plain worker has no job-exclusion lock; both branches share only the cleanup lifetime lock. The locked worker uses a shared file and a nonblocking exclusive flock, recording a skip when acquisition fails. Each event includes the worker PID and a monotonic timestamp, so distinct invocations and their overlap can be checked directly.

The original fixture’s historical Debian cron 3.0pl1-162 run recorded two plain starts, 60.003 seconds apart, before its first invocation ended. The guarded worker started once and recorded one skip. That original run reported all assertions passing and cleanup via its stop file. This revision instead revokes a positive permit and has not reproduced those runtime results. The experiment verifies local overlap and explicit exclusion; it does not implement durable retries, distributed locking, recovery after reboot, or exactly-once effects. Select those policies from the job’s actual business contract instead of assuming the calendar expression supplies them.

Sources

Debian 12: crontab(5), calendar fields, steps and the two day fields, checked 2026-09-13.
Debian 12: cron(8), daemon execution, logging and clock-change behavior, checked 2026-09-13.
Linux: flock(2), advisory locks and nonblocking acquisition, checked 2026-09-13.
Python 3.11 subprocess, timeout and child cleanup semantics, checked 2026-10-19.
Debian cron 3.0pl1-162 crontab.c, distinguish absent-table diagnostics from other failures, checked 2026-10-19.
Python 3.11 os.replace, atomic same-filesystem publication, checked 2026-10-19.

Linux · XB Field Notes

LVM snapshots capture a block device in time

An exception store, not a copy

Exhaustion is invalidation

Merging, and why thin changes the failure

When it breaks

Check it quickly

Sources

The SYN backlog holds incomplete handshakes

The backlog argument is the accept queue

SYN cookies remain a fallback

The accept queue is where the pressure lands

When it breaks

Check it quickly

Sources

iotop shows which process is burning the disks

Per-process counters

See whatever sweeps the disk

Watch before you act

When it breaks

Check it quickly

Sources

Network namespaces isolate networking; their names do not own every process

When it breaks

Check it quickly

Keep naming, process ownership, and connectivity separate

Sources

Strace explains selected system-call boundaries

When it breaks

Check it quickly

Follow a hypothesis through to an outcome

Sources

Cron system and user tables have different contracts

The account column changes the parser boundary

When it breaks

Check it quickly

Sources

Shell word splitting turns unquoted variables into several arguments

Splitting depends on the context

When it breaks

Check it quickly

Sources

Huge pages trade allocation flexibility for translation reach

Page size is an input

Explicit pools and transparent allocation

Budget capacity and placement

When it breaks

Check it quickly

Further reading

Sources

Linux overcommit budgets virtual-memory commitments

Virtual commitment differs from residence

Three admission policies

Strict accounting is not OOM immunity

When it breaks

Check it quickly

Further reading

Sources

systemd-tmpfiles applies creation and age policies

When it breaks

Check it quickly

Match housekeeping to the producer

Further reading

Sources

A systemd oneshot completes one activation

When it breaks

Check it quickly

Give the task its own correctness guarantees

Further reading

Sources

A veth pair carries frames between two endpoints; routing supplies their meaning

When it breaks

Check it quickly

Diagnose the layers in order

Sources

A page fault is not always a storage read

Minor does not mean free

Major describes a different path

When it breaks

Check it quickly