Systemd · XB Field Notes

systemd dependencies separate activation, ordering, and readiness

Fri, 18 Feb 2028 00:00:00 +0000

A dependency graph needs to express more than “start the database first.” One relationship decides whether another unit is pulled into the transaction; another orders the jobs; the service type defines when startup has finished. Application readiness is a further promise that the program must actually fulfill. Confusing these layers produces a boot sequence that looks ordered while the dependent service still starts too early.

Activation, ordering and readiness are separate contracts. Requires plus After blocks a consumer when its provider fails activation; notification can hold ordered startup until READY=1. This is a source-based explanation, not a new systemd run or a continuous-health guarantee.

When it breaks

After=provider.service orders this unit after the provider when both have start jobs. It does not request that the provider start. If only the consumer is activated, the ordering edge alone can leave the provider inactive. Add an appropriate activation dependency when the consumer really should bring the provider into the transaction.

Wants requests another unit but tolerates its startup failure. Requires expresses a stronger relationship; together with After, a provider that fails activation prevents the dependent unit from starting. Neither keyword by itself establishes startup order. Use the combination that matches the service’s actual need rather than turning every optional component into a hard prerequisite.

Ordered startup is only as strong as the provider’s completion signal. Type=exec establishes successful program execution, not completion of database initialization. Type=notify can hold startup until the program sends READY=1; a oneshot can hold it until finite setup finishes. The application must send notification only when its promised readiness condition is true.

Check it quickly

Run this complete Python script as an administrator in a disposable Linux systemd environment. It installs two temporary runtime units and exercises six configurations sequentially. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. Ensure both example unit names are unused. Temporary files make the provider’s process entry, readiness, and the consumer’s observation independently visible:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,time

provider='xbcatm-provider-lab.service';consumer='xbcatm-consumer-lab.service'
paths={name:Path('/run/systemd/system')/name for name in (provider,consumer)}
assert all(not p.exists() for p in paths.values())
def ctl(*args,success=True):
 result=subprocess.run(['systemctl',*args],text=True,capture_output=True,timeout=10)
 assert (result.returncode==0)==success,(args,result.stderr)
 return result.stdout.strip()
def value(name,key):return ctl('show',name,'-p',key,'--value')
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'state transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix='xbcatm-dependency-',dir='/run') as directory:
 root=Path(directory)
 (root/'provider.py').write_text(r"""from pathlib import Path
import os,socket,sys,time
root=Path(__file__).parent;mode=(root/'mode').read_text()
if mode=='fail': sys.exit(7)
(root/'entered').touch()
if mode in ('exec','notify'):
 while not (root/'gate').exists(): time.sleep(0.01)
(root/'ready').touch()
if mode=='notify':
 address=os.environ['NOTIFY_SOCKET']
 if address.startswith('@'): address='\0'+address[1:]
 with socket.socket(socket.AF_UNIX,socket.SOCK_DGRAM) as notification:
 notification.connect(address);notification.sendall(b'READY=1')
if mode in ('exec','notify'): time.sleep(60)
""")
 (root/'consumer.py').write_text("from pathlib import Path\nr=Path(__file__).parent\n(r/'observed').write_text(str((r/'ready').exists()))\n")
 def install(mode,requirement):
 (root/'mode').write_text(mode)
 kind='oneshot' if mode in ('oneshot','fail') else mode
 paths[provider].write_text(f'[Service]\nType={kind}\nExecStart=/usr/bin/python3 {root}/provider.py\nTimeoutStartSec=5s\n'+('RemainAfterExit=yes\n' if kind=='oneshot' else ''))
 paths[consumer].write_text(f'[Unit]\nAfter={provider}\n'+(f'{requirement}={provider}\n' if requirement else '')+f'[Service]\nType=oneshot\nExecStart=/usr/bin/python3 {root}/consumer.py\n')
 subprocess.run(['systemd-analyze','verify',*[str(p) for p in paths.values()]],check=True)
 ctl('daemon-reload')
 def reset():
 ctl('stop',consumer,provider)
 for name in paths:
 if value(name,'ActiveState')=='failed':ctl('reset-failed',name)
 for name in ('entered','gate','ready','observed'):(root/name).unlink(missing_ok=True)
 pending=None
 install('oneshot','')
 try:
 ctl('start',consumer)
 assert (root/'observed').read_text()=='False' and value(provider,'ActiveState')=='inactive'
 print('After alone: provider not pulled in; consumer ran before readiness')
 reset();install('fail','Wants');ctl('start',consumer)
 assert value(provider,'ActiveState')=='failed' and (root/'observed').read_text()=='False'
 print('Wants plus After: consumer runs despite provider failure')
 reset();install('fail','Requires');ctl('start',consumer,success=False)
 assert value(provider,'ActiveState')=='failed' and not (root/'observed').exists()
 print('Requires plus After: failed provider prevents consumer execution')
 reset();install('oneshot','Requires');ctl('start',consumer)
 assert (root/'observed').read_text()=='True'
 print('oneshot provider: consumer sees completed setup')
 reset();install('exec','Requires');ctl('start',consumer)
 assert (root/'observed').read_text()=='False' and value(provider,'ActiveState')=='active'
 (root/'gate').touch();wait_for(lambda:(root/'ready').exists())
 print('Type=exec provider: process started before application readiness')
 reset();install('notify','Requires')
 pending=subprocess.Popen(['systemctl','start',consumer],text=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
 wait_for(lambda:(root/'entered').exists())
 assert pending.poll() is None and not (root/'observed').exists()
 assert value(provider,'ActiveState')=='activating'
 (root/'gate').touch();out,err=pending.communicate(timeout=10)
 assert pending.returncode==0,err
 assert (root/'observed').read_text()=='True'
 print('Type=notify provider: dependent waits for actual READY=1')
 finally:
 reset()
 if pending is not None and pending.poll() is None:pending.communicate(timeout=5)
 for path in paths.values():path.unlink()
 ctl('daemon-reload')

With only After, the consumer runs and records false for readiness while the provider remains inactive. That is a missing activation relationship, not a slow provider. With Wants plus After, the provider deliberately exits with status seven, but the consumer still runs and records false. The fixture accepts that observation; a real consumer might then fail its own work if the dependency was actually mandatory.

With Requires plus After, the same provider failure causes the consumer’s start command to fail and no observation file appears. The absent file proves the consumer program never ran. Replacing the failing provider with successful oneshot setup produces a true observation because ordering now waits for that setup to complete.

The next case uses Type=exec. The provider process is active but deliberately waits at an application gate before publishing readiness. The consumer still starts and records false. Releasing the gate later makes the provider ready, demonstrating the timing gap without depending on an unreliable arbitrary startup sleep.

Finally, Type=notify keeps the provider activating while it waits at the same gate. The consumer’s start job remains pending, and its program has not executed. After the gate opens, the provider publishes readiness and sends the actual systemd notification from its main process. Only then does the consumer run and record true.

The script verifies each unit configuration and cleans up both services, including a pending job if an assertion interrupts the exercise. These assertions require real processes and a real notification socket; source reading alone does not establish their outcomes. They do not establish that a production database’s readiness definition is sufficient for every client operation.

Match dependencies to failure semantics

Choose a readiness definition with the consumers in mind. A listening port may exist before migrations, credentials, or required data are ready. Conversely, waiting for an optional remote dependency can make a local service unavailable unnecessarily. Name the specific condition and decide which failures should block startup versus be handled during normal operation.

A completed startup does not guarantee that the provider stays healthy forever. Ordinary requirement dependencies are not continuous business-health checks, and condition-based skips differ from failed activation. Plan how the application handles later disconnects, restarts, and unavailable external services rather than treating the boot graph as a permanent availability guarantee.

Inspect the loaded unit definitions, their effective dependencies, and the relevant startup results when diagnosing order. Reload the manager after changing unit files, then test the failure case as well as a clean boot. Keep unrelated services independent so they can start concurrently, while retaining only the ordering edges needed by the actual completion contract.

Sources

systemd 252 unit dependencies, ordering, activation strength, and condition behavior, checked 2026-09-12.
systemd 252 service types, exec, oneshot, and notification startup boundaries, checked 2026-09-12.
systemd 252 systemctl, start jobs, inspection, and manager reloads, checked 2026-09-12.

Socket activation starts a service only when needed

Tue, 01 Feb 2028 00:00:00 +0000

A socket unit can keep a listening endpoint available while its compatible application is stopped. This can reduce the resident memory and periodic work of rarely used services. The socket, kernel queues and manager still consume resources, and installed software still needs security updates. Availability of the listener also depends on successful binding, permissions and the socket unit remaining active. A listening port is an opportunity to request work, not evidence that the application is ready.

The handoff depends on the configured mode. With Accept=no, systemd passes listening descriptors and the daemon accepts connections. With Accept=yes, it accepts a connection and starts a suitable service instance for that connection. Clients may connect before initialization finishes, then wait for an application response or time out. Measure that delay rather than assuming the handoff makes on-demand startup indistinguishable from a warm service.

With the socket unit active, a request can start a compatible service. Explicit application-defined idle exit can stop the process so later demand starts it again; without idle exit it stays running. Measure connect-to-first-response cold-start delay, client deadlines and the resident footprint under the real workload. Monitoring can activate multiple services together, so test combined startup load. Selected mechanisms only; the article remains held/pending and its incorrect general claims are not endorsed. Illustration of selected source mechanisms; no example executed in this round.

The listener waits first

systemd opens the configured ListenStream endpoint when the socket unit starts. With Accept=no, foo.socket normally activates foo.service and passes the listener; Service= can choose another service. With Accept=yes, the matching template is normally foo@.service and each accepted stream connection gets an instance. Datagram sockets do not use this per-connection model. The program must match the selected descriptor contract, rather than simply sharing a familiar unit basename.

This split can reduce startup-order coupling, but it does not remove dependencies. The socket unit must be active and listening before clients can use it, and required service, mount, device, and ordering dependencies still apply. Connections can wait in a finite kernel backlog while a compatible service starts, but clients can still fail or time out before the application is ready. The backlog size still deserves sizing: a flood of early clients against a slow-starting service overflows the same as any listen queue.

Service wakes on demand

On-demand activation defers the initial application launch while the socket is active. It does not put a running daemon to sleep or automatically stop one that becomes idle. Administrative services and infrequently used protocol handlers are plausible candidates when their activation interface, request deadlines and resource profile fit this lifecycle. Other dependencies or manual starts can still activate the service.

Idle exit completes a recurring savings cycle only when the application implements it safely. It must finish or transfer in-flight work before exiting and leave the manager-owned listener usable. With the socket unit still active, later demand can request another start. Without idle exit, the daemon may remain resident after its first activation. Neither case is a zero-cost state or a promise that the socket can never close.

Trade speed for memory

Budget cold-start time from connection through the first valid application response, and compare resident memory under representative load. A hypothetical 300 ms start saving 800 MB may fit one workload and violate another workload’s deadline. A 20-second warm-up might be acceptable for an asynchronous task but unsuitable for an interactive request. These are illustrative tradeoffs, not measurements or universal thresholds.

Some workloads cannot tolerate the gap. Health checks can activate a service themselves, or classify it as unavailable while it initializes. Client deadlines and load-balancer policy must fit the observed startup behavior. Keeping the daemon running may be simpler when predictable response time matters more than idle memory; warming it at boot does not itself implement later idle exit.

When it breaks

A monitoring sweep can wake many services together and create an aggregate CPU or memory spike. Staggering checks may reduce simultaneous demand. Per-service start-rate limits constrain repeated starts of each unit, but do not cap host-wide concurrency or memory use. Very short idle timeouts can increase churn. Test combined startup load and use suitable resource controls instead of treating these knobs as a global capacity guarantee.

A daemon that binds the occupied endpoint itself may fail with address-in-use errors. It must consume inherited descriptors through its native activation interface or a supported inetd-style interface. Merely inheriting a descriptor and executing an unaware daemon does not rewrite its bind logic. A protocol-aware adapter needs its own verified integration; there is no universal inherit-and-exec fix.

Check it quickly

The following inspection sketch assumes a separately configured, authorized foo.socket/foo.service pair serving the loopback health endpoint on port 8080. No such configuration is supplied here; this review did not execute the commands:

systemctl status foo.socket foo.service
ss -ltnp | grep ':8080'
curl -m 10 localhost:8080/health; systemctl show foo.service -p ActiveState,ExecMainStartTimestamp

Confirm the service is initially inactive, the intended socket is listening, and the request receives the expected application response. Compare invocation or start information before and after. Inspect each command’s result: the semicolon allows show to run even if curl fails, and status output alone does not prove health or activation causality.

Sources

systemd.socket manual, Official socket unit reference, checked 2026-06-01.
systemd.service manual, service activation, termination, and restart behavior, checked 2026-09-05.

systemd-tmpfiles applies creation and age policies

Fri, 10 Dec 2027 00:00:00 +0000

systemd-tmpfiles describes filesystem objects and housekeeping rules in configuration files. A rule can create a directory with a chosen mode, while an age rule can make old entries eligible for cleanup. The command must actually run the relevant operation for either effect to occur. This is useful lifecycle policy, but it is not a continuous watcher or a hard storage quota.

This tree describes the article’s specific d and x age-cleanup fixture. An x exclusion does not override explicit r or R deletion. A cleanup invocation, pathname deletion and storage release are different events; no cleanup was run for this artwork.

When it breaks

An age is not simply “time since the file was created” or “time since its contents last changed.” The documented default considers several timestamps, including access and modification, status change for files, and creation time where available. A recently changed qualifying timestamp can preserve an entry even when its mtime looks old. Rewriting timestamps for a test can therefore change the result in an unexpected direction.

--create and --clean perform different operations. Creating the configured directory does not prove that old contents have been removed. The d rule used below keeps its directory while making eligible contents subject to age cleanup. Other types have different removal semantics; do not substitute a recursive removal rule simply because both actions sound like cleaning.

An x exclusion preserves a matching directory and its contents during age-based cleanup, whereas X excludes the directory itself without the same recursive protection. These exclusions do not override explicit r or R removal rules. Read the effective configuration and operation together before deciding which paths a maintenance run can delete.

Check it quickly

Run this Python script as an administrator in a disposable Linux environment with systemd-tmpfiles. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It passes one explicit configuration file and a temporary --root, so all created objects and deletion targets belong to the exercise. The live system’s ordinary tmpfiles configuration is not used:

from pathlib import Path
from tempfile import TemporaryDirectory
import os,stat,subprocess,time

with TemporaryDirectory(prefix='xbcatm-tmpfiles-') as directory:
 root=Path(directory);config=root/'policy.conf'
 definition='d /work 0750 - - 1s -\nx /work/keep - - - - -\n'
 config.write_text(definition)
 def apply(action):
 result=subprocess.run(['systemd-tmpfiles','--root='+str(root),action,str(config)],capture_output=True,text=True,timeout=10)
 assert result.returncode==0,result.stderr
 apply('--create');work=root/'work'
 assert stat.S_IMODE(work.stat().st_mode)==0o750
 old=work/'old';old.write_text('fixture');past=time.time()-60;os.utime(old,(past,past))
 apply('--clean');assert old.exists()
 print('old atime and mtime alone do not satisfy the default age rule')
 config.write_text(definition.replace('1s','m:1s'));apply('--clean')
 assert not old.exists()
 print('explicit mtime-only age removes the same recently created file')
 config.write_text(definition);old.write_text('aged fixture')
 keep=work/'keep';keep.mkdir();(keep/'preserved').write_text('excluded fixture')
 outside=root/'outside';outside.write_text('outside cleanup tree')
 link=work/'link';link.symlink_to(outside)
 with old.open('rb') as retained:
 time.sleep(1.2)
 fresh=work/'fresh';fresh.write_text('recent fixture')
 apply('--clean')
 assert not old.exists() and not link.is_symlink()
 assert retained.read()==b'aged fixture'
 print('unlinked file remains readable through its already open descriptor')
 assert fresh.read_text()=='recent fixture'
 assert (keep/'preserved').read_text()=='excluded fixture'
 assert outside.read_text()=='outside cleanup tree'
 assert work.is_dir()
 print('aged entries removed; recent file and recursively excluded subtree remain')
 print('symlink removed without following it; target outside the cleaned tree survives')
print('all creation and cleanup occurred in a temporary alternate root')

The create operation produces /work inside the temporary root with mode 0750. A new file then receives deliberately old access and modification timestamps. Default cleanup leaves it present because other relevant timestamps are still recent. Merely backdating mtime and atime is not enough to make this file old under the complete default rule.

Changing only the age specification to m:1s makes modification time the selected file timestamp. Cleanup now removes that same recently created file. This proves that timestamp selection changes policy, not just display formatting. The script restores the default rule afterward; mtime-only aging is an intentional alternative to assess against the application’s access pattern, not an automatic improvement.

The next phase creates an ordinary file, an excluded subtree, and a symbolic link to another harmless file outside the cleaned tree. After waiting long enough for their timestamps to age, it creates a fresh file and runs cleanup. The aged ordinary entry and symlink disappear, while the recent file and excluded subtree remain. The symlink’s target is still intact, and the configured /work directory still exists.

The ordinary file is held open across cleanup. Although its pathname is gone, its existing descriptor can still read the original bytes until it is closed. Removing a directory entry and releasing the storage backing every open reference are different events. A process holding deleted files can consequently explain why a cleanup removed names without reclaiming all the expected space.

Assertions compare actual modes, path existence, and file contents. No test treats a zero exit status alone as proof that the intended deletion happened. The entire alternate root is removed afterward, leaving the host’s existing temporary files and normal policy untouched.

Match housekeeping to the producer

Inspect the distribution’s tmpfiles setup and cleanup units to establish when these rules are invoked. Eligibility at one second does not mean deletion occurs exactly one second later; the cleanup cadence and current workload matter. An inactive timer or failed cleanup command can leave eligible files in place indefinitely.

Age policies also cannot bound how many bytes a producer creates between cleanup runs. High write volume, recent accesses, exclusions, and open descriptors all affect retained space. Use appropriate storage limits and disk monitoring when capacity must be bounded, and diagnose the producing application instead of shortening retention until useful evidence disappears.

Keep service-owned runtime, cache, and persistent state conceptually separate. Data needed after restart should not accidentally inherit a temporary-file deletion policy. When managed service directories already express the required ownership and lifetime, reuse that mechanism. Add tmpfiles rules for a concrete lifecycle requirement and verify both eligible deletion and required preservation.

Sources

systemd 252 tmpfiles.d, creation types, timestamp-based aging, exclusions, and symbolic links, checked 2026-09-12.
systemd 252 systemd-tmpfiles, separate create/clean operations, alternate roots, and scheduled integration, checked 2026-09-12.

A systemd oneshot completes one activation

Sun, 28 Nov 2027 00:00:00 +0000

A Type=oneshot service represents finite work whose startup completes when its start commands finish successfully. That makes it useful for setup steps and scheduled jobs whose completion matters to a dependent unit. The word oneshot describes one activation’s process model. It does not create a permanent record that prevents the task from running again after a restart, another activation, or a reboot.

A oneshot completes one activation. Its earlier side effects survive later-command failure, and retained active/exited state can have no live main process. The three-command and PID 0 values illustrate the article’s fixture; no service was executed for this artwork.

When it breaks

A successful oneshot normally becomes inactive when its commands finish. That is expected behavior for a completed task, not evidence that a daemon crashed. With RemainAfterExit=yes, the manager instead keeps the unit active after the commands exit. An active (exited) state can therefore have no main process at all. Monitoring that assumes every active service needs a live PID will misclassify this service type.

Multiple ExecStart entries run in sequence for a oneshot. Under ordinary success rules, a failing command prevents later start commands from running and marks the activation failed. Earlier commands may already have changed files or databases. systemd does not roll those effects back, so retrying the activation can repeat work that succeeded before the failure.

RemainAfterExit is manager state, not a durable exactly-once guarantee. Starting an already active unit does not rerun it, but restarting it does. A timer targeting an active oneshot also does not turn each elapsed interval into another execution. Choose the retained state only when it represents a real ongoing condition, and keep repeatable scheduled work inactive between runs.

Check it quickly

Run this Python script as an administrator in a disposable Linux environment managed by systemd. It creates one runtime unit and a temporary step program. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. Ensure the example unit filename is unused. Each step records its name before the second step deliberately fails:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess

unit='xbcatm-oneshot-lab.service';path=Path('/run/systemd/system')/unit
assert not path.exists()
def ctl(*args,success=True):
 result=subprocess.run(['systemctl',*args],text=True,capture_output=True,timeout=10)
 assert (result.returncode==0)==success,(args,result.stderr)
 return result.stdout.strip()
def value(key):return ctl('show',unit,'-p',key,'--value')
with TemporaryDirectory(prefix='xbcatm-oneshot-',dir='/run') as directory:
 root=Path(directory);worker=root/'step.py';calls=root/'calls'
 worker.write_text("""from pathlib import Path
import sys
root=Path(__file__).parent
with (root/'calls').open('a') as output: output.write(sys.argv[1]+'\\n')
if sys.argv[1]=='second' and not (root/'healthy').exists(): sys.exit(7)
""")
 definition=f'''[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutStartSec=3s
ExecStart=/usr/bin/python3 {worker} first
ExecStart=/usr/bin/python3 {worker} second
ExecStart=/usr/bin/python3 {worker} third
'''
 path.write_text(definition)
 try:
 subprocess.run(['systemd-analyze','verify',str(path)],check=True)
 ctl('daemon-reload');ctl('start',unit,success=False)
 assert calls.read_text().splitlines()==['first','second']
 assert value('Result')=='exit-code' and value('ExecMainStatus')=='7'
 print('second command fails; third command does not run')
 (root/'healthy').touch();ctl('reset-failed',unit);ctl('start',unit)
 assert calls.read_text().splitlines()==['first','second','first','second','third']
 assert (value('ActiveState'),value('SubState'),value('MainPID'))==('active','exited','0')
 before=calls.read_text();ctl('start',unit);assert calls.read_text()==before
 print('active/exited with PID 0; repeated start does not rerun')
 ctl('restart',unit)
 assert calls.read_text()==before+'first\nsecond\nthird\n'
 print('restart executes all commands again')
 ctl('stop',unit);path.write_text(definition.replace('RemainAfterExit=yes','RemainAfterExit=no'))
 ctl('daemon-reload');before=calls.read_text()
 for _ in range(2):
 ctl('start',unit)
 assert value('ActiveState')=='inactive' and value('Result')=='success'
 assert calls.read_text()==before+'first\nsecond\nthird\n'*2
 print('without RemainAfterExit: two starts, two successful executions')
 finally:
 ctl('stop',unit)
 if value('ActiveState')=='failed':ctl('reset-failed',unit)
 path.unlink();ctl('daemon-reload')

The initial activation records only first and second, returns a failing start command, and exposes exit status seven. The third command does not run. The existing log entries remain, which makes the partial effects visible instead of pretending that a failed activation left no work behind.

After the healthy marker is created, the retry records all three steps. The complete file now contains the two earlier entries followed by a fresh sequence of three. State becomes active and exited, with main PID zero. That combination means the manager retains a successful activation, not that the program remains running in the background.

A second start while active leaves the file unchanged. Restarting the same unit appends another complete sequence. This is the direct counterexample to interpreting oneshot as “execute this migration once for the lifetime of the machine.” The manager is doing exactly what the new activation requests.

The script then stops the unit, changes RemainAfterExit to no, reloads the manager, and starts it twice. Both executions succeed and both end inactive. The log gains two full sequences. This behavior fits a job that should run whenever triggered and complete before another trigger can start it again.

The unit has an explicit three-second startup timeout. Oneshot startup timeouts are disabled by default in the documented version, so a stuck command otherwise may hold an activation open indefinitely. Choose a realistic bound for actual work rather than copying this small laboratory timeout into a long migration.

Give the task its own correctness guarantees

Keep each start command finite and in the foreground. If the task launches a long-running worker, describe that worker with an appropriate service lifecycle instead of inferring continued operation from the parent’s successful exit. Unit type, command completion, and the business result should describe the same event.

For a migration, use the database’s migration ledger, transaction behavior, and concurrency controls to decide whether a change has already been applied. A retained active state in one manager cannot coordinate multiple hosts or survive every operational transition. Make retries safe at the layer where side effects occur, and preserve the result needed to diagnose partial completion.

Inspect the service’s recorded result and application output, not just whether a timer or dependency triggered it. A timer can be active while its last job failed. Separate installation, scheduling, and execution guarantees, and verify a failed path as well as the successful one before depending on the task during boot or release.

Sources

systemd 252 service types, oneshot command ordering, retained state, and startup timeouts, checked 2026-09-12.
systemd 252 systemctl, start, restart, and manager reload behavior, checked 2026-09-12.
systemd 252 timers, activation of an already active target service, checked 2026-09-12.

Socket activation passes a listener to a compatible service

Mon, 08 Nov 2027 00:00:00 +0000

Socket activation lets systemd open a listening endpoint before the application process starts. Activity on that endpoint can then trigger a compatible service, which receives descriptors for sockets the manager already owns. The useful boundary is listener ownership versus application lifetime. An open endpoint is not yet proof that the application has initialized, accepted a request, or produced a correct response.

The active socket unit precedes a service with MainPID 0. A first connection starts a worker that validates LISTEN_PID, LISTEN_FDS 1 and listening FD 3, handles a framed request and replies. The worker deliberately exits; the manager retains the listener. A later connection starts a different worker using FD 3. Rebinding would get EADDRINUSE. Stopping the socket removes the path only because this fixture sets RemoveOnStop=yes. Illustration of selected source mechanisms; no example executed in this round.

When it breaks

The application must understand the activation protocol. A daemon that unconditionally binds the same endpoint again can fail because the manager already occupies it. Activation moves binding responsibility into the socket unit; it does not eliminate address conflicts, missing interfaces, filesystem permission errors, or other reasons that initial bind can fail.

With Accept=no, the service receives the listening socket and handles acceptance itself. Accept=yes instead supports a per-connection service-instance model for suitable stream sockets. These are different application contracts. Neither mode implies that an arbitrary daemon will automatically become idle, exit after work, or restart only when doing so is safe.

Queued connections also have limits. Clients can establish transport state before application startup finishes, then time out waiting for a response. Socket activation does not make database dependencies ready or create an unlimited request queue. Keep explicit service dependencies, readiness behavior, and client timeouts appropriate to the real workload.

Check it quickly

Run this Python script as an administrator in a disposable Linux system whose PID 1 is systemd. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It creates a unique socket/service pair using a Unix stream socket in a temporary directory. No public TCP port is exposed. The worker intentionally serves one newline-terminated message and exits:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,socket,subprocess,time,uuid

name='xbcatm-socket-'+uuid.uuid4().hex[:8];service=name+'.service';unit=name+'.socket'
paths={n:Path('/run/systemd/system')/n for n in (service,unit)}
assert all(not p.exists() for p in paths.values())
def run(*args):
 result=subprocess.run(args,capture_output=True,text=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def value(key):return run('systemctl','show',service,'-p',key,'--value')
def wait_inactive():
 deadline=time.monotonic()+5
 while value('ActiveState')!='inactive':
 assert time.monotonic()<deadline,'service did not exit';time.sleep(0.02)
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);endpoint=root/'listener.sock';worker=root/'worker.py'
 worker.write_text(r'''from pathlib import Path
import errno,json,os,socket
root=Path(__file__).parent
assert int(os.environ['LISTEN_PID'])==os.getpid()
assert int(os.environ['LISTEN_FDS'])==1
with socket.socket(fileno=3) as listener:
 assert listener.getsockopt(socket.SOL_SOCKET,socket.SO_ACCEPTCONN)==1
 with socket.socket(socket.AF_UNIX,socket.SOCK_STREAM) as duplicate:
 try:duplicate.bind(str(root/'listener.sock'))
 except OSError as error:assert error.errno==errno.EADDRINUSE
 else:raise AssertionError('unexpected duplicate bind success')
 connection,_=listener.accept()
 with connection,connection.makefile('rb') as incoming:
 message=incoming.readline(256);assert message.endswith(b'\n')
 connection.sendall(b'ack '+message)
 (root/(str(os.getpid())+'.json')).write_text(json.dumps({'pid':os.getpid(),'fd':3,'message':message.decode().strip()}))
''')
 paths[unit].write_text(f'[Socket]\nListenStream={endpoint}\nAccept=no\nSocketMode=0600\nRemoveOnStop=yes\n')
 paths[service].write_text(f'[Service]\nType=exec\nExecStart=/usr/bin/python3 {worker}\n')
 try:
 run('systemd-analyze','verify',*[str(p) for p in paths.values()]);run('systemctl','daemon-reload')
 run('systemctl','start',unit)
 assert endpoint.is_socket() and value('ActiveState')=='inactive' and value('MainPID')=='0'
 print('socket listening while service has no process')
 for message in ('first','second'):
 with socket.socket(socket.AF_UNIX,socket.SOCK_STREAM) as client:
 client.settimeout(5);client.connect(str(endpoint));client.sendall((message+'\n').encode())
 with client.makefile('rb') as response:assert response.readline()==('ack '+message+'\n').encode()
 wait_inactive();assert endpoint.is_socket() and value('Result')=='success'
 records=[json.loads(p.read_text()) for p in root.glob('*.json')]
 assert len(records)==2 and len({row['pid'] for row in records})==2
 assert {row['message'] for row in records}=={'first','second'}
 assert all(row['fd']==3 for row in records)
 print('two requests: two deliberate one-request workers; each uses inherited listening fd 3')
 print('rebinding the occupied socket fails; manager retains listener between workers')
 run('systemctl','stop',unit);assert not endpoint.exists()
 print('stopping socket removes its endpoint with RemoveOnStop=yes')
 finally:
 run('systemctl','stop',unit,service)
 for name,path in paths.items():
 if run('systemctl','show',name,'-p','ActiveState','--value')=='failed':run('systemctl','reset-failed',name)
 path.unlink()
 run('systemctl','daemon-reload')

After starting only the socket unit, the endpoint exists while the corresponding service is inactive and has no main PID. A first client connection then causes the worker to execute. The client sends a complete framed message and checks an application reply, rather than treating a successful connect call as proof of usable service.

The worker verifies that LISTEN_PID names its own process and LISTEN_FDS reports one inherited descriptor. It wraps descriptor three and checks that it is already listening. The numbering and environment checks match the native activation contract. A production program handling multiple descriptors should also identify and validate the sockets it actually expects.

Attempting another bind to the same Unix socket path produces EADDRINUSE. The worker continues using the inherited listener and returns the expected reply. This controlled failure demonstrates why an application cannot simply retain its ordinary bind path while assuming activation will somehow avoid the conflict.

After the worker exits successfully, the manager still holds the listening endpoint. A second connection launches another worker, which handles the second message and exits. The recorded PIDs are different and both invocations used descriptor three. Two processes appear because this specific worker deliberately exits after one request; Accept=no did not itself require one process per connection.

Stopping the socket unit removes the endpoint because the fixture explicitly sets RemoveOnStop=yes. The script then stops any remaining test units and removes their runtime definitions and temporary files. This cleanup behavior is configured and asserted by the fixture, not a universal promise that every socket path disappears whenever any associated daemon exits.

Design the handoff and shutdown together

For a long-running daemon, accept and serve the inherited sockets using its documented integration, and let its normal shutdown behavior decide when descriptors close. For a daemon intended to leave when idle, verify the idle-exit logic and what happens to in-flight requests. systemd cannot infer when an application’s work is safely complete.

Stopping future activation and terminating current work are separate operational decisions. Inspect both socket and service units during maintenance. A service may be started explicitly through another dependency or by an operator, and a still-active listener may trigger it again. Test the deployed unit relationships rather than assuming that stopping one visible process permanently disables the endpoint.

Set appropriate endpoint permissions and service identity. Passing an already-open descriptor allows listener setup and application privileges to differ, but it does not make the service a sandbox. Verify real request handling, restart behavior, queued-client outcomes, and failure diagnostics. The practical advantage is a deliberate, testable handoff between listener and process, not automatic readiness or zero resource cost.

Sources

systemd 252 systemd.socket, listener ownership, Accept modes, activation, and endpoint removal, checked 2026-09-12.
systemd 252 sd_listen_fds, inherited descriptor numbering, count, and process-identity checks, checked 2026-09-12.
systemd 252 systemd.service, service startup types and process lifecycle, checked 2026-09-12.

systemd restart policies need explicit failure and rate rules

Sun, 07 Nov 2027 00:00:00 +0000

A restart policy decides whether a service manager should try another process after the current one ends. It can recover from a transient failure, but it cannot correct a missing configuration or a deterministic bug. The useful contract has three parts: which endings qualify, how long to wait, and how many starts are permitted within a time window. Those settings answer different questions.

First classify the ending, then apply the fixed delay and start-window limit. The 100 ms, ten-second and three-start values come from the article’s systemd 252.39 fixture and are schematic, not a fresh run or recommended production values. Read the failure and repair its cause before resetting the limiter.

When it breaks

Restart=on-failure does not mean restart after every signal. For service types other than oneshot, ordinary SIGHUP, SIGINT, SIGTERM, and SIGPIPE terminations are normally considered clean. A nonzero exit or an unclean signal such as SIGKILL qualifies under the default success rules. SuccessExitStatus and the restart-specific exception directives can change classification, so inspect the effective unit.

on-abnormal is not simply a stricter synonym for on-failure: it excludes ordinary nonzero process exit codes while covering abnormal signals and certain timeout or watchdog failures. Choose according to how the program reports recoverable and intentional endings. A process that remains alive but stops doing useful work needs a separately configured detection mechanism; exit-based restart logic cannot infer that condition.

RestartSec supplies a delay before restarting. In the systemd 252 exercise below, it is a fixed delay, not an exponentially increasing backoff. StartLimitIntervalSec and StartLimitBurst limit starts within an interval, including manual starts. They are not a lifetime retry counter, and reaching the limit does not schedule an automatic recovery attempt at the next interval boundary.

Check it quickly

Run this Python script as an administrator in a disposable Linux systemd environment. It creates a uniquely named runtime unit and a temporary worker, then removes them. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. The worker initially exits with status seven and records every actual invocation:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,time,uuid

unit='xbcatm-restart-lab-'+uuid.uuid4().hex[:8]+'.service'
path=Path('/run/systemd/system')/unit
assert not path.exists()
def ctl(*args):
 result=subprocess.run(['systemctl',*args],text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def value(key):return ctl('show',unit,'-p',key,'--value')
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'state transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix='xbcatm-restart-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py';calls=root/'calls';healthy=root/'healthy'
 worker.write_text("""from pathlib import Path
import sys,time
root=Path(__file__).parent
with (root/'calls').open('a') as output: output.write('started\\n')
if not (root/'healthy').exists(): sys.exit(7)
time.sleep(60)
""")
 path.write_text(f'''[Unit]
StartLimitIntervalSec=10s
StartLimitBurst=3
[Service]
Type=exec
ExecStart=/usr/bin/python3 {worker}
Restart=on-failure
RestartSec=100ms
''')
 try:
 ctl('daemon-reload');ctl('start',unit)
 wait_for(lambda: value('ActiveState')=='failed')
 assert calls.read_text().splitlines()==['started']*3
 assert value('Result')=='exit-code'
 subprocess.run(['journalctl','--sync'],check=True)
 logs=subprocess.run(['journalctl','-u',unit,'--no-pager','-o','cat'],capture_output=True,text=True,check=True).stdout
 assert 'Start request repeated too quickly' in logs
 print('three starts; failed/exit-code with start-rate-limit journal message')
 healthy.touch();ctl('reset-failed',unit);ctl('start',unit)
 wait_for(lambda: len(calls.read_text().splitlines())==4)
 first=value('MainPID');assert first!='0'
 ctl('kill','--kill-who=main','--signal=TERM',unit)
 wait_for(lambda: value('ActiveState')=='inactive')
 assert value('NRestarts')=='0'
 print('on-failure treats ordinary SIGTERM as clean; no restart')
 ctl('start',unit);first=value('MainPID')
 ctl('kill','--kill-who=main','--signal=KILL',unit)
 wait_for(lambda: value('MainPID') not in ('0',first))
 assert value('NRestarts')=='1'
 print('SIGKILL triggers one restart')
 ctl('stop',unit);time.sleep(0.2)
 assert value('ActiveState')=='inactive'
 print('explicit stop remains stopped')
 finally:
 ctl('stop',unit)
 if value('ActiveState')=='failed':ctl('reset-failed',unit)
 path.unlink();ctl('daemon-reload')

The first phase permits three starts in ten seconds with a short fixed restart delay. The worker’s own file contains exactly three entries, so the count represents actual process launches rather than a counter whose meaning was assumed. The manager then leaves the unit failed and records that the start request was repeated too quickly.

The retained fixture expects terminal Result=exit-code alongside a start-limit journal message. The v252.39 source preserves an earlier non-success result in service_enter_dead; this supports that possibility, but does not reproduce the reported run. Use state, application invocation evidence, and the corresponding journal message together rather than relying on one expected diagnostic label across versions and failure paths.

The script repairs the fixture by creating its healthy marker, clears the failed state and rate counter, and starts it again. This time the process remains alive. Sending an ordinary SIGTERM produces an inactive unit with no automatic restart under on-failure. A later SIGKILL does cause one replacement process.

Finally, an explicit systemctl stop leaves the service inactive after the restart delay. Manager-requested shutdown is not treated as a crash that policy should immediately reverse. The cleanup handles failed state only when present, since an inactive unreferenced unit can be unloaded by the manager.

Recover the service, not just its counter

Read the failure before resetting its evidence. Inspect the configured command, exit status, recent invocation logs, and any dependency errors. Correct the cause or deploy a known working version, then reset the limiter if an immediate retry is justified. Repeatedly clearing the counter without changing the failure simply converts a bounded crash loop into an operator-driven one.

Alert on sustained unavailability and excessive restarts using the monitoring system actually responsible for the service. The unit settings shown here do not create an alert destination. Retain enough diagnostic context to distinguish failed execution, an application exit, resource exhaustion, and a deliberate stop without logging sensitive configuration.

Select the delay and burst to match startup cost and dependency behavior. A rapid retry may recover quickly from one fault but amplify load during a shared outage. Verify both recovery and termination in the deployed configuration, including any drop-ins. A policy is useful when the application resumes useful work and remains controllable, not merely when a PID repeatedly changes.

Sources

systemd 252 service restart rules, exit classification, fixed delay, and restart exceptions, checked 2026-09-12.
systemd 252 unit start limits, interval limits, manual starts, and counter reset, checked 2026-09-12.
systemd 252 systemctl, stop, state inspection, and reset-failed operations, checked 2026-09-12.
systemd-stable v252.39 service source, service_enter_dead retains an earlier non-success result; source evidence only, checked 2026-10-19.

The systemd journal needs an explicit retention contract

Tue, 26 Oct 2027 00:00:00 +0000

The systemd journal stores messages with structured metadata in journal files. That makes service output searchable without first inventing a parser for every application’s text format. Keeping those messages for a useful period is a separate requirement. Storage location, size policy, rotation, and forwarding determine which evidence remains available when an incident is investigated later.

Storage path and retention are separate questions. Rotation closes the old file as an archive and opens a replacement; vacuum can delete archives, not active files. A journald service restart does not establish reboot or power-loss durability, and a byte target does not guarantee a fixed evidence window. This is a source model, not a new execution result.

When it breaks

The default namespace normally uses Storage=auto: persistent storage is selected when /var/log/journal exists, otherwise runtime storage is used. Storage=volatile writes beneath /run/log/journal; this is not simply the kernel’s text ring buffer. Those runtime files normally disappear across a machine reboot, but restarting the journald service is a different event and need not erase them.

Storage=persistent prefers /var/log/journal, with runtime storage used during early boot or when persistent storage cannot be written. A flush operation transfers runtime data under the documented conditions. Inspect actual files and effective configuration instead of inferring location solely from the presence of an option in one configuration fragment.

journald has default size limits and rotation behavior. It does not grow without bounds merely because no administrator wrote a rotation job. However, SystemMaxUse is not an exact hard quota that guarantees the filesystem can never fill. Runtime storage has separate settings, free-space constraints also apply, other programs consume space, and only archived journal files can be deleted during vacuuming.

Check it quickly

Run this Python script as an administrator in a disposable Linux system managed by systemd. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It creates a unique journal namespace, gives it an explicit persistent-storage configuration, emits two harmless messages through transient services, and removes only that namespace’s data afterward. The vacuum operations deliberately delete the experiment’s old message:

from pathlib import Path
import json,shutil,subprocess,uuid

name='xbcatm-journal-'+uuid.uuid4().hex[:8]
config=Path('/etc/systemd')/('journald@'+name+'.conf')
directory=Path('/var/log/journal')/(Path('/etc/machine-id').read_text().strip()+'.'+name)
assert not config.exists() and not directory.exists()
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=15)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout
def journal(*args):return run('journalctl','--namespace='+name,*args)
def emit(message):
 run('systemd-run','--quiet','--wait','--collect','--unit='+name+'-'+message,
 '--property=LogNamespace='+name,'--property=StandardOutput=journal',
 '--property=SyslogIdentifier='+name,'/usr/bin/printf',message+'\n')
 journal('--sync')
def messages():
 rows=journal('--no-pager','-o','json','SYSLOG_IDENTIFIER='+name)
 return [json.loads(line)['MESSAGE'] for line in rows.splitlines()]
config.write_text('[Journal]\nStorage=persistent\nSystemMaxUse=16M\nRuntimeMaxUse=8M\n')
try:
 emit('old');journal('--flush');journal('--sync')
 assert messages()==['old'] and list(directory.glob('*.journal'))
 run('systemctl','restart','systemd-journald@'+name+'.service')
 assert messages()==['old']
 print('persistent namespace: message survives journald service restart')
 journal('--vacuum-size=1')
 assert messages()==['old']
 print('vacuum cannot remove the active journal file to satisfy a one-byte target')
 journal('--rotate');emit('new')
 assert messages()==['old','new']
 assert list(directory.glob('*@*.journal'))
 journal('--vacuum-size=1')
 assert messages()==['new'] and not list(directory.glob('*@*.journal'))
 print('after rotation: archived old message removed; active new message remains')
 print('persistent files observed under /var/log/journal; no reboot durability claim')
finally:
 run('systemctl','stop','systemd-journald@'+name+'.service',
 'systemd-journald@'+name+'.socket','systemd-journald-varlink@'+name+'.socket')
 config.unlink(missing_ok=True)
 for path in (directory,Path('/run/log/journal')/directory.name,Path('/run/systemd')/('journal.'+name)):
 if path.exists():shutil.rmtree(path)

The first message is present in journal files beneath the namespace’s /var/log/journal directory. After restarting that namespace’s journald service, the message is still queryable. This verifies the selected storage path and survival across a daemon restart. It does not simulate a power failure, prove storage-device durability, or test a complete machine reboot.

Vacuuming to a one-byte target initially leaves the message visible because its file is still active. The requested size is too small, but vacuum is not permitted to delete that active file just to meet the number. This provides a concrete counterexample to treating a vacuum target as a guaranteed final disk-usage figure.

The script then rotates the journal and writes a new message. Rotation closes the earlier file as an archive and opens a replacement. Both messages are readable before the second vacuum. Afterward, the archived old message is gone while the new message in the active file remains. Rotation and deletion are separate operations with different effects on evidence.

All journal commands select the unique namespace, so they do not vacuum the system’s default journal. Namespace service and socket instances are stopped before their test directories are removed. A normal query without the namespace selector would look in the default namespace and could miss these records entirely, even though they were successfully stored.

Size the evidence window

Choose retention from the incident-response window and observed message volume. A size limit cannot promise a fixed number of days when traffic changes. A retention-age limit likewise does not reserve enough bytes to preserve every message until that age. Monitor actual disk use and the earliest available records, especially after an unusually noisy failure.

Log collection can also lose messages before retention becomes relevant. Rate limits, application output routing, permissions, and unavailable storage affect what reaches the journal. Preserve the distinction between “no matching stored record” and “the event never occurred.” Avoid putting credentials into messages merely because local journal access is restricted.

If records must survive loss of the machine, provide and verify an independent collection path. Forwarding configuration alone is not evidence that a remote destination received or retained the message. Check delivery, access control, and the remote retention window using an identifiable harmless event. Local searchable logs and durable external evidence answer related but different operational needs.

Sources

systemd 252 journald.conf, storage selection, size defaults, free-space limits, and retention, checked 2026-09-12.
systemd 252 journalctl, namespace selection, rotation, synchronization, and archive vacuuming, checked 2026-09-12.
systemd 252 systemd-journald.service, journal namespaces, transport, and persistent file locations, checked 2026-09-12.

Query the journal with explicit field and priority rules

Fri, 30 Jul 2027 00:00:00 +0000

A useful journal query selects the records that answer a specific incident question. Unit identity, invocation, time, priority, and application fields provide different dimensions. Combining them correctly matters more than finding a familiar word somewhere in the machine’s history. journald can attach process metadata, but it does not automatically collect every file an application writes or infer the business meaning of each message.

Different fields are AND conditions; repeated values of one field are OR alternatives, and a separate + joins groups with OR. In the article’s first-run sample, -p err selects API priorities 2 and 3, while PRIORITY=3 selects only 3. Unit identity is broader than one invocation. Empty output can be a successful query and does not prove an event never occurred. No runtime result is newly claimed.

When it breaks

Field matches have a particular Boolean grammar. Different fields are combined with AND. Multiple matches for the same field are alternatives combined with OR. A separate + joins groups with OR. Consequently, adding another value for a field can broaden a result rather than narrow it. Treat the query as a predicate and check its result against known records before using it in an incident script.

-p err includes error and more severe priorities. It does not mean only priority three. Use an exact PRIORITY=3 field match when that is the intended condition. Application text saying “error” is not itself a priority assignment; stdout collection uses configured priority rules unless the application supplies a supported priority representation.

A successful command with no returned records is not proof that the service never ran. The selected boot, time window, namespace, reader permissions, retention, or output routing might explain the absence. Conversely, querying an entire unit history can find a previous successful startup and mistakenly attribute it to the current process. Separate unit identity from one specific invocation.

Check it quickly

Run this Python script as an administrator in a disposable Linux system whose PID 1 is systemd. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It installs a uniquely named temporary service and activates it twice. Each worker submits four short structured datagrams to the native journal socket, with two application components and several explicit priorities:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid

name='xbcatm-query-'+uuid.uuid4().hex[:8];unit=name+'.service'
path=Path('/run/systemd/system')/unit
assert not path.exists()
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def query(*matches):
 output=run('journalctl','--no-pager','-o','json','-u',unit,*matches)
 return [json.loads(line) for line in output.splitlines() if line]
def labels(rows):return sorted(row['MESSAGE'] for row in rows)
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py'
 worker.write_text(r'''from pathlib import Path
import socket
phase=Path(__file__).with_name('phase').read_text()
with socket.socket(socket.AF_UNIX,socket.SOCK_DGRAM) as log:
 log.connect('/run/systemd/journal/socket')
 for component,priority in [('api',2),('api',3),('api',6),('db',3)]:
 fields=[f'MESSAGE={phase}-{component}-{priority}',f'PRIORITY={priority}',
 f'LAB_RUN={phase}',f'LAB_COMPONENT={component}',
 '_SYSTEMD_UNIT=forged.service']
 log.sendall(('\n'.join(fields)+'\n').encode())
''')
 path.write_text(f'[Service]\nType=oneshot\nExecStart=/usr/bin/python3 {worker}\n')
 try:
 run('systemd-analyze','verify',str(path));run('systemctl','daemon-reload')
 for phase in ('first','second'):
 (root/'phase').write_text(phase);run('systemctl','start',unit)
 run('journalctl','--sync')
 first=query('LAB_RUN=first')
 assert len(first)==4 and all(row['_SYSTEMD_UNIT']==unit for row in first)
 second=query('LAB_RUN=second');assert len(second)==4
 invocation=first[0]['_SYSTEMD_INVOCATION_ID']
 assert {row['_SYSTEMD_INVOCATION_ID'] for row in first}=={invocation}
 assert {row['_SYSTEMD_INVOCATION_ID'] for row in second}!={invocation}
 assert labels(query('LAB_RUN=first','LAB_COMPONENT=api','-p','err'))==['first-api-2','first-api-3']
 print('different fields AND; -p err includes critical priority 2 and error priority 3')
 assert labels(query('LAB_RUN=first','LAB_COMPONENT=api','PRIORITY=3'))==['first-api-3']
 assert labels(query('LAB_RUN=first','LAB_COMPONENT=api','LAB_COMPONENT=db','PRIORITY=3'))==['first-api-3','first-db-3']
 print('exact PRIORITY=3 differs from threshold; repeated component fields OR')
 union=query('LAB_RUN=first','LAB_COMPONENT=db','+','LAB_RUN=second','LAB_COMPONENT=db')
 assert labels(union)==['first-db-3','second-db-3']
 assert labels(query('_SYSTEMD_INVOCATION_ID='+invocation,'LAB_COMPONENT=db'))==['first-db-3']
 print('explicit + joins match groups; invocation filter separates repeated starts')
 assert query('LAB_RUN=absent')==[]
 print('empty query still succeeds; underscore unit field comes from journal metadata')
 finally:
 run('systemctl','stop',unit)
 if run('systemctl','show',unit,'-p','ActiveState','--value')=='failed':run('systemctl','reset-failed',unit)
 path.unlink();run('systemctl','daemon-reload')

The first activation produces an API critical message, an API error, an API informational message, and a database error. The second activation produces the same categories with a different application run label. Filtering by the first run, API component, and -p err returns two records: priorities two and three. Switching to exact priority three returns only the API error.

Repeating the component field with API and database values returns both components’ error records for the first run. It does not demand that one record simultaneously have both values. The explicit + example then selects the database record from either run. The assertions check complete expected message lists, so an accidentally broad filter cannot pass by merely containing one desired phrase.

The service’s two activations have different journal invocation identifiers. Filtering with the first identifier selects only that activation’s database message, even though the unit name is unchanged. This is useful during crash loops and repeated manual starts: the unit groups a service’s history, while an invocation identifies one execution lifecycle.

The worker also attempts to submit a forged underscore-prefixed unit field. Returned records carry the actual service unit instead. Those trusted fields are supplied by the journal from process context rather than accepted as ordinary application labels. In contrast, the custom component and run fields are application claims; the experiment controls them, but a production parser should not treat their names as an independent authentication mechanism.

Finally, a match for an absent run returns an empty list while the query command succeeds. The script explicitly checks this case. It removes the temporary service and worker files afterward; the eight harmless journal entries remain subject to the host’s normal retention policy, rather than deleting unrelated system history to erase a small fixture.

Preserve context when exporting results

Use structured output when another program consumes the query. The sample parses JSON and its deliberately small textual fields. General journal JSON can represent repeated values or binary data differently, so define the consumer’s accepted schema rather than assuming every arbitrary field is always one simple string. Keep timestamps and attribution fields when exporting evidence for later comparison.

Record the relevant boot or invocation and make time-zone assumptions explicit for time-window queries. Reader access and selected namespace are part of the observation, not incidental details. Before concluding that an application event is missing, verify that a known harmless event from the same route is visible. A bounded, reproducible query should make both its selection rule and its limits clear.

Sources

systemd 252 journalctl, field-match Boolean rules, priority filters, and JSON output, checked 2026-09-12.
systemd 252 systemd.journal-fields, trusted metadata, application fields, and invocation identifiers, checked 2026-09-12.
systemd 252 systemd-journald.service, native logging transport, namespace separation, and collection scope, checked 2026-09-12.

A systemd timer schedules a service, not its success

Mon, 21 Jun 2027 00:00:00 +0000

A systemd timer separates the scheduling rule from the service that performs the work. The timer decides when to request activation; the service supplies the command, execution identity, environment, timeout, and result. This separation makes a maintenance job inspectable using normal unit tools. It does not turn a scheduled attempt into proof that a backup, cleanup, or report completed correctly.

Timer state does not prove service success. The source fixture’s completion-relative ordering is shown without a measured time scale or exact launch-time promise. Stopping future scheduling and stopping current work are separate decisions; the historical fixture was not rerun.

When it breaks

An active timer can coexist with a failed service. Monitoring only the timer’s active state therefore misses failures in the actual work. Likewise, a recent trigger timestamp identifies scheduling activity, not successful business output. Check the service’s result and the artifact or state change the job was supposed to produce, with enough context to distinguish successive activations.

A timer normally activates the service with the matching basename, unless Unit selects another target. If that service is already active, the timer does not launch another independent copy. This prevents overlapping activations of that one unit, but it is not a distributed lock or protection against another service running the same command. A oneshot kept active by RemainAfterExit=yes can consequently prevent later scheduled executions.

Stopping a timer stops future scheduling. It does not automatically stop a service invocation already in progress. That distinction matters when pausing maintenance during an incident: inspect the target service and decide deliberately whether current work should finish or be interrupted. Interrupting a command also does not roll back its earlier filesystem or database changes.

Check it quickly

Run this Python script as an administrator in a disposable Linux system with systemd as PID 1. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It creates a unique timer/service pair and a temporary worker. The worker records real process entries and completion times; explicit release files hold successful jobs open so the scheduling behavior is observable:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,time,uuid

name='xbcatm-timer-'+uuid.uuid4().hex[:8];service=name+'.service';timer=name+'.timer'
paths={unit:Path('/run/systemd/system')/unit for unit in (service,timer)}
assert all(not p.exists() for p in paths.values())
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def value(unit,key):return run('systemctl','show',unit,'-p',key,'--value')
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py'
 worker.write_text(r'''from pathlib import Path
import json,sys,time
root=Path(__file__).parent;number=len(list(root.glob('start-*.json')))+1
def record(kind):
 temp=root/'record.tmp';temp.write_text(json.dumps(time.monotonic()))
 temp.replace(root/(kind+'-'+str(number)+'.json'))
record('start')
if (root/'fail').exists():sys.exit(7)
while not (root/('release-'+str(number))).exists():time.sleep(0.01)
record('end')
''')
 paths[service].write_text(f'[Service]\nType=oneshot\nTimeoutStartSec=10s\nExecStart=/usr/bin/python3 {worker}\n')
 definition='[Timer]\nOnActiveSec=100ms\nOnUnitInactiveSec=10s\nAccuracySec=1ms\n'
 paths[timer].write_text(definition);(root/'fail').touch()
 try:
 run('systemd-analyze','verify',*[str(p) for p in paths.values()])
 run('systemctl','daemon-reload');run('systemctl','start',timer)
 wait_for(lambda:value(service,'ActiveState')=='failed')
 assert value(service,'ExecMainStatus')=='7' and value(timer,'ActiveState')=='active'
 run('systemctl','stop',timer)
 print('timer active while its first job failed with exit status 7')
 (root/'fail').unlink();run('systemctl','reset-failed',service)
 paths[timer].write_text(definition.replace('OnUnitInactiveSec=10s','OnUnitInactiveSec=250ms'))
 run('systemctl','daemon-reload');run('systemctl','start',timer)
 wait_for(lambda:(root/'start-2.json').exists())
 time.sleep(0.65)
 assert len(list(root.glob('start-*.json')))==2
 (root/'release-2').touch();wait_for(lambda:(root/'start-3.json').exists())
 gap=json.loads((root/'start-3.json').read_text())-json.loads((root/'end-2.json').read_text())
 assert gap>=0.25,gap
 print('no overlapping launch; next job starts at least 250ms after prior deactivation')
 pid=value(service,'MainPID');assert pid!='0'
 run('systemctl','stop',timer)
 assert value(service,'MainPID')==pid and value(service,'ActiveState')=='activating'
 print('stopping timer leaves its already running oneshot alive')
 (root/'release-3').touch();wait_for(lambda:value(service,'ActiveState')=='inactive')
 time.sleep(0.4)
 assert len(list(root.glob('start-*.json')))==3 and value(service,'Result')=='success'
 print('released job completes; stopped timer schedules no further run')
 finally:
 run('systemctl','stop',timer,service)
 for unit,path in paths.items():
 if value(unit,'ActiveState')=='failed':run('systemctl','reset-failed',unit)
 path.unlink()
 run('systemctl','daemon-reload')

The first timer activation launches a worker that deliberately exits with status seven. The service becomes failed while the timer remains active. Its next completion-relative interval is initially ten seconds, allowing the script to inspect this state and stop the timer before another attempt. The output confirms a scheduler can be functioning while the scheduled work is failing.

After correcting the fixture’s failure condition, the script clears the failed state, changes the interval to 250 milliseconds, and starts scheduling again. The second worker waits at its release gate. Even after more than two nominal intervals pass, no third worker has started. OnUnitInactiveSec is measured from deactivation, so unfinished work has not yet supplied that reference event.

Releasing the second worker lets it complete. The third worker then starts at least one configured interval after the recorded completion marker; actual unit deactivation occurs after that marker. The assertion verifies the completion-relative relationship without requiring a fragile exact launch timestamp. AccuracySec=1ms reduces coalescing in this short exercise, but it is not a real-time execution guarantee.

The script stops the timer while the third worker is still waiting. Its PID remains unchanged and the oneshot remains activating. Only releasing that worker lets the service complete successfully. After another interval has passed, the invocation count is unchanged, confirming that the stopped timer no longer schedules work. Both runtime unit files and the temporary worker are removed afterward.

Migrate the execution contract too

When moving a cron job, inspect more than its time expression. Use the intended working directory, explicit command paths, required environment, suitable user, and an appropriate runtime limit. systemd command lines are not implicitly shell programs. A pipeline, redirection, or shell expansion needs an intentional shell invocation or a separate script whose behavior is tested.

Installation and activation are also separate. Starting a runtime timer once does not enable it at boot, and enabling the service itself can introduce an unintended startup run. Install and enable the timer according to the deployment’s policy, then verify the loaded definitions and next activation. Keep the service independently runnable for controlled diagnosis.

Choose how failures should be retried and reported instead of assuming scheduling provides that policy. A later timer activation may try again, while service restart settings introduce their own rules and limits. Make repeated side effects safe and preserve a last-success signal tied to the actual result. The operational objective is completed useful work within its required window, not merely a timer that remains active.

Sources

systemd 252 systemd.timer, target selection, active-target behavior, and completion-relative intervals, checked 2026-09-12.
systemd 252 systemd.service, oneshot lifecycle, retained state, execution failure, and timeouts, checked 2026-09-12.
systemd 252 systemctl, starting, stopping, enabling, and inspecting timer and service units, checked 2026-09-12.

systemd-sysusers creates missing service accounts

Tue, 08 Jun 2027 00:00:00 +0000

A service package often needs a dedicated account before it can own files or run without root. systemd-sysusers describes that account and its groups in a small configuration file. Its central operation is creating missing identities and establishing requested memberships. It is not a general account migration engine, a directory creator, or a guarantee that automatically allocated numeric IDs match on every host.

sysusers creates missing identities and requested memberships. It does not migrate existing account fields, guarantee equal automatic UIDs across hosts, or create home directories. The diagram is source-based; no account database was changed.

When it breaks

A u entry creates a missing system user and normally its matching primary group. A g entry creates a group, and m adds a supplementary membership. An ID of - requests automatic allocation. An r entry adds a range to the allocation pool; multiple ranges combine. An occupied number changes what remains available. The same pool therefore does not establish cross-machine identity equality.

Changing a declaration for an existing account does not automatically renumber that account or update its home directory and descriptive fields. This behavior makes repeated package setup useful, but it also means a configuration diff is not proof that live account records changed. A real UID migration needs a separate plan for existing ownership, running processes, shared storage, and rollback.

The home-directory column sets an account-database field; it does not create the directory. A service can consequently have a valid account and still fail because its state path is missing or unwritable. Use an appropriate managed service directory or a tmpfiles rule when a directory must be created with defined ownership and permissions.

Check it quickly

Run this Python script as an administrator in a disposable Linux environment containing systemd-sysusers. It does not need to alter the machine’s live account database. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. Each test operates on files inside a separate temporary root selected with --root; shadow database contents are compared internally and never printed:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess

definition='''r - 61000-61010
u field-worker - "Field worker" /var/lib/field-worker /usr/sbin/nologin
g field-readers -
m field-worker field-readers
'''
def prepare(root,occupied=None):
 (root/'etc').mkdir(parents=True)
 passwd='root:x:0:0:root:/root:/bin/sh\n';group='root:x:0:\n'
 if occupied is not None:
 passwd+=f'occupied:x:{occupied}:{occupied}:Fixture:/nonexistent:/usr/sbin/nologin\n'
 group+=f'occupied:x:{occupied}:\n'
 (root/'etc/passwd').write_text(passwd);(root/'etc/group').write_text(group)
 directory=root/'usr/lib/sysusers.d';directory.mkdir(parents=True)
 (directory/'field.conf').write_text(definition)
def apply(root):
 result=subprocess.run(['systemd-sysusers','--root='+str(root)],capture_output=True,text=True,timeout=10)
 assert result.returncode==0,result.stderr
def entries(root,name):
 return {line.split(':')[0]:line.split(':') for line in (root/'etc'/name).read_text().splitlines()}
with TemporaryDirectory(prefix='xbcatm-sysusers-') as directory:
 first=Path(directory)/'first';second=Path(directory)/'second';prepare(first);apply(first)
 worker=entries(first,'passwd')['field-worker'];groups=entries(first,'group')
 assert worker[3]==groups['field-worker'][2]
 assert 'field-worker' in groups['field-readers'][3].split(',')
 assert worker[5:]==['/var/lib/field-worker','/usr/sbin/nologin']
 assert not (first/'var/lib/field-worker').exists()
 before={name:(first/'etc'/name).read_bytes() for name in ('passwd','group','shadow','gshadow')}
 apply(first)
 assert before=={name:(first/'etc'/name).read_bytes() for name in before}
 print('user, primary group, and supplementary membership created; repeat leaves databases unchanged')
 print('home and nologin shell recorded; home directory was not created')
 prepare(second,int(worker[2]));apply(second)
 other=entries(second,'passwd')['field-worker']
 assert other[2]!=worker[2]
 print('same allocation range and definition choose different UIDs when existing accounts differ')
 changed=definition.replace('u field-worker - "Field worker" /var/lib/field-worker',
 'u field-worker 62000 "Changed description" /changed-home')
 (first/'usr/lib/sysusers.d/field.conf').write_text(changed);apply(first)
 assert entries(first,'passwd')['field-worker']==worker
 print('changing a declaration does not renumber or rewrite an existing account')
 assert not (first/'changed-home').exists()
print('only temporary alternate-root account databases were changed')

The first root begins with only its fixture root account. Applying the declaration creates the worker, its primary group, the readers group, and the worker’s supplementary membership. The script checks the numeric primary-group relationship and actual membership list. A successfully parsed configuration alone would not establish either relationship.

The account record contains the requested home path and a nologin shell, but that home directory does not exist. This directly separates identity declaration from filesystem provisioning. An account intended for service execution can still be used by an authorized service manager; a nologin shell is not a replacement for all permission and credential controls.

Applying the same declaration again leaves the passwd, group, shadow, and gshadow file contents unchanged. This verifies repeat behavior for the exact fixture instead of assuming that every imaginable declaration is harmless. Conflicting input and allocation exhaustion still need diagnosis when the command reports them.

The second root reserves the first worker’s numeric identity for another fixture account before applying the identical configuration. Its worker receives a different UID. Names and requested allocation range match, but existing database state differs. Copying numeric file ownership between these roots without an identity mapping would therefore be unsafe.

Finally, the first root’s declaration requests a new numeric UID, description, and home path for the existing worker. Reapplying leaves its complete account record unchanged and creates no new home. This is the expected boundary of creation-oriented setup, not evidence that the edited fields were successfully migrated. The temporary roots are removed when the exercise ends.

Make packaging and identity policy explicit

Packages normally place declarations under /usr/lib/sysusers.d, while administrator files can override matching filenames under /etc/sysusers.d. Filename ordering and conflicting names matter; declarations are not universally order-independent statements. Inspect the effective input when more than one package or local policy defines the same identity.

Writing a file does not execute sysusers by itself. Distribution boot units and package integration determine when the tool is invoked. Verify the relevant installation path and service dependency so a daemon does not start before its account or state directory exists. Keep account creation and directory provisioning coordinated without treating them as the same operation.

Use automatically allocated IDs when host-local identity is sufficient. If shared storage or a published interface requires a particular numeric identity, establish and verify that allocation policy explicitly, including collisions and existing accounts. Test the actual service’s access after installation. Stable names, stable numbers, and correct filesystem ownership are related requirements that each need evidence.

Sources

systemd 252 sysusers.d, user/group directives, allocation ranges, file precedence, and home-directory fields, checked 2026-09-12.
systemd 252 systemd-sysusers, account creation, repeat execution, and alternate-root processing, checked 2026-09-12.
systemd 252 tmpfiles.d, separate creation and ownership of service directories, checked 2026-09-12.

Linger extends a systemd user manager beyond login

Tue, 20 Apr 2027 00:00:00 +0000

Linger changes when an account’s systemd user manager is kept available. With lingering enabled, logind can start that manager at boot and retain it after the account’s login sessions end. This supports unattended work owned by a normal account. The flag controls manager lifetime; individual services still have their own activation, failure, shutdown, and installation rules.

Linger controls manager lifetime. Unit activation is separate, and manager restart creates fresh processes from enabled units. PID A and PID B are schematic identities, not measured values. This artwork explains the source; the historical runtime check remains blocked.

When it breaks

Enabling linger does not enable every unit file found in the account’s configuration directory. A service must be requested through enablement, a dependency, a timer, a socket, or another explicit activation path. A manager can be running correctly while the intended application remains inactive. Check both objects instead of interpreting the linger flag as a list of running services.

Processes do not survive a machine reboot. Linger can arrange for a new user manager to start at boot, and enabled units can then start new processes. Their previous in-memory state is gone. An encrypted or late-mounted home can also make unit files or application data unavailable at the time the manager starts. Persist required state and make boot-time dependencies intentional.

Linger is not an instruction to restart a crashed application forever or to adopt arbitrary shell jobs into managed services. Restart policy, resource limits, and logind’s treatment of session processes are separate controls. A stable background role needs a real user unit and an observed result, not just a shell command launched before logout.

Check it quickly

Run this Python script as an administrator in a disposable Linux system with systemd, logind, user D-Bus support, and account-management tools. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It creates a temporary account and home, enables lingering only for that account, and removes its changes afterward. No login session is opened for the account. The final 20-second wait is a fixture timeout, not a promised stop delay. This no-login experiment does not establish last-logout UserStopDelaySec behavior:

from pathlib import Path
from tempfile import TemporaryDirectory
import os,pwd,subprocess,time,uuid

name='xbcatm-linger-'+uuid.uuid4().hex[:8];unit='field-worker.service'
def run(*args):
 result=subprocess.run(args,capture_output=True,text=True,timeout=20)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def wait_for(check,seconds=5):
 deadline=time.monotonic()+seconds
 while not check():
 assert time.monotonic()<deadline,'state transition timed out';time.sleep(0.05)
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);root.chmod(0o755);home=root/'home';home.mkdir()
 run('useradd','--no-create-home','--user-group','--home-dir',str(home),'--shell','/bin/sh',name)
 account=pwd.getpwnam(name);uid=account.pw_uid;gid=account.pw_gid;manager=f'user@{uid}.service'
 def userctl(*args):
 return run('setpriv','--reuid='+str(uid),'--regid='+str(gid),'--init-groups',
 'env',f'XDG_RUNTIME_DIR=/run/user/{uid}',f'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{uid}/bus',
 'systemctl','--user',*args)
 def manager_value(key):return run('systemctl','show',manager,'-p',key,'--value')
 try:
 units=home/'.config/systemd/user';units.mkdir(parents=True)
 (home/'worker.py').write_text("from pathlib import Path\nimport os,time\np=Path(__file__).with_name('worker-pid');t=p.with_suffix('.tmp');t.write_text(str(os.getpid()));t.replace(p)\ntime.sleep(60)\n")
 (units/unit).write_text('[Service]\nType=exec\nExecStart=/usr/bin/python3 %h/worker.py\n[Install]\nWantedBy=default.target\n')
 for path in [home,*home.rglob('*')]:os.chown(path,uid,gid)
 run('loginctl','enable-linger',name)
 wait_for(lambda:Path(f'/run/user/{uid}/bus').exists())
 assert run('loginctl','show-user',name,'-p','Linger','--value')=='yes'
 assert run('loginctl','show-user',name,'-p','Sessions','--value')==''
 assert manager_value('ActiveState')=='active'
 assert userctl('show',unit,'-p','ActiveState','--value')=='inactive'
 print('linger starts a manager with zero login sessions; merely installing a user unit does not start it')
 userctl('enable','--now',unit);wait_for(lambda:(home/'worker-pid').exists())
 first=(home/'worker-pid').read_text();first_manager=manager_value('MainPID')
 run('systemctl','restart',manager)
 wait_for(lambda:Path(f'/run/user/{uid}/bus').exists() and (home/'worker-pid').read_text()!=first)
 assert manager_value('MainPID')!=first_manager
 assert userctl('show',unit,'-p','MainPID','--value')==(home/'worker-pid').read_text()
 assert run('loginctl','show-user',name,'-p','Sessions','--value')==''
 print('enabled user service starts as a new process after a real manager restart')
 userctl('disable',unit)
 assert userctl('show',unit,'-p','ActiveState','--value')=='active'
 print('disable removes future enablement but does not stop the current service')
 userctl('stop',unit);run('loginctl','disable-linger',name)
 wait_for(lambda:manager_value('ActiveState')=='inactive',seconds=20)
 print('with no sessions and linger disabled, user manager stops after policy delay')
 finally:
 run('loginctl','disable-linger',name)
 run('systemctl','stop',manager,f'user-runtime-dir@{uid}.service')
 run('userdel',name)

After enable-linger, the user manager becomes active and its bus appears. logind reports lingering enabled and an empty session list. However, the installed worker unit is still inactive. This confirms that keeping the manager available is different from requesting the application to run, even though the configuration file already exists.

The script then enables and starts the worker through the correct user manager. Its installation section links it to the user’s default target, while the explicit start begins current execution. A PID marker identifies the actual worker. The controller uses setpriv and the user’s runtime bus, so the control command itself does not quietly create a login session.

Restarting the system-level user@UID.service replaces the account manager. The enabled worker subsequently publishes a different PID, and the new user manager reports that same process as its main worker. The session list remains empty. This tests reconstruction from persistent unit enablement across a real manager restart, rather than merely observing the same process for a little longer.

Disabling the worker without a stop request leaves the current service active. Disablement removes future enablement links; it is not an implicit termination command. The script checks that distinction, then explicitly stops the worker and disables lingering. The fixture expects the manager to become inactive after linger is disabled; it does not measure a last-logout delay.

Cleanup disables lingering again if needed, stops the experiment’s manager and runtime-directory unit, and deletes the temporary account and home. The exercise deliberately uses temporary configuration and does not reboot the machine. Its reboot-related statements come from the documented lifecycle contract; the process-replacement assertion concerns a manager restart, not a machine reboot.

Operate an unattended account deliberately

Keep the account’s enabled units reviewable and remove services that are no longer needed. A service that remains available after logout can retain memory, open files, credentials, and network listeners. Ordinary user authority limits some actions but still permits everything granted to that identity. Use suitable permissions and supported resource controls for the actual workload.

Make configuration and credentials available through mechanisms that work without an interactive shell. A successful launch from a terminal does not prove unattended startup has the same environment. Verify working directories, input paths, and access after restarting the manager, then perform a real boot test when boot-time availability is a deployment requirement.

When retiring the role, distinguish disabling a service, stopping current execution, disabling linger, and deleting account data. Other sessions or activation paths may still matter. Inspect the account’s actual manager and service states after the intended change. The useful guarantee is a documented unattended lifecycle with explicit activation and shutdown, not the indefinite survival of one PID.

Sources

systemd 252 loginctl, enabling and disabling per-account lingering and inspecting sessions, checked 2026-09-12.
systemd 252 user@.service, account manager and runtime-directory lifecycles, checked 2026-09-12.
systemd 252 logind.conf, user stop delay and the distinction between sessions and user services, checked 2026-09-12.
systemd 252 systemctl, enable/disable versus start/stop behavior, checked 2026-09-12.
systemd-stable v252.39 user lifecycle source, user_may_gc checks stop delay only after a last-session timestamp exists; source evidence only, checked 2026-10-19.

Check timesyncd’s service state and clock evidence separately

Thu, 11 Mar 2027 00:00:00 +0000

systemd-timesyncd is a minimal SNTP client for synchronizing the local system clock against configured time sources. It can help keep certificate checks and wall-clock timestamps meaningful, but it does not provide an unconditional accuracy guarantee. Whether the daemon can run, whether it receives acceptable samples, and whether the system reports synchronization are separate facts to establish.

When it breaks

The daemon does not always smooth corrections. The documented behavior steps the clock for large offsets and adjusts it gradually for smaller differences. A machine starting far from correct time may therefore experience a discontinuity. Use monotonic clocks for elapsed-duration measurements, and design applications that interpret wall-clock timestamps to handle corrections rather than assuming time always moves smoothly.

FallbackNTP is not a general emergency list used whenever an explicitly configured server becomes unreachable. It applies when no other server information is known. Global NTP entries and per-link sources supplied through networkd take precedence over that fallback. Inspect the actual source selection and configuration drop-ins before concluding that a particular backup server must have been contacted.

An unsynchronized state also does not prove UDP traffic is blocked. A service can be absent, disabled, excluded by unit conditions, unable to adjust time, or rejecting unsuitable samples. Conversely, a global synchronization property does not identify which process supplied time. Distinguish the manager’s service state, the time client’s observations, and the kernel-level synchronization indication.

In the prepared source scenario, unmet conditions skip activation despite a zero start-command return; clock evidence is separate.

Check it quickly

The original failure-mode report used root in a disposable Debian 12 Docker container with systemd 252.39 as PID 1. Its init process was launched through /usr/bin/setpriv --bounding-set=-sys_time,-sys_module /sbin/init; the capability restriction prevents adjustment of the Docker VM’s shared clock. This is a probe for an already prepared container with Python 3, D-Bus, the timesyncd package, and unchanged effective unit conditions, not a container boot recipe. Save it as timesyncd-state.py and run python3 timesyncd-state.py with assertions enabled, without -O or PYTHONOPTIMIZE; its prerequisite checks precede service activation:

from pathlib import Path
import json,subprocess

def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def properties(unit):
 output=run('systemctl','show',unit,'-p','ActiveState','-p','SubState','-p','MainPID','-p','ConditionResult')
 return dict(line.split('=',1) for line in output.splitlines())
# This failure-mode exercise requires an isolated container that cannot set time.
status=dict(line.split(':',1) for line in Path('/proc/1/status').read_text().splitlines() if ':' in line)
assert int(status['CapBnd'].strip(),16)&(1<<25)==0,'PID 1 must lack CAP_SYS_TIME'
assert run('systemd-detect-virt','--container')=='docker'
unit='systemd-timesyncd.service';definition=run('systemctl','cat',unit)
assert 'ConditionCapability=CAP_SYS_TIME' in definition
assert 'ConditionVirtualization=!container' in definition
assert properties(unit)['ActiveState']=='inactive'
run('systemctl','start',unit)
state=properties(unit)
assert state=={'MainPID':'0','ActiveState':'inactive','SubState':'dead','ConditionResult':'no'},state
clock=dict(line.split('=',1) for line in run('timedatectl','show','-p','CanNTP','-p','NTP','-p','NTPSynchronized').splitlines())
assert all(value in ('yes','no') for value in clock.values())
print('start command returned zero; unit stayed inactive with PID 0 and failed conditions')
print('clock properties:',json.dumps(clock,sort_keys=True))
print('PID 1 lacks time-setting capability; Docker condition also excludes this packaged service')
print('no clock-setting command was issued and no NTP synchronization was claimed')

The start command returns zero, yet the unit remains inactive with main PID zero and ConditionResult=no. A skipped conditional activation is not the same as a successfully running daemon. The fixture’s packaged unit requires the time-setting capability and excludes containers; this environment satisfies neither prerequisite. The observed outcome is therefore explained before any theory about blocked NTP packets is needed.

The recorded clock properties are CanNTP=yes, NTP=no, and NTPSynchronized=no. They describe different aspects of the system’s time-service interface. In particular, knowing that NTP service support is available does not make that service active or establish that the clock has synchronized. The script reports the actual values rather than replacing them with an assumed healthy result.

No clock-setting command is issued, and no successful NTP exchange or clock adjustment is claimed. The original report describes this activation failure mode; its raw runtime log was unavailable and this review did not rerun it. Testing successful clock discipline requires an appropriate machine or independently clocked virtual machine with the required authority and a reachable, suitable time source.

When the daemon is actually running, its timedatectl timesync-status interface provides client-specific server and synchronization information. Check service state before relying on that interface; a request to an unavailable service can wait for D-Bus activation and time out. The separately reported five-second probe is absent from this script and was not rerun; its timeout supplies no synchronization evidence.

Build a time requirement that can be verified

Choose sources appropriate to the network and inspect offset, delay, and accepted-source quality over time. Reachability alone does not make a server accurate or trustworthy. Polling and root-distance settings influence sample handling; they do not turn a minimal client into a complete time-distribution service or establish an accuracy service-level agreement by themselves.

The saved /var/lib/systemd/timesync/clock timestamp helps initialize time across boots without a reliable hardware clock. It can also be updated periodically without a recent synchronization, so its existence is not proof of a fresh accepted network sample. Rough initialization and synchronization to a reference clock are different milestones.

If dependent work requires synchronized time, configure and verify the appropriate wait service and ordering after time-sync.target. Starting timesyncd alone does not delay every application until an accurate reference is reached. Keep the startup requirement and runtime monitoring explicit, and use the evidence from the actual deployment rather than a generic “NTP enabled” label.

Sources

systemd 252 systemd-timesyncd.service, SNTP behavior, large clock steps, saved clock state, and synchronization signals, checked 2026-09-12.
systemd 252 timesyncd.conf, source selection, fallback precedence, polling, and root-distance limits, checked 2026-09-12.
systemd 252 org.freedesktop.timedate1, NTP availability, enablement, and the kernel synchronization property, checked 2026-09-12.
systemd 252 systemd-time-wait-sync.service, ordering dependent services after verified time synchronization, checked 2026-09-12.
Python 3.11: assert statement, optimization removes assertion checks, checked 2026-10-18.

EnvironmentFile supplies values when systemd executes a process

Fri, 19 Feb 2027 00:00:00 +0000

An environment file separates deployment values from the unit definition that describes process supervision. systemd reads the configured file before executing a service command and supplies those values in that process’s environment. Editing the file does not rewrite the environment of an already running process. A reload helper is another process, however, and can receive newer file values than the service’s existing main process.

EnvironmentFile is read when a service command executes. The second, third and fourth values are the article’s fixture values, not new observations. File parsing follows systemd rules rather than shell execution. The historical runtime check remains blocked.

When it breaks

daemon-reload tells the manager to reread unit definitions and rebuild its dependency information. reload requests the service’s configured reload operation, such as ExecReload. Neither command is a universal instruction to replace an existing process environment. A service may implement application-specific configuration reload, but that behavior must come from the application or its reload helper.

An EnvironmentFile path prefixed with a minus sign is optional. If it is missing, systemd ignores that absence without the warning an operator might expect. A typo can therefore silently remove an override. Use optional files only when their absence has a valid meaning; a required deployment input should not disappear behind that prefix.

The file is parsed by systemd, not sourced by a shell. Quoting and escape rules exist, but command substitution and variable expansion are not shell execution. A value containing $HOME remains that text. Files listed later override earlier values, and file assignments override Environment assignments. Inspect precedence rather than assuming the first visible definition wins.

Check it quickly

Run this Python script as an administrator in an isolated Linux environment with systemd as PID 1. It uses only harmless fixture settings and temporary paths. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. Ensure the example runtime unit name is unused. The worker records only four selected test values, never a general environment dump:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,time

unit='xbcatm-environment-lab.service';unit_path=Path('/run/systemd/system')/unit
assert not unit_path.exists()
def ctl(*args,success=True):
 result=subprocess.run(['systemctl',*args],capture_output=True,text=True,timeout=10)
 assert (result.returncode==0)==success,(args,result.stderr)
 return result.stdout.strip()
def value(key):return ctl('show',unit,'-p',key,'--value')
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'state transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix='xbcatm-environment-',dir='/run') as directory:
 root=Path(directory);first=root/'required.env';second=root/'optional.env';worker=root/'worker.py'
 command=f'$(touch {root}/unexpected)'
 initial=f'MODE=first\nLITERAL=$HOME\nQUOTED="two words"\nCOMMAND={command}\n'
 first.write_text(initial);first.chmod(0o600);second.write_text('MODE=second\n')
 worker.write_text("""from pathlib import Path
import json,os,sys,time
root=Path(__file__).parent;name=sys.argv[1]
data={key:os.environ.get(key) for key in ('MODE','LITERAL','QUOTED','COMMAND')}
data['pid']=os.getpid()
tmp=root/(name+'.tmp');tmp.write_text(json.dumps(data));tmp.replace(root/(name+'.json'))
if name=='main': time.sleep(60)
""")
 unit_path.write_text(f'''[Service]
Type=exec
Environment=MODE=unit-default
EnvironmentFile={first}
EnvironmentFile=-{second}
ExecStart=/usr/bin/python3 {worker} main
ExecReload=/usr/bin/python3 {worker} reload
''')
 def read(name):return json.loads((root/(name+'.json')).read_text())
 try:
 subprocess.run(['systemd-analyze','verify',str(unit_path)],check=True)
 ctl('daemon-reload');ctl('start',unit)
 wait_for(lambda:(root/'main.json').exists())
 current=read('main');assert current['MODE']=='second'
 assert current['LITERAL']=='$HOME' and current['QUOTED']=='two words'
 assert current['COMMAND']==command and not (root/'unexpected').exists()
 assert 'MODE=unit-default' in value('Environment')
 print('ordered environment files override unit value; quoting works; shell text stays literal')
 second.write_text('MODE=third\n')
 ctl('daemon-reload');assert value('MainPID')==str(current['pid'])
 live=Path(f'/proc/{current["pid"]}/environ').read_bytes().split(b'\0')
 assert b'MODE=second' in live
 ctl('reload',unit)
 assert read('reload')['MODE']=='third' and read('main')==current
 assert value('MainPID')==str(current['pid'])
 live=Path(f'/proc/{current["pid"]}/environ').read_bytes().split(b'\0')
 assert b'MODE=second' in live
 print('reload helper receives new file value; existing main process retains old environment')
 second.write_text('MODE=fourth\n');ctl('restart',unit)
 wait_for(lambda:read('main')['pid']!=current['pid'])
 assert read('main')['MODE']=='fourth'
 print('restart reads changed environment file without another daemon-reload')
 first.unlink()
 subprocess.run(['systemd-analyze','verify',str(unit_path)],check=True)
 ctl('restart',unit,success=False)
 assert value('ActiveState')=='failed' and value('MainPID')=='0'
 print('missing required environment file prevents start')
 first.write_text(initial);first.chmod(0o600);second.unlink()
 ctl('reset-failed',unit);ctl('start',unit)
 wait_for(lambda:read('main')['MODE']=='first')
 print('missing optional file is ignored; required-file value remains')
 finally:
 ctl('stop',unit)
 if value('ActiveState')=='failed':ctl('reset-failed',unit)
 unit_path.unlink();ctl('daemon-reload')

The main process initially receives MODE=second: the optional second file overrides the required first file, which overrides the unit’s default. The quoted value becomes two words with a space. The variable-looking and command-looking values stay literal, and the file that a shell substitution would have created does not exist.

The manager’s Environment property still contains the unit’s default assignment. It is not a merged report of every value read from environment files. The worker’s selected observations provide the effective values for this fixture; the property identifies configured unit assignments. Confusing those views can lead an operator to diagnose the wrong setting.

After editing the file to a third value, daemon-reload leaves the main PID and its actual /proc environment unchanged. Running the reload helper gives that new helper the third value, while the original main process still contains the second. The code checks the specific harmless entry without printing the rest of the process environment.

The file is edited again to a fourth value, then the service is restarted without another manager reload. The replacement process receives the fourth value. This isolates fresh file reading at process execution from reloading unit definitions. It also shows why a successful reload helper does not alone prove that the main application adopted a replacement setting.

Removing the required file still permits static unit verification but makes the actual restart fail with no main PID. Restoring that file and removing the optional one allows startup with the first file’s value. Static validation, successful file loading, and application adoption are three separate checks.

Keep configuration visibility selective

Use environment files for suitable configuration, with ownership and permissions appropriate to the service. Sensitive values still become process environment values and may be inherited or exposed through diagnostics. The filename being outside version control does not establish a complete secret boundary. Use the platform’s supported credential-delivery mechanism when that better matches the secret’s requirements.

Document whether a change requires a new process, a supported application reload, or both. Change unit definitions through the appropriate override location and reload the manager when those definitions change. Editing only an existing environment file does not require that reload, but the intended consuming process must actually execute or otherwise implement an explicit refresh path.

Validate the resulting behavior or selected non-sensitive configuration, not a wholesale dump of environment variables into incident logs. Check the correct PID or invocation so a stale process observation is not mistaken for the replacement. Preserve enough information to identify which configuration version is active without turning troubleshooting into another disclosure channel.

Sources

systemd 252 execution environment, environment-file parsing, precedence, optional paths, and per-process loading, checked 2026-09-12.
systemd 252 systemctl, service reload versus daemon-reload and restart, checked 2026-09-12.
systemd 252 service commands, ExecStart and ExecReload lifecycle, checked 2026-09-12.

One systemd user manager serves an account’s sessions

Tue, 26 Jan 2027 00:00:00 +0000

A systemd user manager runs services on behalf of one account. Multiple login sessions for that account normally share the same manager, rather than each shell receiving a private service universe. User unit files can live under ~/.config/systemd/user, and systemctl --user addresses the user’s manager. This provides lifecycle management with the account’s authority, not automatic isolation from every shared system resource.

Sessions share an account-level manager. The final-session stop shown here is conditional on no linger and the configured stop delay, rather than an immediate universal rule. The source fixture was not rerun for this artwork.

When it breaks

Closing one terminal does not necessarily stop the user’s services. Another login session may remain, and the manager’s lifetime also depends on lingering and the configured stop delay. Even after the final session ends, immediate termination is not a universal rule. Applications that must follow one graphical session need an explicit lifecycle relationship rather than assuming every user unit has that scope.

Selecting the wrong manager is another common failure. Ordinary systemctl addresses the system manager, while systemctl --user needs access to the appropriate user manager and its runtime communication endpoints. A command run through a different identity or a context without session setup may fail to connect. Setting an environment variable alone neither creates the manager nor grants authority to control another account.

A user service also does not automatically inherit the environment of whichever terminal launches a control command. Required paths, configuration, and graphical-session variables must be provided through the intended mechanism. Running as a normal account reduces authority, but that account can still access shared writable files, use network sockets, and consume resources according to existing permissions and limits.

Check it quickly

Run this Python script as an administrator in a disposable Linux system with systemd, logind, PAM integration, a user D-Bus session, and account-management tools. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It creates one temporary account, two real PAM-backed sessions, and a user service; it does not alter an existing person’s units or login state. The final 20-second wait assumes a shorter finite UserStopDelaySec; the valid setting infinity prevents that expected shutdown:

from pathlib import Path
from tempfile import TemporaryDirectory
import os,pwd,subprocess,time,uuid

name='xbcatm-user-'+uuid.uuid4().hex[:8];sessions=[name+'-a.service',name+'-b.service'];unit='field-worker.service'
def run(*args):
 result=subprocess.run(args,capture_output=True,text=True,timeout=20)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def wait_for(check,seconds=5):
 deadline=time.monotonic()+seconds
 while not check():
 assert time.monotonic()<deadline,'state transition timed out';time.sleep(0.05)
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);root.chmod(0o755);home=root/'home';home.mkdir()
 run('useradd','--no-create-home','--user-group','--home-dir',str(home),'--shell','/bin/sh',name)
 account=pwd.getpwnam(name);uid=account.pw_uid;gid=account.pw_gid;manager=f'user@{uid}.service'
 def userctl(*args):
 return run('setpriv','--reuid='+str(uid),'--regid='+str(gid),'--init-groups',
 'env',f'XDG_RUNTIME_DIR=/run/user/{uid}',f'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{uid}/bus',
 'systemctl','--user',*args)
 def manager_value(key):return run('systemctl','show',manager,'-p',key,'--value')
 try:
 units=home/'.config/systemd/user';units.mkdir(parents=True)
 (home/'worker.py').write_text("from pathlib import Path\nimport os,time\np=Path(__file__).with_name('worker-pid');t=p.with_suffix('.tmp');t.write_text(str(os.getpid()));t.replace(p)\ntime.sleep(60)\n")
 (units/unit).write_text('[Service]\nType=exec\nExecStart=/usr/bin/python3 %h/worker.py\n[Install]\nWantedBy=default.target\n')
 for path in [home,*home.rglob('*')]:os.chown(path,uid,gid)
 for session in sessions:
 run('systemd-run','--quiet','--unit='+session,'--property=Type=simple',
 '--property=User='+name,'--property=PAMName=login','/usr/bin/sleep','60')
 wait_for(lambda:Path(f'/run/user/{uid}/bus').exists())
 wait_for(lambda:len(run('loginctl','show-user',name,'-p','Sessions','--value').split())==2)
 assert run('loginctl','show-user',name,'-p','Linger','--value')=='no'
 initial=manager_value('MainPID');assert initial!='0'
 userctl('enable','--now',unit);wait_for(lambda:(home/'worker-pid').exists())
 worker_pid=(home/'worker-pid').read_text()
 assert userctl('show',unit,'-p','MainPID','--value')==worker_pid
 run('systemctl','stop',sessions[0])
 wait_for(lambda:len(run('loginctl','show-user',name,'-p','Sessions','--value').split())==1)
 assert manager_value('MainPID')==initial and userctl('show',unit,'-p','MainPID','--value')==worker_pid
 print('two PAM sessions share one user manager; closing one leaves its service unchanged')
 run('systemctl','stop',sessions[1])
 wait_for(lambda:manager_value('ActiveState')=='inactive',seconds=20)
 assert not Path('/proc/'+worker_pid).exists()
 print('after the final session and configured stop delay, non-lingering manager and worker stop')
 print('controllers use setpriv and the actual user bus; they do not create extra PAM sessions')
 finally:
 for session in sessions:
 if run('systemctl','show',session,'-p','LoadState','--value')!='not-found':run('systemctl','stop',session)
 run('systemctl','stop',manager,f'user-runtime-dir@{uid}.service')
 run('userdel',name)

The two transient session launchers use PAMName=login and the same account. The script waits until logind reports two sessions and the user bus is available. Their launch type is simple, so command return is not treated as evidence that PAM setup has completed. The explicit session and bus observations establish that boundary before user-service control begins.

The worker is enabled under the user’s default target and started through that user’s manager. It publishes its PID atomically in a harmless file, and the script compares that PID with the user unit’s actual main-process property. The system manager’s user@UID.service supplies the separate PID of the account’s manager.

Stopping the first PAM session reduces the session count to one. Both the manager PID and worker PID remain unchanged. This directly disproves the idea that each login shell owns a separate manager whose services must disappear whenever that shell ends. The surviving session keeps the same account-level service context available.

After the second session stops, the script waits for the non-lingering user manager to become inactive under the configured policy. The worker’s process is also gone. The bounded wait accommodates the documented stop delay; the result does not imply that every installation has the same delay or that unrelated shell processes follow identical termination rules.

Control commands switch identity with setpriv and address the real user bus. They do not create additional PAM sessions that would accidentally keep the manager alive and invalidate the final-session test. Cleanup stops the experiment’s remaining launchers and runtime manager, deletes the temporary account, and removes its temporary home.

Give each service the lifetime it needs

Decide whether the workload belongs to one graphical session, all sessions of an account, or an unattended machine role. A user manager is useful for account-owned background work, while linger can deliberately extend its lifetime beyond login. That choice should be explicit because it changes when resources and network listeners remain active.

Use the account’s permissions as a starting point, then add supported limits and sandboxing for the actual workload. Some execution restrictions need privileges or kernel support that are unavailable to ordinary user managers. Verify enforcement and useful behavior instead of assuming system-unit security settings transfer unchanged.

Keep unit enablement, application environment, and observable results consistent. Inspect the intended user’s units and logs, especially when several sessions or deployment tools control the same account. A shared manager simplifies supervision only when its shared lifetime and authority are part of the service’s documented contract.

Sources

systemd 252 user@.service, per-user managers, session scopes, and runtime directories, checked 2026-09-12.
systemd 252 logind.conf, last-session handling, user-manager stop delay, and process policy, checked 2026-09-12.
systemd 252 systemd.exec, PAM sessions, execution identity, environment, and sandbox limitations, checked 2026-09-12.
systemd 252 loginctl, user/session inspection and lingering policy, checked 2026-09-12.

A read-only root restricts filesystem mutation within its scope

Wed, 23 Dec 2026 00:00:00 +0000

A read-only root filesystem prevents ordinary writes to the filesystem layer containing a service’s packaged code and configuration. Explicit mounts can still provide writable locations for runtime data. Its boundary is filesystem mutation. It does not by itself remove process privileges, prevent reading available data, or block communication with other services.

When it breaks

Applications often write outside their main data directory: temporary files, caches, lock files, generated configuration, or runtime state can appear during startup and uncommon operations. Making the root read-only can expose those assumptions as EROFS failures. Testing only a health endpoint may miss the path that generates a report, refreshes a cache, or performs shutdown cleanup.

A writable mount remains writable even when the container root layer is read-only. It can contain state that a compromised or faulty process can corrupt. Broadly mounting a writable directory over packaged paths can also defeat the intended protection for those paths. Inspect the effective mount layout rather than inferring the entire pathname tree’s behavior from one root-level option.

Read-only and confidential are different properties. A process may still read packaged files, execute existing programs, and send data through allowed network paths. The setting also does not guarantee protection after a container escape or against authority capable of changing the mount arrangement. Its contribution must be understood alongside credentials, capabilities, isolation, and application behavior.

A read-only root restricts its own layer; writable mounts, reading, execution and local communication remain separate.

Check it quickly

Use root inside a disposable Linux container with a read-only root, writable tmpfs at /tmp, and a separate writable volume at /work. Python 3.7 or later and /etc/os-release must be present; assertions must remain enabled. Earlier records report Linux 6.12.76 and Python 3.11.2, without host bind mounts, privileged mode or external networking. That run has not been reproduced here; these launch conditions are prerequisites, not all checked by the script:

from pathlib import Path
from tempfile import mkdtemp
import errno,os,socket,subprocess,sys,time,uuid

def cleanup_steps(steps, primary):
 failures=[]
 for label, action in steps:
 try:action()
 except BaseException as error:
 failures.append(label+': '+type(error).__name__+': '+str(error))
 if failures:
 message='cleanup incomplete or uncertain: '+'; '.join(failures)
 try:print(message,file=sys.stderr)
 except (OSError,ValueError):pass
 if primary is None:raise RuntimeError(message)
 return failures

def remove_file(path):
 try:path.unlink()
 except FileNotFoundError:pass

def require_empty(value):
 if value:raise RuntimeError('residual resources: '+str(value))

assert os.geteuid()==0,'run inside a disposable read-only-root container as root'
assert os.statvfs('/').f_flag&os.ST_RDONLY
name='xbcatm-readonly-'+uuid.uuid4().hex
print('fixture root-tree probe:',name,flush=True)
for base in [Path('/'),Path('/usr'),Path('/etc')]:
 path=base/name
 fd=None
 try:
 try:fd=os.open(path,os.O_WRONLY|os.O_CREAT|os.O_EXCL,0o600)
 except OSError as error:assert error.errno==errno.EROFS,error
 else:raise AssertionError('root filesystem accepted a new file')
 finally:
 primary=sys.exc_info()[1]
 if fd is not None:
 cleanup_steps([('probe descriptor',lambda:os.close(fd)),('unexpected root-tree file '+str(path),lambda:remove_file(path))],primary)
print('even this root process receives EROFS on paths in the read-only root, usr and etc tree')
for base in ['/tmp','/work']:
 assert not os.statvfs(base).f_flag&os.ST_RDONLY
 directory=None
 try:
 directory=Path(mkdtemp(prefix='xbcatm-write-',dir=base))
 path=directory/'data';path.write_bytes(b'allowed');assert path.read_bytes()==b'allowed'
 path.write_bytes(b'changed');assert path.read_bytes()==b'changed'
 finally:
 primary=sys.exc_info()[1]
 if directory is not None:
 cleanup_steps([('temporary data',lambda:remove_file(directory/'data')),('temporary directory',directory.rmdir)],primary)
print('explicit tmpfs and data-volume mounts remain writable, including overwriting their contents')
assert Path('/etc/os-release').read_text()
result=subprocess.run([sys.executable,'-c','print("executed")'],capture_output=True,text=True,check=True,timeout=5)
assert result.stdout.strip()=='executed'
def remaining(deadline):
 seconds=deadline-time.monotonic()
 if seconds<=0:raise TimeoutError('loopback exchange deadline expired')
 return seconds

def receive_exact(connection,count,deadline):
 chunks=[];received=0
 while received<count:
 connection.settimeout(remaining(deadline))
 chunk=connection.recv(count-received)
 if not chunk:raise EOFError('loopback peer closed before the complete payload')
 chunks.append(chunk);received+=len(chunk)
 return b''.join(chunks)

listener=client=connection=None
try:
 deadline=time.monotonic()+5
 listener=socket.socket();listener.settimeout(remaining(deadline))
 listener.bind(('127.0.0.1',0));listener.listen(1)
 client=socket.socket();client.settimeout(remaining(deadline));client.connect(listener.getsockname())
 listener.settimeout(remaining(deadline));connection,_=listener.accept()
 client.settimeout(remaining(deadline));client.sendall(b'local')
 assert receive_exact(connection,5,deadline)==b'local'
finally:
 primary=sys.exc_info()[1]
 cleanup_steps([(label,sock.close) for label,sock in [('connection',connection),('client',client),('listener',listener)] if sock is not None],primary)
print('reading packaged files, executing an existing program, and loopback network I/O still work')

The script first checks the root filesystem’s reported read-only flag. It then attempts exclusive creation of uniquely named files under the root, /usr, and /etc. Each fails with EROFS, even for this root identity. Exclusive creation and unique names ensure the probes cannot overwrite a preexisting configuration file.

The fixture next checks that /tmp and /work do not report that flag. Temporary files can be created, read, and overwritten in both mounts. This confirms that writable exceptions remain operational and also establishes their exposure: the root setting does not protect the contents of those deliberately writable locations against the same process.

It reads /etc/os-release and executes an existing Python interpreter successfully. This distinguishes read-only storage from blocked reads or execution. Disallowing modification of packaged files can be useful, but it is not a general restriction on what existing code may do with its remaining authority.

Finally, a listener and client exchange a harmless payload over loopback. The container has no external networking, but local socket operations still work. The result shows that the filesystem option does not itself disable networking. It does not establish an external connectivity policy or test communication with any production service.

This revised fixture is unrun. Receive accumulates short reads, rejects early EOF and shares a five-second monotonic deadline with connect, accept and send. The interpreter timeout is also five seconds. Guards precede creation; independent socket/file cleanup preserves the original exception and fails on cleanup-only errors. Root identity, container markers and read-only flags do not prove isolation; verify launch conditions independently. The launcher owns container/volume teardown. Inspect reported leftovers after failure. Process creation, stalled filesystems and abrupt termination remain outside these bounds. These probes cover selected operations only; failure of root writes neither covers every application path nor restricts future writable mounts.

Separate packaged files, scratch space, and persistent state

Choose writable paths from observed application needs and configure the application to use them. Keep the exceptions narrow enough to explain their purpose. A failure should lead to identifying the required path and lifetime, rather than reopening the whole root layer to accommodate one unexpected cache or temporary file.

Use tmpfs for suitable temporary data and account for its memory use and lifecycle. Docker notes that tmpfs pages can be written to swap; calling this mount memory-backed is not a guarantee that sensitive bytes never reach persistent storage. Use a separately managed volume when state needs a lifetime independent of the container’s writable layer, with appropriate backup and recovery. A writable volume supplies a storage location; it does not automatically establish crash durability, confidentiality, or a tested restore procedure.

For systemd-managed services, mechanisms such as ProtectSystem and designated state or runtime directories express related filesystem restrictions with their own documented semantics. Validate the service’s actual view and identity rather than translating an option name mechanically between platforms. Exercise startup, ordinary work, exceptional operations, and cleanup, and confirm both prohibited writes and required writes behave as intended.

Sources

Docker container run reference, read-only root filesystems and separately mounted writable paths, checked 2026-09-12.
Docker tmpfs mounts, temporary mount storage and lifecycle, checked 2026-09-12.
Docker volumes, volume lifetime independent of a container root layer, checked 2026-09-12.
systemd 252 execution environment, ProtectSystem and explicitly writable service directories, checked 2026-09-12.

Choose the clock and catch-up rule for a systemd timer

Fri, 20 Nov 2026 00:00:00 +0000

A schedule must specify what time means. “At nine each morning in New York” refers to a civil-time calendar. “A quarter hour after this job finishes” refers to an elapsed interval and an event. systemd timers support both, but the two clocks and reference points have different behavior around time-zone changes, suspension, and periods when the timer is inactive.

Calendar persistence requests one catch-up activation rather than replaying every missed business period. The dates and three-event example come from the article; this illustration adds no timer runtime evidence.

When it breaks

OnCalendar follows wall-clock calendar expressions. Omitting a time zone generally uses the system’s local zone, which may differ between machines. A daily local time can move relative to UTC when daylight-saving rules change. Preview the intended expression with a fixed reference date instead of assuming that today’s UTC offset applies throughout the year.

Monotonic options name their reference event explicitly. OnActiveSec refers to activation of the timer itself, OnStartupSec to the service manager’s startup, and OnUnitInactiveSec to the target unit’s deactivation. These are not interchangeable ways to say “every few minutes.” Ordinary monotonic timers pause with system suspension; WakeSystem changes clock selection and depends on platform support.

Persistent=yes affects calendar timers. It records trigger timing and requests a catch-up activation when the timer becomes active after missing at least one event. It does not replay every missed occurrence or certify successful job completion. The catch-up can also be delayed by configured randomization. A trigger history is not a durable queue of all business work that should have happened.

Check it quickly

Run this Python script as an administrator in a disposable Linux system with systemd as PID 1 and the time-zone database installed. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It first previews a calendar expression, then installs two unique timer/service pairs using the same three near-future calendar events. It never adjusts the machine’s clock:

from pathlib import Path
from tempfile import TemporaryDirectory
from datetime import datetime,timezone
import subprocess,time,uuid

name='xbcatm-calendar-'+uuid.uuid4().hex[:8]
names=[name,name+'-plain'];paths=[]
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
preview=run('systemd-analyze','calendar','--base-time=2026-03-07 00:00:00 UTC',
 '--iterations=3','*-*-* 09:00:00 America/New_York')
for expected in ('2026-03-07 14:00:00 UTC','2026-03-08 13:00:00 UTC','2026-03-09 13:00:00 UTC'):
 assert expected in preview,preview
print('09:00 New York: 14:00 UTC before spring DST, 13:00 UTC after it')
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py'
 worker.write_text("from pathlib import Path\nimport sys\np=Path(sys.argv[1])\nwith p.open('a') as output:output.write('run\\n')\n")
 now=int(time.time());events=[now+3,now+4,now+5]
 calendar=''.join('OnCalendar='+datetime.fromtimestamp(t,timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')+'\n' for t in events)
 try:
 for index,prefix in enumerate(names):
 service=Path('/run/systemd/system')/(prefix+'.service');timer=service.with_suffix('.timer')
 assert not service.exists() and not timer.exists()
 service.write_text(f'[Service]\nType=oneshot\nExecStart=/usr/bin/python3 {worker} {root}/{index}.log\n');paths.append(service)
 timer.write_text('[Timer]\n'+calendar+'AccuracySec=1ms\nRandomizedDelaySec=0\nPersistent='+('yes' if index==0 else 'no')+'\n');paths.append(timer)
 run('systemd-analyze','verify',*[str(p) for p in paths]);run('systemctl','daemon-reload')
 run('systemctl','start',name+'.timer');run('systemctl','stop',name+'.timer')
 assert not (root/'0.log').exists() and time.time()<events[0]
 while time.time()<events[-1]+0.15:time.sleep(0.05)
 run('systemctl','start',name+'.timer',name+'-plain.timer')
 deadline=time.monotonic()+3
 while not (root/'0.log').exists():
 assert time.monotonic()<deadline,'catch-up did not run';time.sleep(0.02)
 time.sleep(0.15)
 assert (root/'0.log').read_text()=='run\n' and not (root/'1.log').exists()
 print('three elapsed calendar events: persistent timer catches up once; plain timer does not')
 run('systemctl','stop',name+'.timer');run('systemctl','start',name+'.timer')
 time.sleep(0.2);assert (root/'0.log').read_text()=='run\n'
 print('reactivating with no newly missed event does not replay the job')
 print('timer inactivity was tested without changing the system clock or rebooting')
 finally:
 if paths:
 run('systemctl','stop',*[p.name for p in paths])
 run('systemctl','clean','--what=state',name+'.timer')
 for path in paths:path.unlink(missing_ok=True)
 run('systemctl','daemon-reload')

The preview asks for nine in the morning in New York, starting from a fixed date before the March 2026 daylight-saving transition. The first occurrence is 14:00 UTC; the following two are 13:00 UTC. The local clock time remains nine. This is a schedule interpretation check, not a request to change the host’s time zone or travel to those dates.

The persistent timer is started and stopped before the first laboratory event. Starting it establishes its timestamp state. While it is inactive, all three specified events pass. On reactivation it launches its service exactly once, producing one line in the result file. The nonpersistent timer, activated after the same events, produces no result file.

This experiment observes an actual inactive period rather than editing a timestamp file to impersonate a missed run. It demonstrates the catch-up decision without rebooting the machine. Three elapsed events become one activation; an application that needs to process three distinct accounting periods must discover and track those periods itself.

Stopping and starting the persistent timer again produces no extra line because no new event was missed. Cleanup stops the runtime units and removes the persistent timer’s state through systemctl clean --what=state. The nonpersistent timer has no matching persistent state to clean. Configuration files and the harmless result directory are then removed.

AccuracySec=1ms and zero randomized delay keep the short experiment interpretable. Neither setting promises exact process-start timing under arbitrary load. The code waits for observable completion within a bounded interval and asserts the resulting invocation count, rather than equating a predicted deadline with actual execution.

Decide which missed work matters

For a backup, one fresh snapshot after downtime may satisfy the recovery policy. For billing, each missing period might require separate reconciliation. Put that distinction into application state and idempotency rules. A service already active when a timer elapses is not automatically duplicated, and retained active oneshot state can prevent the command from running again.

Use the timer’s loaded configuration and next/last trigger properties to inspect scheduling, and the target service’s result to inspect execution. Calendar ordering after time-related targets does not by itself prove that a remote time source has synchronized the clock; the actual time-service configuration must supply that promise when it is required.

Test representative dates and lifecycle transitions before relying on a schedule. Include time-zone boundaries, timer disablement, and the task’s expected duration. Choose a monotonic reference for elapsed-delay requirements and a named calendar zone for civil-time requirements. Preserve evidence of completed useful work separately from evidence that the manager issued a trigger.

Sources

systemd 252 systemd.timer, clock references, calendar persistence, accuracy, and randomization, checked 2026-09-12.
systemd 252 systemd.time, calendar expressions and named time-zone syntax, checked 2026-09-12.
systemd 252 systemd-analyze, calendar previews from a fixed base time and multiple iterations, checked 2026-09-12.
systemd 252 systemctl, timer activation and removal of persistent timer state, checked 2026-09-12.

systemd supervises process lifecycles

Sun, 01 Nov 2026 00:00:00 +0000

A service manager turns process startup, failure, and shutdown into observable state transitions. With systemd, a unit can name the foreground command, its restart policy, and its output destination. That removes the need for a shell loop that repeatedly launches a daemon without recording why it stopped. It does not prove that the running application can answer requests or that restarting repairs the underlying fault.

Track the managed process and its invocation, bound restart attempts, and test readiness separately. An intentional stop is distinct from a crash, and a restart count is not an availability metric.

When it breaks

Choose a service type that matches the program’s startup contract. Type=simple considers startup successful early, before the executable has necessarily been invoked successfully. Type=exec waits for successful execution of the program, so a missing executable can fail the start job directly. Neither type waits for the application to finish connecting to its database or loading its state.

A program that backgrounds itself can confuse a unit expecting a foreground main process. Use the program’s foreground mode and let the manager own its lifetime, or choose the documented service type for its actual daemonization protocol. Avoid guessing a PID from a process-name search: names can collide, and wrappers can leave the wrong process designated as the service’s main process.

Restart policy also needs explicit limits. Restart=on-failure covers selected failure exits, not every possible termination signal or every unhealthy condition. A hung process that never exits will not automatically become a restart event. Combine meaningful application health observation with a bounded restart policy rather than equating an active unit with successful service delivery.

Check it quickly

Run the following Python script as an administrator in a disposable Linux environment whose PID 1 is systemd. It writes only named temporary runtime units and a temporary working directory. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. Confirm the two unit filenames are unused before executing:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,time

unit='xbcatm-supervise-lab.service';bad='xbcatm-exec-failure-lab.service'
unit_path=Path('/run/systemd/system')/unit;bad_path=unit_path.with_name(bad)
assert not unit_path.exists() and not bad_path.exists()
def run(args,success=True):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert (result.returncode==0)==success,(args,result.returncode,result.stderr)
 return result.stdout.strip()
def value(name,key):return run(['systemctl','show',name,'-p',key,'--value'])
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'state transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix='xbcatm-supervise-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py';ready=root/'ready'
 worker.write_text("""from pathlib import Path
import os,signal,sys
signal.signal(signal.SIGTERM,lambda *_: sys.exit(0))
Path(__file__).with_name('ready').write_text(str(os.getpid()))
print('worker ready',flush=True)
while True: signal.pause()
""")
 unit_path.write_text(f'''[Unit]
Description=Disposable supervision exercise
StartLimitIntervalSec=10s
StartLimitBurst=3
[Service]
Type=exec
ExecStart=/usr/bin/python3 -u {worker}
Restart=on-failure
RestartSec=200ms
TimeoutStopSec=2s
StandardOutput=journal
StandardError=journal
''')
 bad_path.write_text(f'[Service]\nType=exec\nExecStart={root}/missing-program\n')
 try:
 run(['systemd-analyze','verify',str(unit_path)])
 run(['systemctl','daemon-reload']);run(['systemctl','start',unit])
 wait_for(lambda: ready.exists() and ready.read_text()==value(unit,'MainPID'))
 first=value(unit,'MainPID');assert first!='0'
 first_invocation=value(unit,'InvocationID')
 run(['systemctl','kill','--kill-who=main','--signal=KILL',unit])
 wait_for(lambda: value(unit,'MainPID') not in ('0',first) and ready.read_text()==value(unit,'MainPID'))
 assert int(value(unit,'NRestarts'))==1
 print('SIGKILL: replacement PID confirmed; NRestarts=1')
 run(['journalctl','--sync'])
 for invocation in (first_invocation,value(unit,'InvocationID')):
 logs=run(['journalctl','_SYSTEMD_INVOCATION_ID='+invocation,'--no-pager','-o','cat'])
 assert 'worker ready' in logs
 print('journal contains both application startups')
 run(['systemctl','stop',unit]);time.sleep(0.3)
 assert value(unit,'ActiveState')=='inactive' and value(unit,'MainPID')=='0'
 print('manager stop leaves service inactive')
 run(['systemctl','daemon-reload']);run(['systemctl','start',bad],success=False)
 assert value(bad,'Result')=='exit-code' and value(bad,'ExecMainStatus')=='203'
 print('Type=exec reports missing executable: status 203/EXEC')
 finally:
 run(['systemctl','stop',unit,bad])
 for name in (unit,bad):
 if value(name,'ActiveState')=='failed':
 run(['systemctl','reset-failed',name])
 unit_path.unlink();bad_path.unlink(missing_ok=True)
 run(['systemctl','daemon-reload'])

The worker stays in the foreground and publishes its own startup marker. The script waits until that marker agrees with the unit’s main PID, then sends SIGKILL specifically to the managed main process. A different PID and one automatic restart confirm that the manager actually replaced the terminated process.

Journal checks use the two invocation identifiers, so older entries from an earlier run cannot satisfy the assertion. Each invocation must contain the worker’s startup message. This checks both lifecycle replacement and attribution of the output produced by the two actual workers, rather than simply finding a familiar phrase somewhere in the unit’s accumulated history.

An explicit manager stop leaves the unit inactive with no main PID after the restart delay has passed. This is an intentional stop, not another crash to undo. The separate failing unit uses a nonexistent executable; its start command fails and records status 203/EXEC. That failure concerns process execution, not a database connection or application-level health probe.

The script verifies the valid unit definition before starting it and removes both runtime unit files afterward. Syntax verification and successful execution are separate evidence: the first checks the manager’s interpretation of configuration, while the second observes the behavior of a real process. Neither substitutes for checking a production endpoint.

Make the lifecycle operationally useful

Keep the installed unit definition, runtime configuration, and application readiness signal consistent. Ordering another unit after this service orders manager startup completion; if a dependent application needs a stronger readiness promise, use a supported notification protocol or an explicit application-level check. Retrying transient dependencies may still be part of the application’s own startup behavior.

Use systemctl show for specific state properties and the journal for the corresponding invocation. A unit’s current PID can change between observations, so identify the instance when diagnosing repeated failures. Track both restart frequency and whether useful work resumes; a successful restart counter is not an availability metric by itself.

Runtime files under /run/systemd/system are suitable for this disposable exercise. A deployed service also needs an intentional installation and enablement policy, appropriate user permissions, and a tested shutdown path. Changing a unit definition requires reloading the manager’s configuration; starting a service once does not automatically arrange for it to start at the next boot.

Sources

systemd 252 service units, startup types and restart semantics, checked 2026-09-12.
systemd 252 execution environment, output routing and execution exit codes, checked 2026-09-12.
systemd 252 unit configuration, runtime unit paths and start-rate limits, checked 2026-09-12.
systemd 252 journalctl, querying recorded fields and synchronizing journal writes, checked 2026-09-12.

A maintenance timer should leave its last result visible

Tue, 04 Aug 2026 00:00:00 +0000

Cron-like syntax is not the whole contract for a maintenance task. The command needs a clear service identity, a bounded runtime, and evidence that distinguishes “not scheduled” from “ran and failed”. A systemd timer and oneshot service keep those concerns separate.

A fired timer is only a scheduling event. Judge this run by its service result, exit status and fresh timestamp.

Define the work as a service

Use an explicit user and absolute command path:

# /etc/systemd/system/site-check.service
[Unit]
Description=Read-only public site check

[Service]
Type=oneshot
User=sitecheck
ExecStart=/usr/local/bin/check-public-site
TimeoutStartSec=2min

TimeoutStartSec limits the oneshot activation, which includes its command runtime; oneshot services otherwise have no start timeout by default. Timeout handling can then require a stop timeout and cannot instantly end uninterruptible kernel I/O, so the helper also needs bounded network requests. The helper should return non-zero when its contract fails and must not modify the service it checks.

Schedule the service

# /etc/systemd/system/site-check.timer
[Unit]
Description=Run the public site check every hour

[Timer]
OnCalendar=hourly
Persistent=true
RandomizedDelaySec=5min
Unit=site-check.service

[Install]
WantedBy=timers.target

Persistent=true triggers one catch-up activation when the timer is activated after missing at least one calendar event; it does not replay every missed hour. The catch-up is still subject to the randomized delay. RandomizedDelaySec spreads routine work across a window; it is not appropriate when the task must run at an exact instant.

Validate before enabling

After creating both unit files, validate them and reload the manager before starting the service manually. These are administrative operations on the intended test host, not read-only checks:

systemd-analyze verify /etc/systemd/system/site-check.service \
 /etc/systemd/system/site-check.timer
systemd-analyze calendar hourly
systemctl daemon-reload
systemctl start site-check.service
systemctl status site-check.service
journalctl -u site-check.service -n 50 --no-pager
systemctl show site-check.service -p Result -p ExecMainCode -p ExecMainStatus

Enable the timer only after the service succeeds with the intended user, environment, paths, and network access:

systemctl enable --now site-check.timer
systemctl list-timers site-check.timer

Alert on the service result

The timer firing is not success. Monitor the oneshot service’s exit status or journal result. Keep stdout concise and send detailed artifacts to a bounded path if they are needed for debugging.

A successful oneshot normally returns to inactive (dead); that alone is not a failure, and systemctl status can return nonzero for an inactive service. Correlate Result and exit status with a fresh journal timestamp so an old successful run is not mistaken for this one. Do not add RemainAfterExit=yes merely to keep it green: a recurring timer does not start a fresh instance while its service is already active. Calendar timing uses the host timezone unless specified, and normal timer accuracy adds scheduling slack beyond the random delay.

Sources

systemd.timer(5), calendar timers, persistence, and delay behavior, checked 2026-08-04.
systemd.time(7), calendar expression syntax, checked 2026-08-04.
systemd.service, oneshot service semantics, checked 2026-08-04.
systemd.service upstream manual (v257), Type=exec/oneshot, readiness, activation timeout, and restart semantics, checked 2026-10-15.
systemd.timer upstream manual (v257), persistence coalesces missed runs; active units, randomized delay, and accuracy, checked 2026-10-15.

Restart policies need a rate limit

Mon, 03 Aug 2026 00:00:00 +0000

Automatic restart is useful for a long-running service, but an unbounded loop can turn a clear failure into a noisy one. The unit should describe both the normal process lifecycle and the amount of recovery that is acceptable.

Count starts within the configured window. A restart rate limit stops rapid churn; it neither proves health nor caps lifetime retries.

Track setup failures

For a long-running process, Type=exec makes failures to execute the configured program visible during start-up rather than treating a fork as success. Pair it with an explicit ExecStart and the least-privileged User that can perform the work.

[Unit]
Description=Example foreground site service
StartLimitIntervalSec=60s
StartLimitBurst=5

[Service]
Type=exec
User=site
ExecStart=/srv/site/bin/server
Restart=on-failure
RestartSec=5s

Type=exec confirms execution, not application readiness. Keep the program in the foreground; use Type=notify when the application supports explicit readiness notification. Restart=on-failure is a recovery policy, not a health check. If the process stays alive while serving bad responses, systemd will not know that the site is broken; an external smoke test still has a job.

Bound the loop

Systemd applies start-rate limiting through StartLimitIntervalSec= and StartLimitBurst=. These directives belong in [Unit], while RestartSec belongs in [Service]. The example allows five starts within 60 seconds; rapid failures separated by five-second restart delays can therefore hit the limit. A longer delay can keep a loop below that rate forever: this is not a lifetime retry budget. Choose limits from the expected failure pattern. When the limit is hit, the journal should contain the evidence needed to distinguish “restarted successfully” from “gave up after repeated failure”.

After installing or editing the unit on an authorized test host, validate the file, reload the manager, and inspect its effective configuration before restarting. The commands below change service state and can interrupt traffic:

systemd-analyze verify /etc/systemd/system/example.service
systemctl daemon-reload
systemctl cat example.service
systemctl show example.service -p Restart -p RestartUSec \
 -p StartLimitIntervalUSec -p StartLimitBurst
systemctl restart example.service
systemctl status --no-pager example.service
journalctl -u example.service -n 80 --no-pager

Do not use a restart policy to conceal missing files, invalid permissions, or a bad environment. Fix the first failure, then test the recovery path.

Once start limiting stops automatic restarts, waiting for the interval alone does not schedule another attempt. After fixing the cause, either start it after the interval or deliberately use systemctl reset-failed example.service to clear the counter before starting. Test missing-executable and rapid-crash cases in an isolated service, and retain the first failure log rather than only the final “start limit hit” message.

Sources

systemd.service, service type, restart, and start-rate limiter, checked 2026-08-03.
systemd.unit, common unit sections and drop-in semantics, checked 2026-08-03.
systemd.service upstream manual (v257), Type=exec/oneshot, readiness, activation timeout, and restart semantics, checked 2026-10-15.
systemd.unit upstream manual (v257), start-limit section placement, rate window, and reset-failed behavior, checked 2026-10-15.

Harden a systemd service by testing its boundaries

Sat, 27 Jun 2026 00:00:00 +0000

Service hardening should turn a specific unwanted operation into a denied operation while preserving the work the application must perform. systemd supplies execution settings for filesystem views, namespaces, privileges, and other resources. They are separate controls with separate limits. Adding four familiar directives does not make every readable file, reachable process, inherited descriptor, or allowed system call disappear.

When it breaks

ProtectSystem=strict makes most of the filesystem hierarchy read-only inside the service’s mount namespace. Read-only does not mean unreadable. The setting also has documented exceptions, including API filesystems that need separate controls. A service can still disclose information it can read, and writable directories remain places where it can damage data. Start with a dedicated service identity and an explicit inventory of required access.

PrivateTmp=yes gives the service private temporary directories. An application expecting another service to exchange files through the ordinary /tmp may consequently fail. PrivateNetwork=yes creates a network namespace with its own loopback interface; a listener on the host’s loopback is no longer that same endpoint. This is inappropriate for a program that must directly reach a database or public API unless its connectivity is intentionally arranged.

NoNewPrivileges=yes prevents execution from granting new privileges through mechanisms such as set-user-ID binaries and file capabilities. It does not remove privileges the process already has. Similarly, IPAddressAllow alone does not imply that all other addresses are denied: address filtering has allow/deny precedence and requires supported kernel functionality. Inspect each control’s contract instead of reading an option name as a complete security policy.

Distinct systemd controls deny selected operations while explicitly managed state and temporary writes remain possible.

Check it quickly

This fixture requires root in a disposable Linux system with systemd as PID 1, working mount/network namespaces, Python 3.8 or later at /usr/bin/python3, and user-management utilities. Keep Python assertions enabled; the generated worker must be readable by the service account. Earlier records report Debian 12 with systemd 252.39; that run has not been reproduced here. It creates a uniquely named account, runtime unit, state directory and files. Both phases use the same non-root workload:

from pathlib import Path
from tempfile import mkdtemp
import grp,json,os,pwd,socket,subprocess,sys,uuid

def cleanup_steps(steps, primary):
 failures=[]
 for label, action in steps:
 try:action()
 except BaseException as error:
 failures.append(label+': '+type(error).__name__+': '+str(error))
 if failures:
 message='cleanup incomplete or uncertain: '+'; '.join(failures)
 try:print(message,file=sys.stderr)
 except (OSError,ValueError):pass
 if primary is None:raise RuntimeError(message)
 return failures

def remove_file(path):
 try:path.unlink()
 except FileNotFoundError:pass

def require_empty(value):
 if value:raise RuntimeError('residual resources: '+str(value))

suffix=uuid.uuid4().hex[:12];name='xbcatm-hardening-'+suffix
unit=name+'.service';unit_path=Path('/run/systemd/system')/unit
state=Path('/var/lib')/name;marker=Path('/tmp')/(name+'-marker');scratch=Path('/tmp')/(name+'-scratch')
assert os.geteuid()==0,'requires an explicitly prepared disposable systemd test system'
for path in [unit_path,state,marker,scratch]:assert not os.path.lexists(path),path
def account_exists():
 try:pwd.getpwnam(name);return True
 except KeyError:return False
def group_exists():
 try:grp.getgrnam(name);return True
 except KeyError:return False
assert not account_exists() and not group_exists()
def command(args):
 result=subprocess.run(args,capture_output=True,text=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def ctl(*args):return command(['systemctl',*args])
def delete_account():
 if account_exists():command(['userdel',name])
def delete_group():
 if group_exists():command(['groupdel',name])
root=None;listener=connection=None;owned_files=[];account_attempted=False;unit_attempted=False
# Reserve each ordinary file exclusively and record it before writing or chmod.
def create_file(path,text,mode):
 with path.open('x') as output:
 owned_files.append(path)
 output.write(text)
 path.chmod(mode)
print('fixture:',name,'unit:',unit_path,'state:',state,'marker:',marker,'scratch:',scratch,flush=True)
try:
 root=Path(mkdtemp(prefix=name+'-',dir='/run'));root.chmod(0o755)
 outside=root/'outside';create_file(outside,'initial\n',0o666)
 create_file(marker,'host namespace marker\n',0o644)
 create_file(scratch,'',0o666)
 listener=socket.socket();listener.settimeout(2)
 listener.bind(('127.0.0.1',0));listener.listen(2)
 worker=root/'worker.py'
 create_file(worker,r"""from pathlib import Path
import json,socket,sys
root=Path(__file__).parent;state=Path(sys.argv[1]);marker=Path(sys.argv[2]);scratch=Path(sys.argv[3]);port=int(sys.argv[4]);phase=sys.argv[5]
result={}
try:
 with (root/'outside').open('a') as output:output.write(phase+'\n')
 result['outside_write']=True
except OSError as error:result['outside_write']=False;result['outside_errno']=error.errno
result['host_tmp_visible']=marker.exists()
scratch.write_text(phase);result['tmp_write']=scratch.read_text()==phase
try:
 with socket.create_connection(('127.0.0.1',port),timeout=0.5):pass
 result['network_connect']=True
except OSError as error:result['network_connect']=False;result['network_errno']=error.errno
status=dict(line.split(':',1) for line in Path('/proc/self/status').read_text().splitlines() if ':' in line)
result['no_new_privs']=status['NoNewPrivs'].strip()
(state/'result.json').write_text(json.dumps(result))
""",0o644)
 definition=f'''[Service]
Type=oneshot
User={name}
Group={name}
StateDirectory={name}
StateDirectoryMode=0700
TimeoutStartSec=5s
TimeoutStopSec=5s
KillMode=control-group
ExecStart=/usr/bin/python3 {worker} {state} {marker} {scratch} {listener.getsockname()[1]} PHASE
'''
 account_attempted=True
 command(['useradd','--system','--no-create-home','--user-group','--shell','/usr/sbin/nologin',name])
 create_file(unit_path,definition.replace('PHASE','baseline'),0o644)
 unit_attempted=True
 ctl('daemon-reload');ctl('start',unit)
 baseline=json.loads((state/'result.json').read_text())
 assert baseline=={'outside_write':True,'host_tmp_visible':True,'tmp_write':True,'network_connect':True,'no_new_privs':'0'},baseline
 connection,_=listener.accept()
 connection.close();connection=None
 original=outside.read_text()
 settings='ProtectSystem=strict\nPrivateTmp=yes\nPrivateNetwork=yes\nNoNewPrivileges=yes\n'
 unit_path.write_text(definition.replace('PHASE','hardened')+settings)
 command(['systemd-analyze','verify',str(unit_path)])
 ctl('daemon-reload');ctl('start',unit)
 hardened=json.loads((state/'result.json').read_text())
 assert hardened['outside_write'] is False and hardened['outside_errno']==30,hardened
 assert hardened['host_tmp_visible'] is False and hardened['tmp_write'] is True
 assert hardened['network_connect'] is False and hardened['network_errno']==111,hardened
 assert hardened['no_new_privs']=='1'
 assert outside.read_text()==original and scratch.read_text()=='baseline'
 print('baseline: outside write, host tmp visibility, loopback connection permitted; NoNewPrivs=0')
 print('hardened: outside write=EROFS; host tmp hidden; host listener refused; NoNewPrivs=1')
 print('managed state directory and private temporary writes still succeed')
finally:
 primary=sys.exc_info()[1];failures=[];quiescent=not unit_attempted
 # A failed stop does not skip the independently attempted state query or socket close.
 def attempt(label,action):
 try:return action()
 except BaseException as error:
 failures.append(label+': '+type(error).__name__+': '+str(error))
 if unit_attempted:
 attempt('stop '+unit,lambda:ctl('stop',unit))
 status=attempt('unit state',lambda:ctl('show',unit,'-p','ActiveState','-p','MainPID','-p','ControlPID'))
 if status is not None:
 properties=dict(line.split('=',1) for line in status.splitlines() if '=' in line)
 quiescent=properties.get('ActiveState') in ('inactive','failed') and properties.get('MainPID')=='0' and properties.get('ControlPID')=='0'
 if connection is not None:attempt('accepted socket close',connection.close)
 if listener is not None:attempt('listener close',listener.close)
 if quiescent:
 if unit_attempted:
 attempt('reset failed state',lambda:ctl('reset-failed',unit))
 for path in reversed(owned_files):attempt('file '+str(path),lambda path=path:remove_file(path))
 if unit_attempted:attempt('daemon reload',lambda:ctl('daemon-reload'))
 if unit_attempted and os.path.lexists(state):
 if state.is_symlink():failures.append('unexpected state symlink; retained '+str(state))
 else:
 attempt('state result',lambda:remove_file(state/'result.json'))
 attempt('state directory',state.rmdir)
 if account_attempted:
 attempt('account '+name,delete_account)
 attempt('group '+name,delete_group)
 if root is not None:attempt('temporary directory '+str(root),root.rmdir)
 for path in owned_files+([state] if unit_attempted else [])+([root] if root is not None else []):
 attempt('absence '+str(path),lambda path=path:require_empty(str(path) if os.path.lexists(path) else ''))
 if account_attempted:
 attempt('account absence',lambda:require_empty(name if account_exists() else ''))
 attempt('group absence',lambda:require_empty(name if group_exists() else ''))
 else:
 failures.append('unit may still run; retain its unit, account/group, state and files for manual recovery: '+name)
 if failures:
 message='cleanup incomplete or uncertain: '+'; '.join(failures)
 try:print(message,file=sys.stderr)
 except (OSError,ValueError):pass
 if primary is None:raise RuntimeError(message)

This revised fixture is unrun. Use an exclusively controlled disposable system; root or container markers do not prove isolation. Guarded setup checks scratch/account/group preexistence, exclusively creates files and sets worker/directory modes to 0644/0755; ACLs still require verification. Management commands time out after ten seconds; unit start/stop limits are five seconds, accept two. Stop failure does not skip state checks or socket closure. Uncertain service state retains its files/account for manual recovery. Otherwise removals and residual checks are independent. Errors preserve the original exception; cleanup-only failures also fail. Inspect printed names after timed-out operations settle. Process creation, filesystem/NSS stalls and abrupt termination remain unbounded.

The baseline intentionally makes one sample file writable by that account. It also creates a marker in the manager’s temporary-directory view and a listener on its loopback interface. Before restrictions, the worker appends to the file, sees the marker, connects to the listener, and reports a zero NoNewPrivs flag. This establishes that ordinary permissions or an absent listener did not already prevent the operations.

The hardened invocation adds the four execution settings and repeats the same probes. Its outside write fails with EROFS, rather than merely failing because of file ownership. The original temporary marker is absent from its view, and connecting to the original listener produces ECONNREFUSED. The process reports NoNewPrivs=1. These observations test actual process behavior, not only whether the unit parser accepted a spelling.

Useful writes still work: StateDirectory creates the service’s managed writable state location, and the private temporary directory accepts its own scratch file. The scratch file seen from the manager’s namespace still contains the baseline value, demonstrating that the second write occurred in a different temporary view. The outside sample also retains its previous contents.

The script checks these exact outcomes and validates the changed unit before activation. It does not test exploitation, syscall filtering, or BPF address policies. A successful configuration check is therefore supplemented by runtime evidence for the particular controls exercised, with no claim that every possible access path has been contained.

Keep required access narrow and visible

Apply restrictions to the actual deployed service definition, including its drop-ins and execution identity. A writable-path exception changes the mount view; it does not grant ordinary Unix permission or turn an inherently read-only filesystem writable. Prefer managed state, cache, or runtime directories when their lifetime and ownership match the application’s data.

After hardening, exercise startup, normal requests, maintenance, reload, and shutdown where the application supports them. A command that works once may still fail later when it rotates a file or opens a new connection. Inspect the specific denial and adjust the necessary boundary. Retain an operational check for useful service behavior alongside the evidence that the unwanted operation remains blocked.

Sources

systemd 252 execution environment, mount namespaces, managed directories, private networking, and privilege controls, checked 2026-09-12.
systemd 252 resource control, IP allow/deny precedence and implementation requirements, checked 2026-09-12.
systemd 252 analysis commands, unit verification and its scope, checked 2026-09-12.

Systemd · XB Field Notes

systemd dependencies separate activation, ordering, and readiness

When it breaks

Check it quickly

Match dependencies to failure semantics

Further reading

Sources

Socket activation starts a service only when needed

The listener waits first

Service wakes on demand

Trade speed for memory

When it breaks

Check it quickly

Sources

systemd-tmpfiles applies creation and age policies

When it breaks

Check it quickly

Match housekeeping to the producer

Further reading

Sources

A systemd oneshot completes one activation

When it breaks

Check it quickly

Give the task its own correctness guarantees

Further reading

Sources

Socket activation passes a listener to a compatible service

When it breaks

Check it quickly

Design the handoff and shutdown together

Further reading

Sources

systemd restart policies need explicit failure and rate rules

When it breaks

Check it quickly

Recover the service, not just its counter

Further reading

Sources

The systemd journal needs an explicit retention contract

When it breaks

Check it quickly

Size the evidence window

Further reading

Sources

Query the journal with explicit field and priority rules

When it breaks

Check it quickly

Preserve context when exporting results

Further reading

Sources

A systemd timer schedules a service, not its success

When it breaks

Check it quickly

Migrate the execution contract too

Further reading

Sources

systemd-sysusers creates missing service accounts

When it breaks

Check it quickly

Make packaging and identity policy explicit

Further reading

Sources

Linger extends a systemd user manager beyond login

When it breaks

Check it quickly

Operate an unattended account deliberately

Further reading

Sources

Check timesyncd’s service state and clock evidence separately

When it breaks

Check it quickly

Build a time requirement that can be verified

Further reading

Sources

EnvironmentFile supplies values when systemd executes a process

When it breaks

Check it quickly

Keep configuration visibility selective

Further reading

Sources