A service manager turns process startup, failure, and shutdown into observable state transitions. With systemd, a unit can name the foreground command, its restart policy, and its output destination. That removes the need for a shell loop that repeatedly launches a daemon without recording why it stopped. It does not prove that the running application can answer requests or that restarting repairs the underlying fault.

Type simple reports startup early; Type exec waits for program execution, not application readiness. In the source fixture SIGKILL causes one bounded on-failure replacement under a 200 millisecond delay and a start limit configured as burst 3 in 10 seconds. A different MainPID, one restart, matching startup marker and logs for both InvocationIDs establish specific lifecycle facts. Explicit manager stop remains inactive. Hangs, endpoint health and boot enablement require separate policies or checks.
Track the managed process and its invocation, bound restart attempts, and test readiness separately. An intentional stop is distinct from a crash, and a restart count is not an availability metric.

When it breaks

Choose a service type that matches the program’s startup contract. Type=simple considers startup successful early, before the executable has necessarily been invoked successfully. Type=exec waits for successful execution of the program, so a missing executable can fail the start job directly. Neither type waits for the application to finish connecting to its database or loading its state.

A program that backgrounds itself can confuse a unit expecting a foreground main process. Use the program’s foreground mode and let the manager own its lifetime, or choose the documented service type for its actual daemonization protocol. Avoid guessing a PID from a process-name search: names can collide, and wrappers can leave the wrong process designated as the service’s main process.

Restart policy also needs explicit limits. Restart=on-failure covers selected failure exits, not every possible termination signal or every unhealthy condition. A hung process that never exits will not automatically become a restart event. Combine meaningful application health observation with a bounded restart policy rather than equating an active unit with successful service delivery.

Check it quickly

Run the following Python script as an administrator in a disposable Linux environment whose PID 1 is systemd. It writes only named temporary runtime units and a temporary working directory. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. Confirm the two unit filenames are unused before executing:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,time

unit='xbcatm-supervise-lab.service';bad='xbcatm-exec-failure-lab.service'
unit_path=Path('/run/systemd/system')/unit;bad_path=unit_path.with_name(bad)
assert not unit_path.exists() and not bad_path.exists()
def run(args,success=True):
    result=subprocess.run(args,text=True,capture_output=True,timeout=10)
    assert (result.returncode==0)==success,(args,result.returncode,result.stderr)
    return result.stdout.strip()
def value(name,key):return run(['systemctl','show',name,'-p',key,'--value'])
def wait_for(check):
    deadline=time.monotonic()+5
    while not check():
        assert time.monotonic()<deadline,'state transition timed out'
        time.sleep(0.02)
with TemporaryDirectory(prefix='xbcatm-supervise-',dir='/run') as directory:
    root=Path(directory);worker=root/'worker.py';ready=root/'ready'
    worker.write_text("""from pathlib import Path
import os,signal,sys
signal.signal(signal.SIGTERM,lambda *_: sys.exit(0))
Path(__file__).with_name('ready').write_text(str(os.getpid()))
print('worker ready',flush=True)
while True: signal.pause()
""")
    unit_path.write_text(f'''[Unit]
Description=Disposable supervision exercise
StartLimitIntervalSec=10s
StartLimitBurst=3
[Service]
Type=exec
ExecStart=/usr/bin/python3 -u {worker}
Restart=on-failure
RestartSec=200ms
TimeoutStopSec=2s
StandardOutput=journal
StandardError=journal
''')
    bad_path.write_text(f'[Service]\nType=exec\nExecStart={root}/missing-program\n')
    try:
        run(['systemd-analyze','verify',str(unit_path)])
        run(['systemctl','daemon-reload']);run(['systemctl','start',unit])
        wait_for(lambda: ready.exists() and ready.read_text()==value(unit,'MainPID'))
        first=value(unit,'MainPID');assert first!='0'
        first_invocation=value(unit,'InvocationID')
        run(['systemctl','kill','--kill-who=main','--signal=KILL',unit])
        wait_for(lambda: value(unit,'MainPID') not in ('0',first) and ready.read_text()==value(unit,'MainPID'))
        assert int(value(unit,'NRestarts'))==1
        print('SIGKILL: replacement PID confirmed; NRestarts=1')
        run(['journalctl','--sync'])
        for invocation in (first_invocation,value(unit,'InvocationID')):
            logs=run(['journalctl','_SYSTEMD_INVOCATION_ID='+invocation,'--no-pager','-o','cat'])
            assert 'worker ready' in logs
        print('journal contains both application startups')
        run(['systemctl','stop',unit]);time.sleep(0.3)
        assert value(unit,'ActiveState')=='inactive' and value(unit,'MainPID')=='0'
        print('manager stop leaves service inactive')
        run(['systemctl','daemon-reload']);run(['systemctl','start',bad],success=False)
        assert value(bad,'Result')=='exit-code' and value(bad,'ExecMainStatus')=='203'
        print('Type=exec reports missing executable: status 203/EXEC')
    finally:
        run(['systemctl','stop',unit,bad])
        for name in (unit,bad):
            if value(name,'ActiveState')=='failed':
                run(['systemctl','reset-failed',name])
        unit_path.unlink();bad_path.unlink(missing_ok=True)
        run(['systemctl','daemon-reload'])

The worker stays in the foreground and publishes its own startup marker. The script waits until that marker agrees with the unit’s main PID, then sends SIGKILL specifically to the managed main process. A different PID and one automatic restart confirm that the manager actually replaced the terminated process.

Journal checks use the two invocation identifiers, so older entries from an earlier run cannot satisfy the assertion. Each invocation must contain the worker’s startup message. This checks both lifecycle replacement and attribution of the output produced by the two actual workers, rather than simply finding a familiar phrase somewhere in the unit’s accumulated history.

An explicit manager stop leaves the unit inactive with no main PID after the restart delay has passed. This is an intentional stop, not another crash to undo. The separate failing unit uses a nonexistent executable; its start command fails and records status 203/EXEC. That failure concerns process execution, not a database connection or application-level health probe.

The script verifies the valid unit definition before starting it and removes both runtime unit files afterward. Syntax verification and successful execution are separate evidence: the first checks the manager’s interpretation of configuration, while the second observes the behavior of a real process. Neither substitutes for checking a production endpoint.

Make the lifecycle operationally useful

Keep the installed unit definition, runtime configuration, and application readiness signal consistent. Ordering another unit after this service orders manager startup completion; if a dependent application needs a stronger readiness promise, use a supported notification protocol or an explicit application-level check. Retrying transient dependencies may still be part of the application’s own startup behavior.

Use systemctl show for specific state properties and the journal for the corresponding invocation. A unit’s current PID can change between observations, so identify the instance when diagnosing repeated failures. Track both restart frequency and whether useful work resumes; a successful restart counter is not an availability metric by itself.

Runtime files under /run/systemd/system are suitable for this disposable exercise. A deployed service also needs an intentional installation and enablement policy, appropriate user permissions, and a tested shutdown path. Changing a unit definition requires reloading the manager’s configuration; starting a service once does not automatically arrange for it to start at the next boot.

Further reading

Sources