Unix · XB Field Notes

Fork creates a process; exec replaces its program

Tue, 12 Oct 2027 00:00:00 +0000

Fork followed by exec is one common Unix process-launch model. fork creates a child process, while a successful execve replaces the calling process’s program image. They solve different problems and have different inheritance rules. Understanding the distinction matters when arranging input, output, environment, and lifecycle management for a command started by another program.

fork creates the child; successful exec replaces its program. Private memory, shared resources, inherited descriptors and explicit environment values cross different boundaries, so verify both launch behavior and the child’s actual completion.

When it breaks

Fork does not make all subsequent state independent. Ordinary private memory diverges between parent and child, commonly through copy-on-write, but shared mappings remain shared. Inherited descriptors also refer to the same open file descriptions as their parent counterparts, so ordinary file offsets can move in both processes when either one reads. Copying the descriptor table does not independently reopen each file.

Exec does not simply preserve every launch setting. Descriptors marked close-on-exec close during successful execution, while other eligible descriptors survive. execve receives an explicit environment array instead of automatically reproducing the caller’s current environment. Caught signal dispositions reset and old memory mappings disappear, even though process identity and several other attributes continue.

Forking a multithreaded process needs particular care. The child contains only the calling thread but can inherit locks whose owning threads are absent. Arbitrary library work in the gap can deadlock. The allowed pre-exec operations are therefore constrained; a convenient single-threaded demonstration must not become an unrestricted recipe for manipulating a live threaded server after fork.

Check it quickly

Save this as forkexec.py and run python3 forkexec.py in a disposable Linux environment. The recorded run used Linux 6.12.76 and Python 3.11.2. This fixture is deliberately single-threaded. It uses one private Python value, one explicitly shared anonymous mapping, a temporary data file, and a pipe for a small result:

from pathlib import Path
from tempfile import TemporaryDirectory
import errno,json,mmap,os,sys

program=r'''
import errno,json,os,sys
original_pid,kept,dropped,report=map(int,sys.argv[1:])
assert os.getpid()==original_pid
try:os.fstat(dropped)
except OSError as error:assert error.errno==errno.EBADF
else:raise AssertionError('close-on-exec descriptor survived')
assert os.read(kept,2)==b'ef'
assert os.environ['XB_FORK_VALUE']=='child-value'
os.write(report,json.dumps({'pid_unchanged':True,'kept_descriptor':'ef','close_on_exec':'closed','explicit_environment':'child-value'}).encode())
'''
with TemporaryDirectory(prefix='xbcatm-fork-') as directory:
 data=Path(directory)/'data';data.write_bytes(b'abcdef')
 descriptor=os.open(data,os.O_RDONLY);dropped=os.dup(descriptor)
 read_end,write_end=os.pipe();os.set_inheritable(descriptor,True);os.set_inheritable(write_end,True)
 assert not os.get_inheritable(dropped)
 private=['parent'];shared=mmap.mmap(-1,8,flags=mmap.MAP_SHARED);shared[:]=b'parent!!'
 assert os.read(descriptor,2)==b'ab'
 sys.stdout.flush();pid=os.fork()
 if pid==0:
 try:
 os.close(read_end);private[0]='child';shared[:]=b'child!!!'
 assert os.read(descriptor,2)==b'cd'
 try:os.execve(str(Path(directory)/'missing-program'),['missing-program'],{})
 except FileNotFoundError:pass
 else:raise AssertionError('missing exec unexpectedly succeeded')
 os.execve(sys.executable,[sys.executable,'-c',program,str(os.getpid()),str(descriptor),str(dropped),str(write_end)],{'XB_FORK_VALUE':'child-value'})
 except BaseException as error:
 os.write(write_end,('ERROR '+repr(error)).encode());os._exit(1)
 os.close(write_end)
 try:
 waited,status=os.waitpid(pid,0);report=os.read(read_end,4096)
 assert waited==pid and os.WIFEXITED(status) and os.WEXITSTATUS(status)==0,report
 result=json.loads(report);assert result['pid_unchanged']
 assert private==['parent'] and shared[:]==b'child!!!'
 assert os.lseek(descriptor,0,os.SEEK_CUR)==6
 print('fork: private Python state diverges, MAP_SHARED changes and open-description offsets remain shared')
 print('missing exec returns to the child; successful exec keeps its PID, honors close-on-exec, and receives an explicit environment')
 print('parent reaped a successful exit with waitpid and retained its original private value')
 finally:
 shared.close()
 for fd in [descriptor,dropped,read_end]:os.close(fd)

Before fork, the parent reads ab from abcdef. The child changes its private Python list and the shared mapping, then reads cd through the inherited descriptor. After the child finishes, the parent’s list still contains its original value, but the shared mapping contains the child’s bytes. Both results are required to describe memory inheritance accurately.

The child first tries to execute a nonexistent file. That ordinary missing-path failure raises FileNotFoundError, and the child continues running its existing program. It then successfully executes Python again with a supplied program and an explicit environment. The example verifies this recoverable failure case; it does not generalize every rare execution failure into the same recovery behavior.

The replacement program compares its PID with the child’s PID passed before exec and finds them equal. Exec has replaced code within the child, not created a third process. The explicitly provided environment contains the fixture value, which the replacement reads directly; it does not depend on a login shell or an implicit profile-loading step.

The retained data descriptor reads ef, showing that the original parent’s read, the child’s read, and the replacement program’s read advanced one shared open-description offset. Meanwhile, the descriptor left non-inheritable fails inspection with EBADF in the replacement. The fixture deliberately marks only the descriptors needed across exec as inheritable. Descriptor numbers can be reused by the new program; this EBADF check assumes no persistent replacement open occupies that number during startup.

The parent uses waitpid to collect the child’s status and confirms a normal zero exit. It then observes offset six through its own descriptor and closes its resources. Waiting obtains lifecycle information and reaps the exited child; the child’s numerical PID alone is not a substitute for collecting completion or verifying which launch succeeded.

Make launch ownership explicit

Define which descriptors a child needs, close unused pipe ends, and choose environment values deliberately. An extra inherited pipe writer can prevent a reader from seeing EOF even after the intended producer exits. Descriptor lifetime is therefore part of the communication protocol, not merely a cleanup detail after program startup.

Prefer the runtime’s supported subprocess or spawn interface when it expresses the needed setup. Such interfaces can use mechanisms other than a literal fork/exec pair. Their documented descriptor and environment behavior is the relevant application contract; assuming every launcher follows an identical internal sequence can produce incorrect portability and safety conclusions.

Distinguish launch failure, program failure, and failure to observe completion. A successful exec never returns to the previous program, whereas the replacement can later exit unsuccessfully. Record the actual child status and preserve enough context to identify the attempted command, without logging confidential environment values. That gives launch supervision a verifiable outcome rather than an inferred success.

Sources

fork(2), private memory, shared mappings, inherited descriptions, and thread restrictions, checked 2026-09-12.
execve(2), image replacement, explicit environment, preserved PID, and close-on-exec, checked 2026-09-12.
waitpid(2), collecting child status and reaping an exited child, checked 2026-09-12.

Setuid changes effective identity at exec

Sat, 02 Oct 2027 00:00:00 +0000

Setuid on a Linux executable can change the process’s effective user ID to the file owner’s ID when exec succeeds. It does not replace every credential with the owner’s login identity. A privileged helper can therefore perform a controlled operation on another account’s behalf, but its input handling and privilege lifetime become part of the security boundary. The mode bit alone proves neither that a transition occurred nor that the helper is safe.

Distinguish the file bit from process credentials

The usual set-user-ID transition changes the effective UID, then copies it into the saved set-user-ID. The real UID remains unchanged. Group credentials are separate: setuid does not automatically install the file owner’s supplementary groups. Linux also has filesystem IDs and capability sets, so real, effective and saved UIDs are useful observations rather than a complete description of authority.

Inspect the deployed file instead of assuming a distribution-wide mode. A privileged password-changing program might be installed differently in a container or customized image. A lowercase s in the owner-execute position indicates both setuid and owner execute; uppercase S indicates setuid without that execute bit. Neither display substitutes for checking ownership, the actual executable, and the process context in which it will run.

Execution can suppress the transition

Linux documents no_new_privs, nosuid mounts and tracing as conditions that suppress the privilege transition; some documented combinations can fail execution instead. File capabilities also have execution constraints. Do not disable one of these protections merely to make a demonstration work. Establish which policy applies to the target path and process, and report the resulting limitation.

Linux ignores set-user-ID and set-group-ID bits on interpreter scripts. Moving a helper from a native executable to a script while preserving its mode therefore does not preserve elevation behavior. Making a general-purpose interpreter privileged is not a safe substitute. Another executable launched by a privileged program may deliberately drop credentials, so running a separate id command is not necessarily a faithful observation of the original helper’s credentials.

Privilege dropping needs a complete design

The setuid system call is distinct from the executable’s mode bit. A sufficiently privileged caller can change real, effective and saved UIDs; an unprivileged caller has a narrower set of permitted transitions. Seteuid can support a reversible effective-UID change when a permitted saved identity remains available. Always check the syscall’s return value before assuming the intended state was reached.

Removing a saved UID is only part of permanent privilege removal. Supplementary groups, retained capabilities, open privileged descriptors, namespaces and later execution paths also matter. A single setuid call is not a universal privilege-drop recipe. The required sequence depends on the program’s existing credentials and resources; security-sensitive helpers need design review rather than a copied one-line pattern.

When it breaks

A mode listing with an s can mask a suppressed transition, while an overly broad helper can expose real authority through unsafe path lookup or user-selected configuration. Loader protections restrict dangerous environment variables in secure-execution mode, but they do not validate the application’s inputs. Avoid shell-based command construction and define exactly which files, arguments and operations the helper may use.

Capabilities can narrow a particular grant, such as permission to bind privileged ports, without making every other permission disappear. Existing user access and other capabilities still apply, and capability bounding and securebits rules qualify root behavior. Likewise, namespace-local root is not host root, but mapped ordinary-user permissions remain meaningful. Socket activation can supply an already-open listener; it does not remove the need to secure the service handling it.

Setuid affects selected exec-time credentials under documented guards. The retained inspection does not execute the target or prove elevation; current runtime remains held.

Check it quickly

On Linux with Bash, GNU stat, util-linux findmnt, libcap getcap and Python 3, run the complete block below. It reads one installed file and the new Python process’s own credential status. It never executes passwd, changes permissions, changes credentials, mounts anything or creates a namespace. The explicit pathname avoids a helper search, but separate metadata reads do not atomically bind a later execution to the same file. This review did not run the inspection; assertions require a trusted test environment.

BASH_ENV= bash --noprofile --norc <<'BASH'
set -eu
target=/usr/bin/passwd
test -f "$target"
stat -c 'mode=%A numeric=%a owner=%u group=%g path=%n' "$target"
findmnt --noheadings --output TARGET,VFS-OPTIONS --target "$target"
getcap -v "$target"
python3 - <<'PYTHON'
import os
from pathlib import Path

status = dict(line.split(':', 1) for line in
 Path('/proc/self/status').read_text().splitlines() if ':' in line)
uids = tuple(map(int, status['Uid'].split()))
gids = tuple(map(int, status['Gid'].split()))
assert uids[:3] == os.getresuid()
assert gids[:3] == os.getresgid()
assert len(uids) == len(gids) == 4
for key in ['Uid', 'Gid', 'Groups', 'NoNewPrivs', 'CapEff', 'CapBnd']:
 print(f'{key}:{status[key]}')
print('PASS: inspected metadata and current credentials; target not executed')
PYTHON
BASH

Expect the actual mode and numeric ownership, the filesystem’s mount options, and the target’s capability listing. An empty capability set does not prove absence of the setuid bit, and a setuid bit does not imply a file capability. The Python assertions compare the first three UID and GID fields with the corresponding system-call results; the fourth fields are filesystem IDs.

The PASS line establishes this read-only inspection only. Values vary by host, and no expected root identity is hardcoded. This exercise does not demonstrate elevation, irreversible dropping, or exploit resistance. Those require a separately authorized disposable security test and observations inside the actual helper, without weakening the host’s protections.

Sources

Linux execve, effective-ID changes, unchanged credentials and suppression conditions, checked 2026-10-15.
Linux credentials, supplementary groups and filesystem IDs, checked 2026-10-15.
Linux setuid, privileged and unprivileged transitions and required error checking, checked 2026-10-15.
Linux seteuid, permitted effective-ID transitions, checked 2026-10-15.
Linux capabilities, capability limits and credential interactions, checked 2026-10-15.
Linux dynamic linker, secure-execution environment restrictions, checked 2026-10-15.
Linux proc_pid_status, credential and capability status fields, checked 2026-10-15.

The sticky bit restricts removal in a shared directory

Fri, 01 Oct 2027 00:00:00 +0000

The sticky bit on a Linux directory adds an ownership restriction to removing or renaming entries. It is commonly combined with broad directory write and search permissions so several users can create temporary files without freely removing one another’s entries. The directory permission check and the sticky ownership check both matter; sticky is an additional condition, not a replacement for normal access control.

When it breaks

Without sticky, a user with the necessary write and search permissions on a directory can often remove another user’s file even when that file’s mode grants no access. Unlink changes the directory’s name-to-object relationship. It does not require permission to read or overwrite the file contents, so making the file itself mode 000 does not protect its pathname in a broadly writable directory.

With sticky enabled, ordinary users must additionally own the entry or the containing directory to perform the relevant removal. A privileged process with the appropriate CAP_FOWNER capability can bypass that ownership restriction. Sticky therefore limits peers under a specific permissions model; it does not protect against every administrator or the directory’s own owner.

Sticky does not stop permitted content access. Another user may still read or overwrite a mode-0666 file while being unable to unlink it. The mode also does not automatically propagate to newly created subdirectories. Confusing pathname protection, content privacy, and inheritance can leave a shared tree with materially different behavior from its apparently protected top level.

Sticky restricts entry removal, with ownership and capability exceptions. File modes separately govern content; fixture outcomes are expected, not newly observed.

Check it quickly

Save as sticky.py and run python3 sticky.py only in an authorized disposable Linux environment with enabled assertions. The controller needs root privileges to assign three numeric identities and launch ordinary children; it creates no accounts. The article reports Linux 6.12.76/Python 3.11.2, but no raw logs were recovered and this review did not rerun it. Filesystem ownership, UID mappings and child capabilities must match the intended permission model:

from pathlib import Path
from tempfile import TemporaryDirectory
import errno,os,stat,subprocess,sys

directory_owner=61000;file_owner=61001;other_user=61002
probe_code=r'''
import os,sys
path=sys.argv[1];operation=sys.argv[2]
try:
 if operation=='unlink':os.unlink(path)
 elif operation=='rename':os.rename(path,path+'-renamed')
 elif operation=='write':
 fd=os.open(path,os.O_WRONLY|os.O_TRUNC)
 try:os.write(fd,b'other-user-write')
 finally:os.close(fd)
 elif operation=='read':
 fd=os.open(path,os.O_RDONLY);os.close(fd)
 elif operation=='mkdir':os.mkdir(path,0o777)
 else:raise AssertionError('unknown operation')
except OSError as error:print(error.errno)
else:print(0)
'''
def probe(uid,path,operation):
 result=subprocess.run([sys.executable,'-c',probe_code,str(path),operation],user=uid,group=uid,extra_groups=[],capture_output=True,text=True,timeout=5)
 assert result.returncode==0,result.stderr
 return int(result.stdout)
with TemporaryDirectory(prefix='xbcatm-sticky-') as directory:
 root=Path(directory);root.chmod(0o755);shared=root/'shared';shared.mkdir();shared.chmod(0o777)
 os.chown(shared,directory_owner,directory_owner);victim=shared/'victim'
 def make_victim(mode):
 victim.write_bytes(b'fixture');os.chown(victim,file_owner,file_owner);victim.chmod(mode)
 make_victim(0o000)
 assert probe(other_user,victim,'unlink')==0
 print('without sticky, directory write/search allows removing another owner’s mode-000 file')
 shared.chmod(0o1777);make_victim(0o666)
 assert probe(other_user,victim,'unlink')==errno.EPERM
 assert probe(other_user,victim,'rename')==errno.EPERM
 assert probe(other_user,victim,'write')==0 and victim.read_bytes()==b'other-user-write'
 victim.chmod(0o600);assert probe(other_user,victim,'read')==errno.EACCES
 print('sticky blocks outsider unlink/rename, but file permissions still govern reading and writing')
 assert probe(file_owner,victim,'unlink')==0
 make_victim(0o600);assert probe(directory_owner,victim,'unlink')==0
 print('both the entry owner and the directory owner may remove it')
 child=shared/'child';assert probe(other_user,child,'mkdir')==0
 assert not child.stat().st_mode&stat.S_ISVTX
 shared.chmod(0o1766);assert stat.filemode(shared.stat().st_mode).endswith('T')
 assert probe(other_user,shared/'blocked','mkdir')==errno.EACCES
 shared.chmod(0o1777);assert stat.filemode(shared.stat().st_mode).endswith('t')
 print('sticky is not inherited by a new subdirectory; uppercase T means other-execute is absent')

The first case uses a 0777 directory owned by one identity and a mode-000 file owned by another. A third user successfully unlinks that file. This demonstrates that directory write/search permissions can allow removal independently of the file’s content permissions. Root sets up the fixture, but the tested removal occurs under the third, unprivileged identity.

The fixture expects EPERM for the third user’s unlink and same-directory rename under 1777; the manual also permits EACCES for sticky restrictions. The same user successfully overwrites the file when its mode is 0666. The write opens an existing file without O_CREAT, avoiding a different Linux protected-regular-file policy that could otherwise obscure which permission rule the exercise is measuring.

Changing the file to 0600 causes the third user’s read attempt to fail with EACCES. Sticky remained enabled throughout, so it is the file-permission change that altered content access. The two error classes arise from different checks and should not be collapsed into a claim that sticky makes each temporary file private.

The file’s owner successfully removes its entry, and a newly recreated file can also be removed by the directory owner. Both outcomes are checked using ordinary child identities. These cases demonstrate the two ownership exceptions directly instead of relying on root, whose capabilities would make the result ambiguous.

Next, the third user creates a subdirectory. It has no sticky bit automatically inherited from the parent. Changing the parent to 1766 produces an uppercase T in its permission display, and another creation attempt fails because directory search permission is missing. Restoring 1777 produces lowercase t, which combines sticky with the other-execute bit.

Design shared storage deliberately

Use sticky when the intended policy allows shared creation but restricts peer removal. Set file permissions or ACLs separately for the intended readers and writers. If each user’s data must remain confidential, verify access under another identity; the presence of a sticky directory alone supplies no evidence that its files are unreadable.

Treat directory ownership as authority over its entries. A directory owner who can remove files may disrupt other users even while unable to read their contents. In a hierarchy, inspect the permissions of the actual parent containing each entry, because a protected ancestor does not automatically apply its sticky behavior to descendants.

For temporary-file creation, use an API that creates a fresh, suitably restricted object without trusting a predictable preexisting name. Sticky helps with a particular shared-directory removal problem, but it does not solve symbolic-link substitution or every unsafe open pattern. Verify creation, subsequent content access, and cleanup under the actual identities and filesystem used by the application.

Sources

inode(7), sticky directories, file ownership, and displayed permission bits, checked 2026-09-12.
unlink(2), directory write/search permissions and sticky removal restrictions, checked 2026-09-12.
capabilities(7), CAP_FOWNER exceptions to sticky-directory ownership checks, checked 2026-09-12.
rename(2), rename permission checks in sticky directories, checked 2026-09-12.

Environment variables are a per-process thing

Sat, 25 Sep 2027 00:00:00 +0000

An environment is a collection of name-value strings supplied to a program when it starts. A child normally receives the environment selected by its parent; changing that child’s copy does not rewrite the parent’s or a sibling’s copy. This is a useful configuration boundary, but it is not a promise that the command has no side effects. The program can still modify shared files, databases, or other resources using those settings.

Follow what the immediate launcher supplies. Environment copies do not write back, but a subshell, a new Bash program and sourcing in the current shell preserve different kinds of state.

Export selects values for another program

A shell variable and an environment entry are related but distinct. Assigning DEMO_SETTING=local creates or updates a shell parameter. export DEMO_SETTING marks it for inclusion when the shell launches external commands. A new Bash process normally cannot see an unexported shell variable, even though the parent can print it. Merely placing an assignment in a configuration file does not load that file into every shell.

A value inherited through the environment is already marked for export by Bash. Changing its value retains that attribute, so a later grandchild can inherit the changed value without another export statement. unset removes the variable; Bash’s export -n removes its export attribute while retaining the shell value. The relevant question is which environment the immediate launcher supplied, not whether every ancestor repeated an export command.

A subshell is not a freshly executed shell

Parenthesized command groups and ordinary $(...) command substitution run in a subshell environment copied from the current shell. They can see ordinary shell variables that were never exported. Their assignments still do not flow back into the outer shell. This differs from executing a separate shell program, which receives exported entries rather than a complete copy of the parent’s shell parameter table. Bash 5.3 also provides current-environment capture forms; the subshell claim here applies to $(...), not every capture syntax.

Sourcing a file with . runs its commands in the current shell. Its assignments can therefore change the caller directly; it is not an exception to process isolation because no separate process is returning those changes. A sourced file is executable shell code, not passive configuration. Only source a file whose contents you intend to execute in that context.

Per-command settings have a defined lifetime

A prefix such as DEMO_SETTING=once external-command supplies a value for that invocation without permanently changing the surrounding shell’s value. This statement concerns an external command; assignment behavior around special builtins and functions has additional shell-specific rules. Do not generalize it into a claim that every command-looking construct creates an isolated process.

At the operating system interface, execve takes an explicit environment array and replaces the current process image. It does not itself create a new process. A launcher can preserve, filter, or replace the environment, which explains why privilege boundaries, service managers, and container configuration may produce values different from an interactive terminal. Process ancestry alone is insufficient to establish the actual settings delivered to an application.

When it breaks

Editing a terminal’s environment does not update an already running service. Likewise, changing an environment file has no effect until the relevant launcher rereads it and starts the process, unless the application provides its own reload mechanism. Check the configuration path used by that service and verify the resulting behavior in the process that actually handles requests.

Treat missing and empty values separately, because applications may assign them different meanings. Environment strings cannot contain an embedded NUL, and platforms limit the combined argument and environment size, so they are unsuitable for arbitrary binary payloads or unbounded documents. Inspect only the specific nonsecret keys needed for diagnosis; dumping an entire environment can expose credentials inherited by unrelated tools.

Check it quickly

Run this Bash example. It prints only a dedicated demonstration variable, uses no external configuration file, and leaves the surrounding terminal unchanged. The small here-document passed to . contains the complete trusted configuration used for the sourcing test.

BASH_ENV= bash --noprofile --norc <<'BASH'
unset DEMO_SETTING
DEMO_SETTING=local
printf 'shell=%s\n' "$DEMO_SETTING"
bash --noprofile --norc -c 'printf "unexported-child=%s\n" "${DEMO_SETTING-unset}"'
(printf 'subshell=%s\n' "$DEMO_SETTING")
export DEMO_SETTING
bash --noprofile --norc <<'CHILD'
printf 'exported-child=%s\n' "$DEMO_SETTING"
DEMO_SETTING=child
bash --noprofile --norc -c 'printf "grandchild=%s\n" "$DEMO_SETTING"'
CHILD
printf 'parent-after-child=%s\n' "$DEMO_SETTING"
DEMO_SETTING=once bash --noprofile --norc -c 'printf "one-command=%s\n" "$DEMO_SETTING"'
printf 'parent-after-command=%s\n' "$DEMO_SETTING"
. /dev/stdin <<'CONFIG'
DEMO_SETTING=sourced
CONFIG
printf 'parent-after-source=%s\n' "$DEMO_SETTING"
BASH

The unexported child prints unset, while the parenthesized subshell sees local. After export, the new child sees local, changes its own copy, and passes child to its grandchild. The outer shell remains local. The one-command override prints once, and the following parent check is still local.

Finally, sourcing changes that outer child shell’s value to sourced. Together these observations distinguish shell parameters, exported entries, subshell copies, per-command overrides, and execution in the current shell. They establish variable scope, not the absence of application side effects or the correctness of a service’s deployment configuration.

Sources

Bash execution environment, exported entries versus the state copied into subshells, checked 2026-09-07.
Bash environment, export attributes and per-command assignments, checked 2026-09-07.
POSIX exec functions, process-image replacement and explicit environment arrays, checked 2026-09-07.
Bash command substitution, standard subshell capture versus Bash 5.3 current-environment forms, checked 2026-10-18.

A bare command the shell cannot find

Thu, 20 May 2027 00:00:00 +0000

“Command not found” identifies a lookup failure, but lookup does not always mean walking every directory in PATH. In ordinary Bash, a command word without a slash can resolve to a function, builtin, cached executable path, or a newly searched executable. An alias may already have changed the command while the input was read. Inspect the shell that actually reported the failure before reinstalling a tool that may already exist.

Resolve the command in the shell that failed. Functions, builtins, the path cache and PATH order can select different commands; a slash selects a path directly. Selection does not prove successful execution. The cache sequence is the article’s example, not a new runtime test.

Know which lookup layer answered

After expansion, Bash checks functions and then builtins before searching for an external command. type name describes the current resolution, and type -a name exposes additional definitions or candidates. command -v name is useful for testing availability in that shell, but its result need not be a filesystem path. It can identify a function or builtin, so do not blindly treat the output as a file to copy.

For external commands, directories in PATH are searched in order. An earlier executable can shadow a newer version later in the list. Bash also caches successful paths; hash -t name inspects a cached entry and hash -r clears the table. A changed file at an already cached location and a different path choice are separate issues. Starting a new terminal can hide this distinction by resetting shell state.

A slash selects a path directly

Any slash in the command word bypasses the external PATH search. ./tool addresses a file in the current directory; /opt/example/bin/tool addresses an absolute location. A bare tool does not automatically search the current directory. Empty or relative entries in PATH can make it do so, which allows the current working directory to change the selected executable.

Use explicit, trusted directories when assembling a service’s environment. A working interactive terminal does not establish that cron, a service manager, or another user has the same path or startup files. Compare their actual environments and working directories. Changing a login configuration file will not update an already running process; apply the intended environment to the process that launches the command.

When it breaks

Bash normally reports 127 when a command cannot be found and 126 when a located command cannot be executed. These values guide investigation, but they are not a complete diagnosis: an existing script can fail because the interpreter named by its shebang is missing. Read the diagnostic and inspect the selected file’s permissions and interpreter instead of repeatedly appending directories.

Do not “fix” lookup by adding writable project directories ahead of trusted tools or by using administrator privileges indiscriminately. First establish whether a function, an alias, or a cached location is responsible. command name bypasses a same-named function in Bash; an explicit executable path bypasses path selection. Neither makes the selected program itself trustworthy or repairs its missing dependencies.

Check it quickly

This Bash example creates two disposable executables with the same name. Its temporary directories are the only additions to PATH, and all changes stay inside the child shell. Cleanup names each file explicitly. No installed executable is modified.

Setup uses errexit to stop if fixture creation fails, then disables it before deliberately testing failures. This review observed the documented 126 and 127 results on Bash 5.2.37. A command-not-found handler can supply its own status for a failed bare-name search; the absent-path test deliberately includes a slash and does not exercise that hook.

bash --noprofile --norc <<'BASH'
set -eu
demo_dir=$(mktemp -d)
trap 'rm -f "$demo_dir/one/demo-tool" "$demo_dir/two/demo-tool" "$demo_dir/blocked"; rmdir "$demo_dir/one" "$demo_dir/two" "$demo_dir"' EXIT
mkdir "$demo_dir/one" "$demo_dir/two"
printf '#!/bin/sh\nprintf "one\\n"\n' > "$demo_dir/one/demo-tool"
printf '#!/bin/sh\nprintf "two\\n"\n' > "$demo_dir/two/demo-tool"
chmod +x "$demo_dir/one/demo-tool" "$demo_dir/two/demo-tool"
PATH="$demo_dir/one:$demo_dir/two:$PATH"
demo-tool
hash -p "$demo_dir/two/demo-tool" demo-tool
demo-tool
hash -r
demo-tool
demo-tool() { printf 'function\n'; }
demo-tool
command demo-tool
unset -f demo-tool
"$demo_dir/two/demo-tool"
: > "$demo_dir/blocked"
chmod 600 "$demo_dir/blocked"
set +e
if "$demo_dir/blocked" 2>/dev/null; then
 printf 'unexpected execution\n'
else
 printf 'not-executable=%s\n' "$?"
fi
if "$demo_dir/missing" 2>/dev/null; then
 printf 'unexpected file\n'
else
 printf 'missing=%s\n' "$?"
fi
BASH

The first three lines should be one, two, one: normal search, an explicitly installed cache entry, and a fresh search after clearing it. Next come function, one, and two, distinguishing function dispatch, function bypass, and an explicit path. Finally, the nonexecutable file returns 126 and the absent path returns 127.

This proves selection behavior with controlled files. It does not reproduce every operating system loader error or service environment. For a real incident, record the resolved command and its version in the failing environment, then verify the corrected invocation there. A successful lookup in an unrelated terminal is insufficient evidence that the original failure is repaired.

Sources

POSIX shell command search, command search rules and the not-found exit status, checked 2026-06-01.
Bash reference manual: command search and execution, function and builtin precedence plus cached executable paths, checked 2026-09-07.

Graceful shutdown stops work, not life

Sat, 19 Dec 2026 00:00:00 +0000

Graceful shutdown is a transition in how a service accepts and finishes work. Receiving a signal is only the trigger. The useful guarantee is narrower than “nothing is lost”: stop admitting new business work, account for what was already accepted, allow bounded completion, and report honestly when that bound expires.

A signal begins coordination

SIGTERM can be handled so an application starts its shutdown sequence. SIGKILL cannot be caught, blocked, or ignored. Verify which process receives the stop signal: a handler inside a worker does not help if a wrapper receives the signal and never forwards it.

Keep the signal handler small. Python executes its handlers in the main interpreter thread and warns against synchronization locks inside them. The fixture only sets a flag; ordinary control flow changes readiness and admission under a lock. It does not perform network I/O or wait for workers inside the handler.

Readiness should be withdrawn early enough for routing changes to begin. Admission still needs its own local gate, because existing connections and delayed routing updates can deliver requests afterward. Changing the gate and registering accepted work must use a consistent boundary.

The deadline covers a lifecycle

Track work until its real completion point. A caller timeout is not proof that a worker stopped, and a sent response is not proof that a business transaction is durable. Select the completion boundary appropriate to the operation instead of treating every closed socket as success.

Use a monotonic deadline for local elapsed time. Leave room for cleanup within the supervisor’s actual allowance. When the deadline expires, remaining work needs an explicit cancellation, abandonment, or durable handoff policy. Merely breaking out of a wait loop does not cancel a running thread.

Successful draining and incomplete shutdown should remain distinguishable in records. A particular nonzero exit code is a service contract, not a universal rule that guarantees correct recovery. A client losing a response may still need idempotency or reconciliation before retrying.

Coordinate with the supervisor

In systemd, TimeoutStopSec governs configured stop-command and service-stop waits. KillMode determines the processes targeted, not how much drain time they receive. With mixed mode, the initial termination signal targets the main process and final escalation targets remaining control-group processes; early main-process exit can also trigger that escalation. Review stop hooks and notification-based extensions in the real unit.

Kubernetes begins the termination grace period before running a configured preStop hook. The hook does not receive an independent extra drain budget. EndpointSlice termination updates and local shutdown proceed alongside each other; they are not an instantaneous global barrier after which no request can arrive.

The normal stop signal can depend on image or runtime configuration. Test the deployed process tree and stop contract rather than assuming the behavior of a locally executed program also proves container, sidecar, and proxy coordination.

When it breaks

A shutdown flag that only hides readiness can leave a keep-alive connection admitting new work forever. Conversely, closing every active socket immediately can discard requests that had enough time to finish. Stopping admission and finishing accepted work are separate decisions.

An unbounded join can exceed the supervisor’s allowance. Daemon threads allow process exit without waiting, but that is abandonment, not safe cancellation or transaction rollback. Libraries have their own shutdown contracts: Python’s socketserver shutdown must be called from a different thread than serve_forever or it deadlocks.

Observe accepted, completed, refused, and unfinished work, together with shutdown duration and exit outcome. A clean-looking exit without those distinctions can hide dropped work. The goal is a verified lifecycle, not just a registered signal handler.

Stop new business admission while counting accepted work to its real completion boundary. The source distinguishes a finished request from deliberate daemon-worker abandonment; it does not validate deployed supervisor coordination.

Check it quickly

Save this code as shutdown_check.py and run python3 shutdown_check.py on a Unix-like system with Python 3.10+, keeping assertions enabled. Both language versions ran once on 2026-10-18 using CPython 3.14.4 / Darwin 25.5.0 arm64; other versions remain untested. Each case creates a child and loopback HTTP server; SIGTERM targets that child. POST release coordinates testing, not production administration.

from concurrent.futures import ThreadPoolExecutor
from http.client import HTTPConnection, RemoteDisconnected
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import selectors
import signal
import subprocess
import sys
from threading import Condition, Event, Thread
import time


def child(grace):
 state = {"draining": False, "active": 0}
 condition, release = Condition(), Event()
 stop_requested = False

 def stop(signum, frame):
 nonlocal stop_requested
 stop_requested = True # No lock, I/O, or shutdown call in the signal handler.

 class Handler(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def reply(self, status, value):
 body = json.dumps(value).encode()
 self.send_response(status)
 self.send_header("Content-Type", "application/json")
 self.send_header("Content-Length", str(len(body)))
 self.send_header("Cache-Control", "no-store")
 self.end_headers()
 self.wfile.write(body)

 def do_GET(self):
 if self.path == "/state":
 with condition:
 snapshot = dict(state)
 self.reply(200, snapshot)
 elif self.path == "/readyz":
 with condition:
 ready = not state["draining"]
 self.reply(200 if ready else 503, {"ready": ready})
 elif self.path == "/work":
 with condition:
 admitted = not state["draining"]
 if admitted:
 state["active"] += 1
 if not admitted:
 self.reply(503, {"error": "draining"})
 return
 try:
 if release.wait(timeout=5):
 self.reply(200, {"done": True})
 else:
 self.reply(503, {"error": "fixture-timeout"})
 finally:
 with condition:
 state["active"] -= 1
 condition.notify_all()
 else:
 self.reply(404, {"error": "not-found"})

 def do_POST(self):
 if self.path == "/release": # Loopback-only test coordination, not workload admission.
 self.reply(200, {"released": True})
 release.set() # Finish the control reply before work can complete and the child exits.
 else:
 self.reply(404, {"error": "not-found"})

 signal.signal(signal.SIGTERM, stop)
 server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
 server.daemon_threads = True # Explicitly permits abandonment in the deadline case.
 serving = Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01})
 serving.start()
 print(json.dumps({"port": server.server_port}), flush=True)
 while not stop_requested:
 time.sleep(0.005)
 deadline = time.monotonic() + grace
 with condition:
 state["draining"] = True
 while state["active"] and time.monotonic() < deadline:
 condition.wait(timeout=max(0, deadline - time.monotonic()))
 unfinished = state["active"]
 server.shutdown() # Called outside the thread running serve_forever.
 serving.join(timeout=2)
 server.server_close() # Does not wait for daemon request threads.
 assert not serving.is_alive()
 print(json.dumps({"unfinished": unfinished, "deadline_expired": unfinished > 0}), flush=True)
 return 3 if unfinished else 0 # Fixture policy, not a universal exit-code convention.


def request(port, route, method="GET"):
 connection = HTTPConnection("127.0.0.1", port, timeout=3)
 try:
 connection.request(method, route)
 response = connection.getresponse()
 return response.status, json.loads(response.read())
 finally:
 connection.close()


def wait_state(port, predicate):
 deadline = time.monotonic() + 3
 while time.monotonic() < deadline:
 status, state = request(port, "/state")
 assert status == 200
 if predicate(state):
 return state
 time.sleep(0.01)
 raise AssertionError("child did not reach the expected state")


def exercise(finish):
 process = subprocess.Popen(
 [sys.executable, __file__, "child", "3" if finish else "1"],
 stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
 )
 try:
 with selectors.DefaultSelector() as selector:
 selector.register(process.stdout, selectors.EVENT_READ)
 assert selector.select(timeout=5), "child did not announce its local port"
 port = json.loads(process.stdout.readline())["port"]
 assert request(port, "/readyz") == (200, {"ready": True})
 with ThreadPoolExecutor(max_workers=1) as pool:
 active = pool.submit(request, port, "/work")
 wait_state(port, lambda state: state["active"] == 1)
 process.send_signal(signal.SIGTERM)
 wait_state(port, lambda state: state["draining"])
 assert request(port, "/readyz") == (503, {"ready": False})
 assert request(port, "/work") == (503, {"error": "draining"})
 assert request(port, "/state")[1]["active"] == 1
 if finish:
 assert request(port, "/release", "POST")[0] == 200
 assert active.result(timeout=4) == (200, {"done": True})
 output, errors = process.communicate(timeout=5)
 assert errors == ""
 record = json.loads(output)
 assert process.returncode == (0 if finish else 3)
 assert record == {"unfinished": 0 if finish else 1, "deadline_expired": not finish}
 if not finish:
 try:
 active.result(timeout=4)
 except (OSError, RemoteDisconnected):
 pass
 else:
 raise AssertionError("abandoned work must not report a successful reply")
 print("completed drain" if finish else "expired drain", record, "exit", process.returncode)
 finally:
 if process.poll() is None:
 process.kill() # Safety cleanup targets only this fixture's child.
 process.wait(timeout=5)
 process.stdout.close()
 process.stderr.close()


if __name__ == "__main__":
 if sys.argv[1:2] == ["child"]:
 raise SystemExit(child(float(sys.argv[2])))
 exercise(True)
 exercise(False)

Both cases admit one request before SIGTERM. After the drain flag becomes visible, readiness returns 503, a new business request receives 503, and the admitted request remains counted.

Releasing that request lets it return 200 before the child exits zero with no unfinished work. Without release, the one-second drain allowance expires; the child reports one unfinished request, exits three, and the waiting client receives no successful reply.

The latter case deliberately abandons a daemon worker. It does not prove cooperative cancellation, durability, SIGKILL cleanup, or safe replay. No systemd unit, Kubernetes workload, real transaction, or production traffic was exercised.

The record shows four original SIGTERM calls, 34 parent-observed loopback HTTP connections, and unchanged child arguments. Expired requests ended with RemoteDisconnected. Owned processes/listeners were absent afterward, without supervisor escalation. These are parent Python API observations, not packet capture or child-thread instrumentation.

Sources

Linux signal(7), termination signals and the uncatchable SIGKILL boundary, checked 2026-09-09.
Python signals, main-thread handlers and the warning against locks in handlers, checked 2026-09-09.
Python socketserver, shutdown-thread requirements and daemon request-thread behavior, checked 2026-09-09.
systemd.service, TimeoutStopSec stop waits and notification extensions, checked 2026-09-09.
systemd.kill, KillMode process scope and termination escalation, checked 2026-09-09.
Kubernetes: Pod lifecycle, preStop, termination grace, stop signals, and EndpointSlice transitions, checked 2026-09-09.

Give durable state and scratch space different lifecycles

Thu, 03 Dec 2026 00:00:00 +0000

Durable application state and disposable scratch files need different retention, capacity, and recovery policies. Separating their paths makes those policies easier to express, but directories alone do not create persistence or isolate disk capacity. The storage backing each path determines what survives process restart, container replacement, host failure, or cleanup. A reliable layout connects each class of data to an explicit lifecycle.

When it breaks

A cleanup job that treats one mixed directory as disposable can remove committed records alongside temporary downloads. The reverse problem is backing up large rebuildable caches while missing the small database that actually matters. File extensions and age are weak substitutes for knowing which application owns a path and whether its contents can be reconstructed from an authoritative source.

Container restart and replacement also differ. Stopping and starting the same container normally retains its writable filesystem layer. Removing that container discards the layer, while an explicitly retained named volume can survive. A tmpfs mount has another lifecycle and loses its contents when the container stops. Calling all three locations temporary hides distinctions that affect real recovery procedures.

Two separate directories on the same full filesystem still compete for that filesystem’s free space. Moving scratch from /data/tmp to /scratch without changing storage policy does not automatically protect database writes. Capacity isolation requires an appropriate mount, quota, or other enforceable budget, together with monitoring that distinguishes scratch growth from durable-state growth.

Storage lifecycles differ across stop/start and replacement. The source also checks a bounded scratch mount and a reopened same-volume database copy; neither is independent disaster recovery.

Check it quickly

Run this script using host Python and Docker after obtaining the pinned image. The recorded environment used Docker 29.3.1 and Linux arm64 Python 3.11.16 with its SQLite module. It creates one private named volume and an eight-mebibyte tmpfs scratch mount. Only its own containers and volume are removed; the small scratch exhaustion stays inside the experiment:

import subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def execute(container,code):return docker('exec',container,'python','-c',code).stdout.strip()

name='xbcatm-state-'+uuid.uuid4().hex[:10];volume=name+'-data';containers=[name+'-one',name+'-replacement']
try:
 docker('volume','create',volume)
 def start(container):
 docker('run','-d','--name',container,'--network','none','--mount','type=volume,src='+volume+',dst=/state','--tmpfs','/scratch:rw,size=8m',BASE,'python','-c','import time;time.sleep(180)')
 start(containers[0])
 execute(containers[0],'from pathlib import Path;import sqlite3;c=sqlite3.connect("/state/data.sqlite");c.execute("create table records(value text)");c.execute("insert into records values (?)",("committed",));c.commit();c.close();Path("/ephemeral").write_text("container-layer");Path("/scratch/crumb").write_text("scratch")')
 docker('stop','--time','1',containers[0]);docker('start',containers[0])
 execute(containers[0],'from pathlib import Path;import sqlite3;assert Path("/ephemeral").exists();assert not Path("/scratch/crumb").exists();c=sqlite3.connect("/state/data.sqlite");assert c.execute("select value from records").fetchall()==[("committed",)];c.close()')
 print('stop/start retains the container writable layer and volume data, while the tmpfs scratch mount loses its file')
 docker('rm','-f',containers[0]);start(containers[1])
 execute(containers[1],'from pathlib import Path;import sqlite3;assert not Path("/ephemeral").exists();c=sqlite3.connect("/state/data.sqlite");assert c.execute("select value from records").fetchall()==[("committed",)];c.close()')
 print('replacement loses the old writable layer but retains the named-volume database')
 execute(containers[1],'''import errno,os,sqlite3
fd=os.open('/scratch/fill',os.O_CREAT|os.O_WRONLY,0o600)
try:
 try:
 while True:os.write(fd,b'x'*1048576)
 except OSError as error:assert error.errno==errno.ENOSPC
finally:os.close(fd)
source=sqlite3.connect('/state/data.sqlite');source.execute('insert into records values (?)',('after-scratch-full',));source.commit()
backup=sqlite3.connect('/state/check-copy.sqlite');source.backup(backup);backup.close();source.close()
restored=sqlite3.connect('/state/check-copy.sqlite');assert restored.execute('select value from records order by rowid').fetchall()==[('committed',),('after-scratch-full',)];restored.close()
''')
 print('bounded scratch reaches ENOSPC while state still commits; a SQLite backup copy reopens with both committed records')
finally:
 docker('rm','-f',*containers,check=False);docker('volume','rm',volume,check=False)

The first container creates a SQLite database under /state, commits one record, writes /ephemeral in its ordinary writable layer, and puts a crumb under /scratch. The three writes have distinct storage locations even though the process accesses all of them through ordinary filesystem paths. SQLite commit is used for the durable application record rather than relying on an unclosed output buffer.

After stopping and starting that same container, the database record and /ephemeral still exist, but the scratch crumb is absent. The container command does not recreate these files during startup, so the observation is about storage lifetime rather than initialization hiding the loss. This distinguishes a process restart within a retained container from destruction of its writable layer.

The original container is then removed, and a replacement mounts the same named volume. Its database still contains committed, while /ephemeral is absent. The replacement gets a new writable layer but reconnects to the retained volume. This demonstrates the volume’s independence from one container’s lifecycle, without claiming protection against losing the Docker host or deleting the volume itself.

Next, the replacement fills its bounded scratch mount until a write reports ENOSPC. It can still insert and commit after-scratch-full into the database on the volume. The successful commit demonstrates the particular capacity boundary configured here. It does not imply unlimited durable storage or protection from exhausting some other shared resource, such as total host memory.

Finally, SQLite’s backup API creates a database copy that is closed, reopened, and checked for both committed records in order. This uses the database’s copying interface instead of assuming that copying one live database filename captures every required file consistently. The copy remains on the same experimental volume, so it validates copy readability and content, not an independent disaster-recovery backup.

Make policy follow the data contract

Use a stable state directory for authoritative application data and a separately managed scratch location for reconstructable work. The Filesystem Hierarchy Standard distinguishes application state under /var/lib; container paths can differ while preserving that semantic distinction. Name the directories in application configuration and document which component owns creation, permissions, cleanup, and migration.

Set scratch budgets deliberately and handle exhaustion without deleting unrelated state. A cache is disposable only if its source and reconstruction procedure are actually available. Some apparently temporary files are required for an in-progress transaction or recovery, so cleanup needs the application’s contract rather than an indiscriminate age rule.

Keep independent backups with retention appropriate to the data, and test restoring into a fresh environment. A volume surviving replacement is persistence evidence, not backup evidence; a same-volume copy cannot survive loss of that volume. Monitor the actual backing stores, verify restore contents, and make destructive volume removal an explicit lifecycle decision.

Sources

Docker volumes, data lifecycle independent of container writable layers, checked 2026-09-12.
Docker tmpfs mounts, temporary in-memory mounts and size limits, checked 2026-09-12.
Filesystem Hierarchy Standard: /var/lib, variable state information retained for applications, checked 2026-09-12.
SQLite online backup API, consistent database copying through the database API, checked 2026-09-12.

Named config files favor clarity

Sat, 21 Nov 2026 00:00:00 +0000

Configuration that is easy to find, read, and diff is more than a preference; it is how a change stays reviewable. Naming a file with a clear convention is the first step to environment-specific, single-source config.

A file is a record of intent

A named, versioned config file is a record of intent: its committed values become explicit and reviewable like code. Project history and pull requests show what changed. To establish what actually ran, also record the deployed revision and runtime overrides; the repository alone cannot reveal them. A clear naming convention (app.conf for application knobs, app.env for environment shape) makes each file’s purpose obvious at a glance, and keeps the settings from being scattered into ad-hoc flags nobody can find.

Env and files solve different problems

Environment variables and files solve different problems. Variables are ideal for deploying the same build into many places without editing an image — you vary the value at run time per environment. Files are better for rich, structured settings a variable string would mangle. Most systems combine both: files keep the structure, the environment injects the values that differ per deploy. The rule that keeps it honest is that the source of truth stays explicit, and the boundary between structure and injectable values is drawn deliberately rather than by accident.

Secrets still move outside

Putting a config file in a repo is not a license to commit secrets into it. Anything that is a credential should be injected or referenced from a secrets store at run time, never baked into the file you commit and diff. The clean split is shape in files, secrets in a store: the committed config stays safe to show while secret storage, transport, and access controls need their own configuration. Environment injection is not encryption, and logging effective configuration can expose credentials. Report key names and winning sources while redacting sensitive values.

When it breaks

Config names drift and the loader picks the wrong file. A deploy writes app-prod.json while the program opens app.prod.json, so the expected file is missing and the loader quietly falls back to a stale development file. The service starts, talks to the wrong database, and nothing in the diff explains it, because the real mistake is the filename rather than any value. Fail closed instead: resolve one explicit path at startup, log that path, and exit with file-not-found rather than guessing a sibling.

Environment overrides fail silently in the other direction. An empty APP_PORT= or a misspelled APP-PORT either overrides a good file value with an empty string or overrides nothing at all, depending on the loader, and the running value no longer matches any committed file. Because the change leaves no diff, the team re-reads the file and concludes nothing changed. Record the winning source for every key — file, environment, or flag — print it at startup, and reject empty values instead of treating them as real input.

Secrets leak when a convenient file becomes the real store. Someone commits a working .env carrying a production credential to fix staging once, removes it in the next commit, and assumes the problem is gone. It is not: the secret lives on in history, mirrors, and backups, and every clone made in between still carries it. Committed files should hold shape with placeholders such as password = "from-store", real values should arrive only from a secrets store or the runtime environment, and a scanner should block the commit before it lands.

Show each value’s winning source. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

Save the following as config_precedence_probe.py and run python3 config_precedence_probe.py. It requires Python 3.7 or newer, uses only the standard library, and writes temporary JSON fixtures without touching real config. Run without -O, -OO, or a nonzero PYTHONOPTIMIZE setting so the assertions execute. The good sample starts from {"host": "127.0.0.1", "port": 8000, "mode": "dev"}, supplies APP_PORT=9001 in an environment-shaped dictionary, then --mode prod in an argument list, and asserts both the merged result and the per-key source map.

import argparse
import json
from pathlib import Path
from tempfile import TemporaryDirectory

KEYS = {"host", "port", "mode"}
ENV = {"APP_HOST": "host", "APP_PORT": "port", "APP_MODE": "mode"}

def validate(key, value):
 if isinstance(value, bool) or value is None or not isinstance(value, (str, int)):
 raise ValueError("invalid type: " + key)
 text = str(value).strip()
 if not text:
 raise ValueError("empty value: " + key)
 if key == "port":
 if not text.isascii() or not text.isdecimal() or not 1 <= int(text) <= 65535:
 raise ValueError("invalid port")
 return int(text)
 if key == "mode" and text not in {"dev", "prod"}:
 raise ValueError("invalid mode")
 if not isinstance(value, str):
 raise ValueError("text required: " + key)
 return text

def load(path, env, argv):
 parser = argparse.ArgumentParser()
 for key in sorted(KEYS):
 parser.add_argument("--" + key)
 cli = {k: v for k, v in vars(parser.parse_args(argv)).items() if v is not None}
 file = json.loads(Path(path).read_text())
 if not isinstance(file, dict) or set(file) - KEYS:
 raise ValueError("unknown file keys or non-object config")
 if any(k.startswith("APP_") and k not in ENV for k in env):
 raise ValueError("unknown application environment key")
 merged, sources = {}, {}
 for label, values in [("file", file), ("env", {ENV[k]: v for k, v in env.items() if k in ENV}), ("cli", cli)]:
 for key, value in values.items():
 merged[key], sources[key] = validate(key, value), label
 if set(merged) != KEYS:
 raise ValueError("missing required fields")
 return merged, sources

with TemporaryDirectory() as directory:
 path = Path(directory) / "app.prod.json"
 path.write_text(json.dumps({"host":"127.0.0.1", "port":8000, "mode":"dev"}))
 value, source = load(path, {"APP_PORT":"9001"}, ["--mode", "prod"])
 assert value == {"host":"127.0.0.1", "port":9001, "mode":"prod"}
 assert source == {"host":"file", "port":"env", "mode":"cli"}
 print("GOOD", value, "via", source) # These synthetic fields are non-secret.
 for env, argv in [({"APP_PORT":""}, []), ({"APP_PORT":"65536"}, []),
 ({"APP_PORT":True}, []), ({"APP_PORRT":"9001"}, []),
 ({}, ["--mode", ""]), ({}, ["--port", "-1"])]:
 try:
 load(path, env, argv)
 except ValueError:
 pass
 else:
 raise AssertionError("invalid config accepted")
 for contents in [{"host":"127.0.0.1"}, {"host":"x","port":None,"mode":"dev"},
 {"host":"x","port":8000,"mode":"dev","password":"placeholder"}]:
 path.write_text(json.dumps(contents))
 try:
 load(path, {}, [])
 except ValueError:
 pass
 else:
 raise AssertionError("invalid file accepted")
 try:
 load(Path(directory) / "app-prod.json", {}, [])
 except FileNotFoundError:
 pass
 else:
 raise AssertionError("wrong path accepted")
print("PASS: precedence, present-empty rejection, types, unknown keys, range and missing path")

The negative cases reject a present-but-empty override instead of falling back, a port outside 1–65535, a Boolean masquerading as an integer, an unknown APP_ key, invalid command-line values, missing fields, nulls, and unknown file keys. The deliberately wrong filename raises file-not-found. Only the successful synthetic merge is printed, followed by PASS; real secret values should never be included in equivalent diagnostics. The fixture validates this loader’s declared policy, not process-environment wiring or every configuration format. Unknown-key rejection covers APP_ names; a misspelling outside that prefix can still be ignored. Names, precedence and empty-value handling are application contracts, rather than automatic consequences of using JSON or environment variables.

Sources

12 Factor: config and environment, environment-driven configuration, checked 2026-10-15.
The Twelve-Factor App: build, release, run, separating stages, checked 2026-10-15.
man page: environment variables, how env shapes a process, checked 2026-10-15.

systemd supervises process lifecycles

Sun, 01 Nov 2026 00:00:00 +0000

A service manager turns process startup, failure, and shutdown into observable state transitions. With systemd, a unit can name the foreground command, its restart policy, and its output destination. That removes the need for a shell loop that repeatedly launches a daemon without recording why it stopped. It does not prove that the running application can answer requests or that restarting repairs the underlying fault.

Track the managed process and its invocation, bound restart attempts, and test readiness separately. An intentional stop is distinct from a crash, and a restart count is not an availability metric.

When it breaks

Choose a service type that matches the program’s startup contract. Type=simple considers startup successful early, before the executable has necessarily been invoked successfully. Type=exec waits for successful execution of the program, so a missing executable can fail the start job directly. Neither type waits for the application to finish connecting to its database or loading its state.

A program that backgrounds itself can confuse a unit expecting a foreground main process. Use the program’s foreground mode and let the manager own its lifetime, or choose the documented service type for its actual daemonization protocol. Avoid guessing a PID from a process-name search: names can collide, and wrappers can leave the wrong process designated as the service’s main process.

Restart policy also needs explicit limits. Restart=on-failure covers selected failure exits, not every possible termination signal or every unhealthy condition. A hung process that never exits will not automatically become a restart event. Combine meaningful application health observation with a bounded restart policy rather than equating an active unit with successful service delivery.

Check it quickly

Run the following Python script as an administrator in a disposable Linux environment whose PID 1 is systemd. It writes only named temporary runtime units and a temporary working directory. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. Confirm the two unit filenames are unused before executing:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,time

unit='xbcatm-supervise-lab.service';bad='xbcatm-exec-failure-lab.service'
unit_path=Path('/run/systemd/system')/unit;bad_path=unit_path.with_name(bad)
assert not unit_path.exists() and not bad_path.exists()
def run(args,success=True):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert (result.returncode==0)==success,(args,result.returncode,result.stderr)
 return result.stdout.strip()
def value(name,key):return run(['systemctl','show',name,'-p',key,'--value'])
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'state transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix='xbcatm-supervise-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py';ready=root/'ready'
 worker.write_text("""from pathlib import Path
import os,signal,sys
signal.signal(signal.SIGTERM,lambda *_: sys.exit(0))
Path(__file__).with_name('ready').write_text(str(os.getpid()))
print('worker ready',flush=True)
while True: signal.pause()
""")
 unit_path.write_text(f'''[Unit]
Description=Disposable supervision exercise
StartLimitIntervalSec=10s
StartLimitBurst=3
[Service]
Type=exec
ExecStart=/usr/bin/python3 -u {worker}
Restart=on-failure
RestartSec=200ms
TimeoutStopSec=2s
StandardOutput=journal
StandardError=journal
''')
 bad_path.write_text(f'[Service]\nType=exec\nExecStart={root}/missing-program\n')
 try:
 run(['systemd-analyze','verify',str(unit_path)])
 run(['systemctl','daemon-reload']);run(['systemctl','start',unit])
 wait_for(lambda: ready.exists() and ready.read_text()==value(unit,'MainPID'))
 first=value(unit,'MainPID');assert first!='0'
 first_invocation=value(unit,'InvocationID')
 run(['systemctl','kill','--kill-who=main','--signal=KILL',unit])
 wait_for(lambda: value(unit,'MainPID') not in ('0',first) and ready.read_text()==value(unit,'MainPID'))
 assert int(value(unit,'NRestarts'))==1
 print('SIGKILL: replacement PID confirmed; NRestarts=1')
 run(['journalctl','--sync'])
 for invocation in (first_invocation,value(unit,'InvocationID')):
 logs=run(['journalctl','_SYSTEMD_INVOCATION_ID='+invocation,'--no-pager','-o','cat'])
 assert 'worker ready' in logs
 print('journal contains both application startups')
 run(['systemctl','stop',unit]);time.sleep(0.3)
 assert value(unit,'ActiveState')=='inactive' and value(unit,'MainPID')=='0'
 print('manager stop leaves service inactive')
 run(['systemctl','daemon-reload']);run(['systemctl','start',bad],success=False)
 assert value(bad,'Result')=='exit-code' and value(bad,'ExecMainStatus')=='203'
 print('Type=exec reports missing executable: status 203/EXEC')
 finally:
 run(['systemctl','stop',unit,bad])
 for name in (unit,bad):
 if value(name,'ActiveState')=='failed':
 run(['systemctl','reset-failed',name])
 unit_path.unlink();bad_path.unlink(missing_ok=True)
 run(['systemctl','daemon-reload'])

The worker stays in the foreground and publishes its own startup marker. The script waits until that marker agrees with the unit’s main PID, then sends SIGKILL specifically to the managed main process. A different PID and one automatic restart confirm that the manager actually replaced the terminated process.

Journal checks use the two invocation identifiers, so older entries from an earlier run cannot satisfy the assertion. Each invocation must contain the worker’s startup message. This checks both lifecycle replacement and attribution of the output produced by the two actual workers, rather than simply finding a familiar phrase somewhere in the unit’s accumulated history.

An explicit manager stop leaves the unit inactive with no main PID after the restart delay has passed. This is an intentional stop, not another crash to undo. The separate failing unit uses a nonexistent executable; its start command fails and records status 203/EXEC. That failure concerns process execution, not a database connection or application-level health probe.

The script verifies the valid unit definition before starting it and removes both runtime unit files afterward. Syntax verification and successful execution are separate evidence: the first checks the manager’s interpretation of configuration, while the second observes the behavior of a real process. Neither substitutes for checking a production endpoint.

Make the lifecycle operationally useful

Keep the installed unit definition, runtime configuration, and application readiness signal consistent. Ordering another unit after this service orders manager startup completion; if a dependent application needs a stronger readiness promise, use a supported notification protocol or an explicit application-level check. Retrying transient dependencies may still be part of the application’s own startup behavior.

Use systemctl show for specific state properties and the journal for the corresponding invocation. A unit’s current PID can change between observations, so identify the instance when diagnosing repeated failures. Track both restart frequency and whether useful work resumes; a successful restart counter is not an availability metric by itself.

Runtime files under /run/systemd/system are suitable for this disposable exercise. A deployed service also needs an intentional installation and enablement policy, appropriate user permissions, and a tested shutdown path. Changing a unit definition requires reloading the manager’s configuration; starting a service once does not automatically arrange for it to start at the next boot.

Sources

systemd 252 service units, startup types and restart semantics, checked 2026-09-12.
systemd 252 execution environment, output routing and execution exit codes, checked 2026-09-12.
systemd 252 unit configuration, runtime unit paths and start-rate limits, checked 2026-09-12.
systemd 252 journalctl, querying recorded fields and synchronizing journal writes, checked 2026-09-12.

Scheduled jobs need to be safe to rerun

Sat, 15 Aug 2026 00:00:00 +0000

A scheduler starts attempts, while the business usually cares about completing a logical task. One daily credit may be attempted several times after an error, a restart, or an operator retry. Giving every attempt a new identifier makes those duplicates indistinguishable from new work. Use a key that identifies the intended task, such as its name and nominal scheduled time, and define what a repeated key means before changing business state.

Scheduling and execution are separate

Do not assume every scheduler catches up missed work. Ordinary cron behavior varies by implementation and does not provide a universal catch-up guarantee. A systemd calendar timer with Persistent=true can trigger after an inactive interval when at least one activation was missed; that does not mean replaying every missed occurrence. Kubernetes documents that a CronJob may create duplicate Jobs or miss an occurrence under some circumstances.

Choose the required business rule explicitly: skip stale work, process each missing period, or reconcile the current state. Use the nominal logical period in the deduplication key, with an unambiguous time zone. An actual start timestamp changes across retries and therefore cannot identify the same intended occurrence. Keep an attempt identifier separately for diagnostics without confusing it with business identity.

When it breaks

A check followed by an unrelated update has a race. Two workers can both observe that a key is absent and both apply the effect. A process lock may prevent overlap on one machine but does not make a business update atomic with a marker file. If the process fails after the update and before writing the marker, a retry repeats the effect. Writing the marker first can instead lose the work.

When both state and completion record live in one transactional database, commit them together under a uniqueness constraint. A failure before commit should preserve neither change; a retry after commit should see the completed logical key. In SQLite, BEGIN IMMEDIATE acquires the write transaction early, and competing writers may wait or fail with a busy error. A timeout bounds that wait; it does not make every workload contention-free.

This boundary does not cover an external email, payment, or HTTP call. A local rollback cannot undo an effect already accepted elsewhere. Such workflows need an appropriate remote idempotency contract or a durable delivery design with deduplication at the receiver. Reusing a key with a different payload also requires a defined rejection policy; the example below always applies the same fixed credit.

The fixed credit and unique completion key commit within one SQLite transaction. The historical eight-worker fixture passed at this local boundary; scheduler delivery, process-kill recovery and external exactly-once effects remain unverified. No example was rerun for this drawing.

Check it quickly

Save the code as idempotent-jobs.py and run python3 idempotent-jobs.py. The earlier source records Python 3.12.14 and SQLite 3.53.1; the separate dated rerun below records this review. Python must include sqlite3 linked to SQLite 3.24.0 or later for UPSERT, with support for separate connections in multiple threads. It uses a temporary SQLite database and eight worker threads, each with its own database connection. The business effect adds ten units to a balance. A non-null primary key records which logical daily credit has completed within the same transaction.

from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from tempfile import TemporaryDirectory
from threading import Barrier
import json
import sqlite3

with TemporaryDirectory() as directory:
 database = str(Path(directory) / 'jobs.db')
 with sqlite3.connect(database) as connection:
 connection.executescript('''
CREATE TABLE balance (id INTEGER PRIMARY KEY, units INTEGER NOT NULL);
INSERT INTO balance VALUES (1, 0);
CREATE TABLE completed (logical_key TEXT PRIMARY KEY NOT NULL);
''')

 def run(key, fail=False, gate=None):
 connection = sqlite3.connect(database, timeout=10, isolation_level=None)
 try:
 if gate:
 gate.wait(timeout=5)
 connection.execute('BEGIN IMMEDIATE')
 inserted = connection.execute(
 'INSERT INTO completed VALUES (?) ON CONFLICT(logical_key) DO NOTHING', (key,))
 if not inserted.rowcount:
 connection.execute('COMMIT')
 return 'duplicate'
 connection.execute('UPDATE balance SET units = units + 10 WHERE id = 1')
 if fail:
 raise RuntimeError('injected failure before commit')
 connection.execute('COMMIT')
 return 'applied'
 except Exception:
 if connection.in_transaction:
 connection.execute('ROLLBACK')
 raise
 finally:
 connection.close()

 key = 'daily-credit:2026-09-13T00:00:00Z'
 try:
 run(key, fail=True)
 except RuntimeError as error:
 assert str(error) == 'injected failure before commit'
 with sqlite3.connect(database) as connection:
 assert connection.execute('SELECT units FROM balance').fetchone() == (0,)
 assert connection.execute('SELECT count(*) FROM completed').fetchone() == (0,)
 gate = Barrier(8)
 with ThreadPoolExecutor(max_workers=8) as pool:
 results = list(pool.map(lambda _: run(key, gate=gate), range(8)))
 assert results.count('applied') == 1 and results.count('duplicate') == 7
 assert run(key) == 'duplicate'
 assert run('daily-credit:2026-09-14T00:00:00Z') == 'applied'
 with sqlite3.connect(database) as connection:
 units = connection.execute('SELECT units FROM balance').fetchone()[0]
 completed = connection.execute('SELECT count(*) FROM completed').fetchone()[0]
 assert (units, completed) == (20, 2)
 print(json.dumps({'rolled_back_attempt_left_no_state': True,
 'concurrent_applied': results.count('applied'),
 'concurrent_duplicates': results.count('duplicate'),
 'final_units': units, 'completed_logical_jobs': completed}))

The first attempt intentionally raises after both inserts and updates but before commit. The script confirms that rollback leaves zero units and no completed key. Eight workers then start together for the same logical occurrence. Exactly one reports applied; seven report duplicate. Their unique-key decision and balance mutation are serialized by the actual database, not a mocked lock.

A further replay after successful commit remains a duplicate. A different nominal date is a new logical task and applies another ten units. The final result is twenty units and two completed keys. This verifies rollback and replay within one SQLite database, not scheduler delivery guarantees, process-kill recovery, or external exactly-once execution.

A separate 2026-10-19 rerun passed the unchanged fixture with Python 3.12.14 and SQLite 3.53.1 on Linux 6.18.55. It confirms the same local transaction assertions, without executing any scheduler or testing process-kill recovery.

Keep failure and completion visible

Record the logical key, attempt outcome, duration, and last successful business period. Distinguish a harmless duplicate from a database error; catching every exception and labeling it completed would hide lost work. Define how long completion records must remain available before deleting them, since a retry after deletion can apply again. Operational evidence should tell an operator whether the intended period completed, rather than merely whether a timer launched a process.

Sources

SQLite: transactions, writer serialization, commit, and rollback, checked 2026-10-15.
SQLite: UPSERT, targeted handling of a uniqueness conflict, checked 2026-10-15.
systemd.timer, the scope of persistent calendar catch-up, checked 2026-10-15.
Kubernetes: CronJob, approximate scheduling and duplicate or missed jobs, checked 2026-10-15.

Least privilege is a threshold, not a feeling

Tue, 11 Aug 2026 00:00:00 +0000

Least privilege means granting a principal the operations required for a defined job, on the intended resources, for an appropriate lifetime. It is a testable policy rather than a reassuring label. Start with the service’s real launch identity and dependencies; an unused permission and a missing permission are different findings that need different responses.

Define the minimum in writing

Record the files the service reads, the directories it writes, its required network destinations, and its administrative operations. Include startup, logging, rotation, recovery, and shutdown. A service may legitimately need temporary storage and shared libraries even when its business data belongs in one directory. A useful inventory explains each access rather than declaring everything outside the data directory unnecessary.

For systemd units, User= and Group= choose identity, while filesystem permissions and sandboxing impose additional boundaries. ProtectSystem=strict makes most of the filesystem hierarchy read-only within the unit, with documented API-filesystem exceptions and possible writable exceptions. Read-only does not mean unreadable, and does not prevent communication through a reachable Unix-domain socket. Protecting confidential content requires separate access controls or visibility restrictions, such as an appropriate ProtectHome= policy.

Test the specific boundary

Different directives solve different problems. PrivateDevices= restricts device exposure, NoNewPrivileges= prevents gaining new privileges through execution, and RestrictAddressFamilies= restricts socket families. The last option is not a destination or port firewall. Kernel support, service-manager context, existing descriptors, and other settings affect what is enforced; a list of directive names is not proof of a complete sandbox.

A capability can be narrower than unrestricted root, but capabilities are not necessarily tiny permissions. Some cover broad administrative powers. Grant one only after checking its documented operations and whether a less powerful mechanism, such as file ownership, an ACL, or a separately constrained helper, fits the need. Test an intended operation and a forbidden neighboring operation under the actual service context.

Why broad access persists

Temporary incident grants commonly outlive the incident. Record an owner, reason, and expiry or review point so recovery access does not silently become routine authority. Compare the deployed bindings with the approved policy after changes. A configuration file in source control does not prove that the running workload uses it.

Credential lifetime is a separate control. A narrowly scoped token can still be misused while valid, and rotating a broadly scoped credential does not reduce its permissions. Prefer appropriately bounded credentials and avoid copying them into logs or runbooks. Review both what the credential permits and how it is revoked when the task or relationship ends.

When it breaks

In Kubernetes, wildcard verbs and resources grant broad access within the rule’s API groups and the binding’s scope. They do not automatically mean cluster administrator across every API group and namespace. Still, new matching resources or verbs can inherit permission without another policy edit. Check effective subjects, namespaces, resource names, and all additive grants before concluding that one narrow Role limits total access.

Observed non-use is evidence for review, not automatic permission removal. A recovery action may be essential but absent from an ordinary week. Conversely, an action used by one role does not justify granting it to every role. Keep usage keyed by identity and role, and obtain the operational reason for exceptions rather than treating a global used-action list as authorization.

Over-tightening can block legitimate deployments and encourage unsafe workarounds. Supply a narrow supported path and test it before removing access. Impersonation is itself permission-controlled; adding an impersonation flag does not magically create administrator rights. Keep emergency procedures accountable rather than relying on hidden shared credentials.

Synthetic grant membership and independent usage observations answer different questions. Unused access invites review, not automatic removal.

Check it quickly

Save this as privilege_audit.py and run python3 privilege_audit.py. Keep assertions enabled, without -O or PYTHONOPTIMIZE. The standard-library model replays seven synthetic requests and separately compares each role with its own synthetic review-window usage. It does not read credentials or change permissions.

ROLES = {
 "api-reader": {"articles:read", "health:read"},
 "deployer": {"articles:read", "articles:publish", "health:read"},
 "ops-admin": {"articles:read", "articles:publish", "secrets:read", "users:write"},
}

# (role, action, should_allow)
REQUESTS = [
 ("api-reader", "articles:read", True),
 ("api-reader", "articles:publish", False),
 ("api-reader", "secrets:read", False),
 ("deployer", "articles:publish", True),
 ("deployer", "secrets:read", False),
 ("ops-admin", "secrets:read", True),
 ("ops-admin", "users:write", True),
]

# Synthetic observations from an independent review window, per role.
USED = {
 "api-reader": {"articles:read", "health:read"},
 "deployer": {"articles:read", "articles:publish", "health:read"},
 "ops-admin": {"articles:read", "articles:publish"},
}


def allowed(role, action):
 return action in ROLES.get(role, set())


for role, action, expected in REQUESTS:
 decision = allowed(role, action)
 assert decision is expected, (role, action, decision)
 print(f"{role:10} {action:17} -> {'allow' if decision else 'deny'}")

for role, granted in sorted(ROLES.items()):
 unused = sorted(granted - USED[role])
 print(f"{role:10} granted={len(granted)} unused={len(unused)} {unused}")
 if role == "ops-admin":
 assert unused == ["secrets:read", "users:write"], unused

assert not allowed("unknown-role", "articles:read")
assert not allowed("api-reader", "unknown:action")
print("PASS: 7 requests match the matrix; ops-admin holds 2 unused grants")

Expect seven decision lines, three usage lines, and the final PASS line. The reader cannot publish or read secrets; the deployer cannot read secrets. The administrator has two grants absent from its review-window set. Unknown roles and actions are denied. The request fixtures and usage fixtures represent different inputs, so a permitted test request does not become a real usage observation.

Retained exact-fence records report both language examples passing with Python 3.12.14 on Linux; this source review did not rerun them. These checks establish set-based decision behavior only. They do not exercise systemd confinement, Kubernetes authorization, admission policies, or token expiry. Production verification must test those actual enforcement points.

Sources

systemd.exec man page, sandboxing options, the unit-level confinement options, checked 2026-08-12.
man page for capabilities, dividing root into fine-grained capabilities, checked 2026-08-12.
Open Web Application Security Project, least privilege, the developer-focused control, checked 2026-08-12.
Kubernetes RBAC: rule and binding scope, checks additive rules, API groups, and binding scope, checked 2026-10-15.
systemd 252 upstream execution manual source, checks filesystem and address-family sandbox boundaries in upstream version 252, checked 2026-10-15.

Unix · XB Field Notes

Fork creates a process; exec replaces its program

When it breaks

Check it quickly

Make launch ownership explicit

Sources

Setuid changes effective identity at exec

Distinguish the file bit from process credentials

Execution can suppress the transition

Privilege dropping needs a complete design

When it breaks

Check it quickly

Sources

The sticky bit restricts removal in a shared directory

When it breaks

Check it quickly

Design shared storage deliberately

Sources

Environment variables are a per-process thing

Export selects values for another program

A subshell is not a freshly executed shell

Per-command settings have a defined lifetime

When it breaks

Check it quickly

Sources

A bare command the shell cannot find

Know which lookup layer answered

A slash selects a path directly

When it breaks

Check it quickly

Sources

Graceful shutdown stops work, not life

A signal begins coordination

The deadline covers a lifecycle

Coordinate with the supervisor

When it breaks

Check it quickly

Further reading

Sources

Give durable state and scratch space different lifecycles

When it breaks

Check it quickly

Make policy follow the data contract

Further reading

Sources

Named config files favor clarity

A file is a record of intent

Env and files solve different problems

Secrets still move outside

When it breaks

Check it quickly

Further reading

Sources

systemd supervises process lifecycles

When it breaks

Check it quickly

Make the lifecycle operationally useful

Further reading

Sources

Scheduled jobs need to be safe to rerun

Scheduling and execution are separate

When it breaks

Check it quickly

Keep failure and completion visible

Further reading

Sources

Least privilege is a threshold, not a feeling

Define the minimum in writing

Test the specific boundary

Why broad access persists

When it breaks

Check it quickly

Further reading

Sources