Automation · XB Field Notes

Cron repeats on the calendar

Thu, 30 Sep 2027 00:00:00 +0000

Cron launches commands at matching calendar times without inherently waiting for earlier invocations. Seventy-second work scheduled every minute can therefore overlap. Treat triggering, concurrency, and successful completion as separate responsibilities.

The illustrative lanes show the next calendar trigger arriving before a seventy-second invocation ends. A shared nonblocking lock makes the later invocation record a skip; it does not provide exactly-once effects. The source’s prior measurements were not rerun.

Read fields as calendar selections

A user crontab entry contains minute, hour, day of month, month, day of week, then command. Asterisks select all field values. A minute step such as */15 selects values within each hour, not fifteen minutes since completion. Steps that unevenly divide a field’s range can produce unequal boundary gaps.

In Debian/Vixie format, when day of month and day of week are both restricted, either matching day permits execution; minute, hour, and month constraints still apply. Month and weekday are not alternatives. Verify representative dates before installing the expression.

Timezone, daylight-saving changes, and downtime affect scheduling. Implementations handle skipped or repeated times differently; Debian documents special handling for certain small clock changes and fixed-time jobs. This experiment neither changes the clock nor tests transitions. Calendar triggers are not durable queues guaranteeing replay of missed runs.

When it breaks

Overlapping jobs can duplicate side effects, compete for resources, or overwrite files. Explicitly enforce any single-invocation policy: nonblocking advisory locks permit recorded skips; blocking locks can leave waiting processes. Neither provides exactly-once business effects or repairs partial operations.

A local file lock only coordinates participants using the same relevant locking facility. It does not automatically serialize jobs on unrelated hosts, and network filesystem semantics require separate examination. Keep the lock held for the work’s lifetime, and make skipped work visible. Silently dropping every later trigger while one worker hangs can look like a healthy schedule unless completion age is monitored.

Cron is not universally log-free. Debian logs job starts to syslog by default and offers additional logging controls. Command output may be mailed when a working mail path exists, but a minimal container often lacks that path. Record useful start, completion, failure, and skip outcomes through an intentional destination. A start message proves dispatch, not completion of the application’s task.

Check it quickly

Repair draft checked 2026-10-19: this revision was checked statically and with offline command/file-state doubles only. No cron, crontab, account, or signal operation was executed. Technical and operational verification remain blocked; the measurements below belong to the original fixture, not this revision.

Only a separately authorized, freshly provisioned disposable Debian 12 lab with Python 3.11+, cron 3.0pl1-162 and procps is suitable for a future run. It must have no host mounts, existing root table, cron daemon, enabled unrelated schedules in /etc/crontab, /etc/cron.d or any user’s spool (including periodic-directory dispatch), or concurrent administrators. Root is a privilege prerequisite, not isolation proof. Save as cron-overlap.py; the future command is python3 cron-overlap.py --isolated-disposable-lab. That flag acknowledges an externally verified prerequisite and cannot establish containment. Allow roughly two minutes for real calendar ticks, and discard the entire lab afterward.

The test table disables mail with MAILTO. Setup is inside the cleanup boundary, including a failed table installation or daemon startup. A failed listing is accepted as absence only for the pinned package’s exact English absent-table diagnostic; permission and command errors abort. Workers publish complete per-event JSON files by same-directory replacement. Cleanup revokes a positive run permit before removing schedules and stopping the owned daemon, then waits at most five seconds for the workers’ lifetime lock and completion records. Each cleanup step reports its own error without skipping later steps; private-file removal is skipped after earlier cleanup errors; deletion itself can fail partway. This checks admitted work, not the exit of every cron wrapper or late-start process. It is not a containment or crash-durability proof.

from pathlib import Path
from tempfile import mkdtemp
import fcntl
import json
import os
import shutil
import subprocess
import sys
import time


def command(arguments, **options):
 return subprocess.run(arguments, capture_output=True, text=True, timeout=5,
 env={**os.environ, 'LC_ALL': 'C'}, **options)


def table_absent(result):
 # Debian cron 3.0pl1-162: other listing failures must abort.
 return (result.returncode == 1 and result.stdout == ''
 and result.stderr == 'no crontab for root\n')


def remove_table():
 current = command(['crontab', '-l'])
 if table_absent(current):
 return
 current.check_returncode()
 command(['crontab', '-r']).check_returncode()
 if not table_absent(command(['crontab', '-l'])):
 raise RuntimeError('could not verify table removal')


def attempt(errors, name, operation):
 try:
 operation()
 except Exception as error:
 errors.append(f'{name}: {error!r}')


def stop_daemon(daemon, errors):
 if daemon is None:
 return
 attempt(errors, 'terminate cron', daemon.terminate)
 try:
 daemon.wait(timeout=5)
 except subprocess.TimeoutExpired:
 attempt(errors, 'kill owned cron', daemon.kill)
 attempt(errors, 'reap cron', lambda: daemon.wait(timeout=5))
 except Exception as error:
 errors.append(f'wait cron: {error!r}')


def read_events(root):
 return sorted((json.loads(path.read_text()) for path in root.glob('*.json')),
 key=lambda event: event['monotonic'])


def wait_workers(root):
 # Workers hold a shared lifetime lock before checking the run permit or publishing.
 with (root / 'lifetime.lock').open() as gate:
 deadline = time.monotonic() + 5
 while True:
 try:
 fcntl.flock(gate, fcntl.LOCK_EX | fcntl.LOCK_NB)
 break
 except BlockingIOError:
 if time.monotonic() >= deadline:
 raise TimeoutError('workers did not release lifetime lock')
 time.sleep(0.1)
 events = read_events(root)
 started = {e['pid'] for e in events if e['event'] == 'start'}
 ended = {e['pid'] for e in events if e['event'] == 'end'}
 if not started <= ended:
 raise RuntimeError('a started worker has no completion record')


worker_source = r'''from pathlib import Path
import fcntl,json,os,sys,time
root=Path(__file__).parent
kind=sys.argv[1]
def record(event):
 destination=root/f'{os.getpid()}-{event}.json'
 temporary=destination.with_suffix('.tmp')
 temporary.write_text(json.dumps({'event':event,'kind':kind,
 'pid':os.getpid(),'monotonic':time.monotonic()}))
 temporary.replace(destination)
def main():
 with (root/'lifetime.lock').open() as gate:
 fcntl.flock(gate,fcntl.LOCK_SH)
 if not (root/'run-permit').exists(): return
 lock=None
 try:
 if kind=='locked':
 lock=open(root/'job.lock','a')
 try: fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
 except BlockingIOError:
 record('skip')
 return
 record('start')
 try:
 until=time.monotonic()+70
 while time.monotonic()<until and (root/'run-permit').exists():
 time.sleep(0.1)
 finally: record('end')
 finally:
 if lock is not None: lock.close()
main()
'''


def main():
 # This flag is an operator prerequisite, NOT evidence of containment.
 if sys.argv[1:] != ['--isolated-disposable-lab']:
 raise SystemExit('requires a separately verified disposable Debian 12 lab')
 if os.geteuid() != 0:
 raise RuntimeError('root is required, but does not prove isolation')
 probe = command(['pgrep', '-x', 'cron'])
 if probe.returncode != 1 or probe.stdout or probe.stderr:
 raise RuntimeError(f'cron absence not established: {probe!r}')
 if not table_absent(command(['crontab', '-l'])):
 raise RuntimeError('expected absent root table; existing table or listing error')
 root = Path(mkdtemp(prefix='cron-overlap-', dir='/tmp'))
 daemon = None
 table_attempted = False
 try:
 (root / 'lifetime.lock').touch()
 (root / 'run-permit').touch()
 worker = root / 'worker.py'
 worker.write_text(worker_source)
 table = 'MAILTO=""\n' + ''.join(f'* * * * * /usr/bin/python3 {worker} {kind} >>{root}/worker.log 2>&1\n'
 for kind in ('plain', 'locked'))
 table_attempted = True
 command(['crontab', '-'], input=table).check_returncode()
 daemon = subprocess.Popen(['cron', '-f'], stdout=subprocess.DEVNULL,
 stderr=subprocess.DEVNULL)
 deadline = time.monotonic() + 140
 while time.monotonic() < deadline:
 if daemon.poll() is not None:
 raise RuntimeError('cron exited')
 events = read_events(root)
 plain = [e for e in events if (e['kind'], e['event']) == ('plain', 'start')]
 locked = [e for e in events if (e['kind'], e['event']) == ('locked', 'start')]
 skipped = [e for e in events if (e['kind'], e['event']) == ('locked', 'skip')]
 if len(plain) >= 2 and skipped:
 break
 time.sleep(0.2)
 else:
 raise AssertionError('two real cron ticks were not observed')
 assert len(plain) == 2 and len(locked) == 1 and len(skipped) == 1
 assert plain[1]['monotonic'] - plain[0]['monotonic'] < 70
 assert not any(e['event'] == 'end' and e['pid'] == plain[0]['pid'] for e in events)
 print(json.dumps({'actual_cron_starts': len(plain), 'overlap_observed': True,
 'seconds_between_starts': round(plain[1]['monotonic']-plain[0]['monotonic'], 3),
 'locked_starts': len(locked), 'locked_skips': len(skipped)}), flush=True)
 finally:
 original = sys.exc_info()[1]
 errors = []
 attempt(errors, 'revoke run permit', lambda: (root / 'run-permit').unlink(missing_ok=True))
 if table_attempted:
 attempt(errors, 'remove root table', remove_table)
 stop_daemon(daemon, errors)
 attempt(errors, 'wait for workers', lambda: wait_workers(root))
 # Retain private files after failed cleanup; discard the whole lab.
 if not errors:
 attempt(errors, 'remove private files', lambda: shutil.rmtree(root))
 if errors:
 message = 'cleanup incomplete; discard lab: ' + '; '.join(errors)
 if original is not None:
 original.add_note(message)
 else:
 raise RuntimeError(message)


if __name__ == '__main__':
 main()

Two every-minute entries run workers that would last seventy seconds. The plain worker has no job-exclusion lock; both branches share only the cleanup lifetime lock. The locked worker uses a shared file and a nonblocking exclusive flock, recording a skip when acquisition fails. Each event includes the worker PID and a monotonic timestamp, so distinct invocations and their overlap can be checked directly.

The original fixture’s historical Debian cron 3.0pl1-162 run recorded two plain starts, 60.003 seconds apart, before its first invocation ended. The guarded worker started once and recorded one skip. That original run reported all assertions passing and cleanup via its stop file. This revision instead revokes a positive permit and has not reproduced those runtime results. The experiment verifies local overlap and explicit exclusion; it does not implement durable retries, distributed locking, recovery after reboot, or exactly-once effects. Select those policies from the job’s actual business contract instead of assuming the calendar expression supplies them.

Sources

Debian 12: crontab(5), calendar fields, steps and the two day fields, checked 2026-09-13.
Debian 12: cron(8), daemon execution, logging and clock-change behavior, checked 2026-09-13.
Linux: flock(2), advisory locks and nonblocking acquisition, checked 2026-09-13.
Python 3.11 subprocess, timeout and child cleanup semantics, checked 2026-10-19.
Debian cron 3.0pl1-162 crontab.c, distinguish absent-table diagnostics from other failures, checked 2026-10-19.
Python 3.11 os.replace, atomic same-filesystem publication, checked 2026-10-19.

GitHub Actions workflows gate the build

Thu, 02 Sep 2027 00:00:00 +0000

A workflow can produce evidence for a merge decision, but YAML alone does not make that evidence mandatory. A dependable build gate connects the intended event, the exact revision being tested, meaningful test results, and a repository rule requiring the correct check. Review those connections together. A green badge on some earlier run is not proof that the proposed merge has satisfied today’s requirements.

The aggregate decision is only one part of a merge gate: repository rules must require the correct check for the intended event and revision. The retained passed fixture covers the local success predicate only. No example was rerun for this drawing.

Separate execution from enforcement

Files under .github/workflows/ describe triggers and jobs. The on key chooses events; branch protection or rulesets separately decide which checks must pass before merging. Record the stable check name and review the rule whenever it changes. Distinct workflows should avoid ambiguous job names, and the expected check source should match the trusted application.

Check the revision shown in the pull request’s checks box. A pull_request run commonly tests a synthetic merge commit, not simply the contributor’s head commit. Merge queues need the separate merge_group trigger for their candidate combinations. A successful local build or manually dispatched workflow cannot automatically substitute for the required event and revision.

Make the dependency result explicit

Jobs without dependencies can run in parallel, while steps within a job normally run sequentially. needs establishes prerequisites; conditions determine whether work executes after failure or skipping. A workflow containing only one required test job does not need an artificial dependency or event condition. Requiring those keys everywhere is not a useful correctness test.

For an aggregate gate, wait for every mandatory job and inspect its result explicitly. Use if: ${{ always() }} so a failed prerequisite does not simply skip the gate, then require actual success. If optional work may be skipped, encode that exception deliberately. Do not let a generic success message, or continue-on-error on mandatory tests, turn missing evidence into a passing gate.

The needs context contains direct dependencies, not every job earlier in the dependency graph. If the gate must inspect both a build and a test that depends on that build, declare both jobs directly in the gate’s needs. Otherwise the build’s result is not exposed there merely because it is a transitive prerequisite.

Permissions describe the token, not all credentials

permissions controls GITHUB_TOKEN; it does not constrain an unrelated secret credential passed to a command. A restrictive workflow-level declaration can be inherited by jobs, so repeating it in every job is unnecessary. Grant additional access only where required. The offline exercise below uses no token because it neither checks out private source nor calls GitHub APIs.

Ordinary fork pull requests normally receive a read-only token and no repository secrets, subject to repository settings. Secrets are not themselves read-only tokens. Never move untrusted pull-request code into a privileged pull_request_target execution just to obtain secrets. Log masking is a defensive aid, not permission to print credentials or a guarantee that transformed values cannot leak.

When it breaks

A workflow omitted by path or branch filters can leave its required check pending. A conditionally skipped job, however, can satisfy a required-check rule without running tests. These are different outcomes. Rename checks carefully, and verify the failing, skipped, and cancelled paths as well as the successful case on the hosting service.

Caches require compatible keys and verified dependency installation. Ordinary pull-request caches belong to their merge reference; matching keys alone do not make them restorable by the base branch. Avoid unsupported blanket claims that every fork cache can poison main. Also separate test cancellation from deployment recovery: cancelling an in-progress job does not undo external changes it already made.

Check it quickly

The exercise isolates one aggregate gate decision: accept a prerequisite only when its result is exactly success. In a real workflow, the result would come from needs.test.result, with the prerequisite declared in needs. Here the inputs are deliberately synthetic. It is a local decision exercise, not a workflow validator or a replacement for the project’s real tests.

Save this standard-library probe as gate_check.py and run python3 gate_check.py with Python 3 and Bash available. It runs the shell predicate with synthetic results and includes a deliberately failing test assertion. It writes no files and contacts no service.

import os
import subprocess
import sys

subprocess.run([sys.executable, "-c", "assert 2 + 2 == 4"], check=True)
negative = subprocess.run([sys.executable, "-c", "assert 2 + 2 == 5"],
 capture_output=True, text=True)
assert negative.returncode != 0 and "AssertionError" in negative.stderr
print("test-negative-control: rejected")
command = 'test "$TEST_RESULT" = success'
for result in ("success", "failure", "cancelled", "skipped", "", "unexpected"):
 run = subprocess.run(["bash", "--noprofile", "--norc", "-c", command],
 env={"PATH": os.environ["PATH"], "TEST_RESULT": result},
 capture_output=True, text=True)
 accepted = run.returncode == 0
 assert accepted == (result == "success"), (result, run)
 print(f"{result or '<missing>'}: {'accept' if accepted else 'reject'}")
print("PASS: local predicate only; GitHub enforcement not exercised")

Expect the negative control to be rejected, only success to be accepted, and the final scope reminder. This verifies the local command and its rejection cases. It does not parse arbitrary YAML, evaluate GitHub expressions, schedule jobs, inspect repository rules, or prove merge blocking. Those remain hosted integration checks against the actual repository configuration.

Sources

GitHub: workflow syntax, dependencies, conditions and token-permission inheritance, checked 2026-10-15.
GitHub: understanding Actions, workflows, events, jobs and steps, checked 2026-10-15.
GitHub: troubleshooting required status checks, revision identity, skipped checks and merge queues, checked 2026-10-15.
GitHub: workflow events, fork pull requests and privileged trigger boundaries, checked 2026-10-15.
GitHub: dependency caching, branch and pull-request cache scopes, checked 2026-10-15.
GitHub: using secrets, secret availability and log handling, checked 2026-10-15.
GitHub: needs context, direct dependencies and job-result values, checked 2026-10-18.

A systemd timer schedules a service, not its success

Mon, 21 Jun 2027 00:00:00 +0000

A systemd timer separates the scheduling rule from the service that performs the work. The timer decides when to request activation; the service supplies the command, execution identity, environment, timeout, and result. This separation makes a maintenance job inspectable using normal unit tools. It does not turn a scheduled attempt into proof that a backup, cleanup, or report completed correctly.

Timer state does not prove service success. The source fixture’s completion-relative ordering is shown without a measured time scale or exact launch-time promise. Stopping future scheduling and stopping current work are separate decisions; the historical fixture was not rerun.

When it breaks

An active timer can coexist with a failed service. Monitoring only the timer’s active state therefore misses failures in the actual work. Likewise, a recent trigger timestamp identifies scheduling activity, not successful business output. Check the service’s result and the artifact or state change the job was supposed to produce, with enough context to distinguish successive activations.

A timer normally activates the service with the matching basename, unless Unit selects another target. If that service is already active, the timer does not launch another independent copy. This prevents overlapping activations of that one unit, but it is not a distributed lock or protection against another service running the same command. A oneshot kept active by RemainAfterExit=yes can consequently prevent later scheduled executions.

Stopping a timer stops future scheduling. It does not automatically stop a service invocation already in progress. That distinction matters when pausing maintenance during an incident: inspect the target service and decide deliberately whether current work should finish or be interrupted. Interrupting a command also does not roll back its earlier filesystem or database changes.

Check it quickly

Run this Python script as an administrator in a disposable Linux system with systemd as PID 1. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It creates a unique timer/service pair and a temporary worker. The worker records real process entries and completion times; explicit release files hold successful jobs open so the scheduling behavior is observable:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,time,uuid

name='xbcatm-timer-'+uuid.uuid4().hex[:8];service=name+'.service';timer=name+'.timer'
paths={unit:Path('/run/systemd/system')/unit for unit in (service,timer)}
assert all(not p.exists() for p in paths.values())
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def value(unit,key):return run('systemctl','show',unit,'-p',key,'--value')
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py'
 worker.write_text(r'''from pathlib import Path
import json,sys,time
root=Path(__file__).parent;number=len(list(root.glob('start-*.json')))+1
def record(kind):
 temp=root/'record.tmp';temp.write_text(json.dumps(time.monotonic()))
 temp.replace(root/(kind+'-'+str(number)+'.json'))
record('start')
if (root/'fail').exists():sys.exit(7)
while not (root/('release-'+str(number))).exists():time.sleep(0.01)
record('end')
''')
 paths[service].write_text(f'[Service]\nType=oneshot\nTimeoutStartSec=10s\nExecStart=/usr/bin/python3 {worker}\n')
 definition='[Timer]\nOnActiveSec=100ms\nOnUnitInactiveSec=10s\nAccuracySec=1ms\n'
 paths[timer].write_text(definition);(root/'fail').touch()
 try:
 run('systemd-analyze','verify',*[str(p) for p in paths.values()])
 run('systemctl','daemon-reload');run('systemctl','start',timer)
 wait_for(lambda:value(service,'ActiveState')=='failed')
 assert value(service,'ExecMainStatus')=='7' and value(timer,'ActiveState')=='active'
 run('systemctl','stop',timer)
 print('timer active while its first job failed with exit status 7')
 (root/'fail').unlink();run('systemctl','reset-failed',service)
 paths[timer].write_text(definition.replace('OnUnitInactiveSec=10s','OnUnitInactiveSec=250ms'))
 run('systemctl','daemon-reload');run('systemctl','start',timer)
 wait_for(lambda:(root/'start-2.json').exists())
 time.sleep(0.65)
 assert len(list(root.glob('start-*.json')))==2
 (root/'release-2').touch();wait_for(lambda:(root/'start-3.json').exists())
 gap=json.loads((root/'start-3.json').read_text())-json.loads((root/'end-2.json').read_text())
 assert gap>=0.25,gap
 print('no overlapping launch; next job starts at least 250ms after prior deactivation')
 pid=value(service,'MainPID');assert pid!='0'
 run('systemctl','stop',timer)
 assert value(service,'MainPID')==pid and value(service,'ActiveState')=='activating'
 print('stopping timer leaves its already running oneshot alive')
 (root/'release-3').touch();wait_for(lambda:value(service,'ActiveState')=='inactive')
 time.sleep(0.4)
 assert len(list(root.glob('start-*.json')))==3 and value(service,'Result')=='success'
 print('released job completes; stopped timer schedules no further run')
 finally:
 run('systemctl','stop',timer,service)
 for unit,path in paths.items():
 if value(unit,'ActiveState')=='failed':run('systemctl','reset-failed',unit)
 path.unlink()
 run('systemctl','daemon-reload')

The first timer activation launches a worker that deliberately exits with status seven. The service becomes failed while the timer remains active. Its next completion-relative interval is initially ten seconds, allowing the script to inspect this state and stop the timer before another attempt. The output confirms a scheduler can be functioning while the scheduled work is failing.

After correcting the fixture’s failure condition, the script clears the failed state, changes the interval to 250 milliseconds, and starts scheduling again. The second worker waits at its release gate. Even after more than two nominal intervals pass, no third worker has started. OnUnitInactiveSec is measured from deactivation, so unfinished work has not yet supplied that reference event.

Releasing the second worker lets it complete. The third worker then starts at least one configured interval after the recorded completion marker; actual unit deactivation occurs after that marker. The assertion verifies the completion-relative relationship without requiring a fragile exact launch timestamp. AccuracySec=1ms reduces coalescing in this short exercise, but it is not a real-time execution guarantee.

The script stops the timer while the third worker is still waiting. Its PID remains unchanged and the oneshot remains activating. Only releasing that worker lets the service complete successfully. After another interval has passed, the invocation count is unchanged, confirming that the stopped timer no longer schedules work. Both runtime unit files and the temporary worker are removed afterward.

Migrate the execution contract too

When moving a cron job, inspect more than its time expression. Use the intended working directory, explicit command paths, required environment, suitable user, and an appropriate runtime limit. systemd command lines are not implicitly shell programs. A pipeline, redirection, or shell expansion needs an intentional shell invocation or a separate script whose behavior is tested.

Installation and activation are also separate. Starting a runtime timer once does not enable it at boot, and enabling the service itself can introduce an unintended startup run. Install and enable the timer according to the deployment’s policy, then verify the loaded definitions and next activation. Keep the service independently runnable for controlled diagnosis.

Choose how failures should be retried and reported instead of assuming scheduling provides that policy. A later timer activation may try again, while service restart settings introduce their own rules and limits. Make repeated side effects safe and preserve a last-success signal tied to the actual result. The operational objective is completed useful work within its required window, not merely a timer that remains active.

Sources

systemd 252 systemd.timer, target selection, active-target behavior, and completion-relative intervals, checked 2026-09-12.
systemd 252 systemd.service, oneshot lifecycle, retained state, execution failure, and timeouts, checked 2026-09-12.
systemd 252 systemctl, starting, stopping, enabling, and inspecting timer and service units, checked 2026-09-12.

Ansible tasks can declare repeatable target states

Sun, 17 Jan 2027 00:00:00 +0000

An Ansible playbook is an ordered orchestration of tasks. State-aware modules can record what a machine should have and repair missing pieces after a partial deploy. Convergence comes from the selected modules and arguments, not an engine-wide guarantee that every task safely repeats. A playbook is an executable contract between the inventory, the modules, and the operator who reviews the change.

Say the end, not each step

An imperative shell script says “run these commands in this order.” A state-oriented task says “this package is installed, this file has these bytes, and this service is enabled and running.” A module compares the current host with that target before it acts. Matching tasks report ok; only a mismatch reports changed. The declaration can therefore be applied to a fresh host, a partly configured host, or one edited by hand.

The target is not vague intent. Module arguments are the contract: state: present, an exact template, an owner and mode, or a service’s enabled/running pair. Keep shared values in variables and roles, but make the final value visible to a reviewer. A hidden default or a command that mutates state outside the module makes the contract harder to reason about.

Modules make reruns converge

Most built-in modules are designed to be idempotent. ansible.builtin.copy compares content and metadata before replacing a file. ansible.builtin.package asks the package manager for the requested state. ansible.builtin.service with state: started changes running state only when necessary; state: restarted requests a restart on each run. A second invocation should be a quiet confirmation, not another restart, duplicate line, or appended record.

Some actions are not naturally idempotent. A raw command that creates a user or calls an API repeats its side effect unless guarded. Prefer a purpose-built module; otherwise use when with a registered read result, or precise creates/removes conditions. changed_when controls reporting and handler notification, not execution. Setting it to false cannot prevent a repeated mutation.

Inventory is the map

The playbook describes outcomes, while inventory selects hosts and variables select context. ansible_user selects the login identity; an SSH agent or Vault can supply authentication material without committing plaintext secrets. Group variables let one declaration cover a fleet while preserving host-specific values such as a port or region. Run with an explicit inventory in CI so “localhost” does not accidentally mean the runner itself.

Check mode (--check) asks modules to predict changes without applying them, and --diff shows content differences for modules that support it. Prediction is not proof: unsupported modules may skip work, and task-level check_mode: false deliberately executes even with --check. Review those overrides; secrets should not be printed in a diff. Treat a check-mode report as review material, then run normal mode with the same inventory and keep the result with the deployment record.

When it breaks

A playbook can drift when a task shells out, a module has incomplete check-mode support, or an unpinned package resolves a different version tomorrow. A host failure can leave a fleet partly changed; serial bounds batches but provides no rollback. Define batch gates, handler failure behavior, and recovery explicitly. A task depending on a fact gathered earlier must state that dependency instead of relying on incidental order.

Scope confusion is another failure: an inventory containing localhost, with a local connection targets the runner, while all selects every matching inventory host. Syntax checking validates structure, not whether the selected hosts match the operator’s intent. Review inventory and --list-hosts; inspect effective variables, including command-line overrides. Protect secrets with no_log and diff: false. Database migrations and cloud APIs still need their own transaction and retry semantics.

Convergence belongs to the chosen module and arguments. The copy-task sequence depicts assertions in the source, not a successful rerun: the retained exact verifier is blocked before its first task by local IPC restrictions. Syntax/version checks succeeded historically, but do not establish the runtime assertions. No example was rerun for this drawing.

Check it quickly

Save the YAML as site.yml and Python as verify_ansible.py in one directory, then run these commands. The local connection modifies only a temporary file.

python3 -m venv .venv-ansible
.venv-ansible/bin/python -m pip install ansible-core==2.21.4
.venv-ansible/bin/python verify_ansible.py

- name: Converge a disposable local file
 hosts: localhost
 connection: local
 gather_facts: false
 tasks:
 - name: Declare the banner content and mode
 ansible.builtin.copy:
 dest: "{{ fixture }}"
 content: "managed by ansible\n"
 mode: "0644"

import json
import os
from pathlib import Path
import re
import stat
import subprocess
import sys
import tempfile

playbook = Path(__file__).with_name("site.yml").resolve()
ansible = Path(sys.executable).parent / "ansible-playbook"
print(subprocess.check_output([str(ansible), "--version"], text=True).splitlines()[0])
with tempfile.TemporaryDirectory(prefix="ansible-state-") as directory:
 work = Path(directory)
 banner = work / "banner"
 config = work / "ansible.cfg"
 config.write_text("[defaults]\nstdout_callback=default\n", encoding="utf-8")
 env = dict(os.environ, ANSIBLE_CONFIG=str(config), ANSIBLE_NOCOLOR="1",
 ANSIBLE_LOCAL_TEMP=str(work / "local-tmp"))
 variables = {"fixture": str(banner), "ansible_python_interpreter": sys.executable,
 "ansible_remote_tmp": str(work / "module-tmp")}
 command = [str(ansible), "-i", "localhost,", str(playbook),
 "--extra-vars", json.dumps(variables)]

 def run(*options):
 result = subprocess.run(command + list(options), cwd=work, env=env,
 check=True, text=True, capture_output=True, timeout=60)
 print(result.stdout)
 return result.stdout

 def changed(expected, *options):
 output = run(*options)
 recap = re.search(r"localhost\s+:\s+ok=(\d+)\s+changed=(\d+)\s+"
 r"unreachable=(\d+)\s+failed=(\d+)", output)
 assert recap and tuple(map(int, recap.groups())) == (1, expected, 0, 0)
 return output

 run("--syntax-check")
 assert "managed by ansible" in changed(1, "--check", "--diff")
 assert not banner.exists(), "check mode created the absent file"
 changed(1)
 assert banner.read_bytes() == b"managed by ansible\n"
 assert stat.S_IMODE(banner.stat().st_mode) == 0o644
 changed(0)
 banner.write_bytes(b"manual drift\n")
 before = (banner.read_bytes(), banner.stat().st_mtime_ns, banner.stat().st_mode)
 diff = changed(1, "--check", "--diff")
 assert "manual drift" in diff and "managed by ansible" in diff
 assert before == (banner.read_bytes(), banner.stat().st_mtime_ns, banner.stat().st_mode)
 changed(1)
 assert banner.read_bytes() == b"managed by ansible\n"
 changed(0)
print("PASS: syntax, absent-file preview, convergence, drift preview, and repair")

The following describes the results the script is intended to check with Ansible Core 2.21.4: syntax checking succeeds. Check mode reports changed=1 without creating the absent file; normal execution creates it and a rerun reports changed=0. After manual drift, the preview shows both versions without changing bytes, mode, or modification time. Normal execution repairs the file and converges again. These assertions are used to check this copy task; other modules and upgrades need their own checks.

The retained 2026-10-15 review record confirms only that the version probe and syntax check passed. The exact verifier then failed during its first absent-file --check --diff invocation because of the local IPC restriction: Local RPC server did not start, exit status 250, before the first task ran. That run did not confirm the expected preview, creation, convergence, drift-preview or repair results. Earlier author observations remain historical; this retained attempt did not reproduce them.

Sources

Ansible playbooks introduction, playbook structure, tasks, and desired state, checked 2026-09-09.
Ansible check mode and diff mode, prediction and change previews, checked 2026-09-09.
Ansible error handling, changed reporting and handler notification, checked 2026-09-10.
Ansible service module, started versus restarted semantics, checked 2026-10-15.

Choose the clock and catch-up rule for a systemd timer

Fri, 20 Nov 2026 00:00:00 +0000

A schedule must specify what time means. “At nine each morning in New York” refers to a civil-time calendar. “A quarter hour after this job finishes” refers to an elapsed interval and an event. systemd timers support both, but the two clocks and reference points have different behavior around time-zone changes, suspension, and periods when the timer is inactive.

Calendar persistence requests one catch-up activation rather than replaying every missed business period. The dates and three-event example come from the article; this illustration adds no timer runtime evidence.

When it breaks

OnCalendar follows wall-clock calendar expressions. Omitting a time zone generally uses the system’s local zone, which may differ between machines. A daily local time can move relative to UTC when daylight-saving rules change. Preview the intended expression with a fixed reference date instead of assuming that today’s UTC offset applies throughout the year.

Monotonic options name their reference event explicitly. OnActiveSec refers to activation of the timer itself, OnStartupSec to the service manager’s startup, and OnUnitInactiveSec to the target unit’s deactivation. These are not interchangeable ways to say “every few minutes.” Ordinary monotonic timers pause with system suspension; WakeSystem changes clock selection and depends on platform support.

Persistent=yes affects calendar timers. It records trigger timing and requests a catch-up activation when the timer becomes active after missing at least one event. It does not replay every missed occurrence or certify successful job completion. The catch-up can also be delayed by configured randomization. A trigger history is not a durable queue of all business work that should have happened.

Check it quickly

Run this Python script as an administrator in a disposable Linux system with systemd as PID 1 and the time-zone database installed. The recovered text reports Debian 12 with systemd 252.39, but its original run log is unavailable. This review did not execute the fixture; the outcomes below describe its intended assertions. It first previews a calendar expression, then installs two unique timer/service pairs using the same three near-future calendar events. It never adjusts the machine’s clock:

from pathlib import Path
from tempfile import TemporaryDirectory
from datetime import datetime,timezone
import subprocess,time,uuid

name='xbcatm-calendar-'+uuid.uuid4().hex[:8]
names=[name,name+'-plain'];paths=[]
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
preview=run('systemd-analyze','calendar','--base-time=2026-03-07 00:00:00 UTC',
 '--iterations=3','*-*-* 09:00:00 America/New_York')
for expected in ('2026-03-07 14:00:00 UTC','2026-03-08 13:00:00 UTC','2026-03-09 13:00:00 UTC'):
 assert expected in preview,preview
print('09:00 New York: 14:00 UTC before spring DST, 13:00 UTC after it')
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py'
 worker.write_text("from pathlib import Path\nimport sys\np=Path(sys.argv[1])\nwith p.open('a') as output:output.write('run\\n')\n")
 now=int(time.time());events=[now+3,now+4,now+5]
 calendar=''.join('OnCalendar='+datetime.fromtimestamp(t,timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')+'\n' for t in events)
 try:
 for index,prefix in enumerate(names):
 service=Path('/run/systemd/system')/(prefix+'.service');timer=service.with_suffix('.timer')
 assert not service.exists() and not timer.exists()
 service.write_text(f'[Service]\nType=oneshot\nExecStart=/usr/bin/python3 {worker} {root}/{index}.log\n');paths.append(service)
 timer.write_text('[Timer]\n'+calendar+'AccuracySec=1ms\nRandomizedDelaySec=0\nPersistent='+('yes' if index==0 else 'no')+'\n');paths.append(timer)
 run('systemd-analyze','verify',*[str(p) for p in paths]);run('systemctl','daemon-reload')
 run('systemctl','start',name+'.timer');run('systemctl','stop',name+'.timer')
 assert not (root/'0.log').exists() and time.time()<events[0]
 while time.time()<events[-1]+0.15:time.sleep(0.05)
 run('systemctl','start',name+'.timer',name+'-plain.timer')
 deadline=time.monotonic()+3
 while not (root/'0.log').exists():
 assert time.monotonic()<deadline,'catch-up did not run';time.sleep(0.02)
 time.sleep(0.15)
 assert (root/'0.log').read_text()=='run\n' and not (root/'1.log').exists()
 print('three elapsed calendar events: persistent timer catches up once; plain timer does not')
 run('systemctl','stop',name+'.timer');run('systemctl','start',name+'.timer')
 time.sleep(0.2);assert (root/'0.log').read_text()=='run\n'
 print('reactivating with no newly missed event does not replay the job')
 print('timer inactivity was tested without changing the system clock or rebooting')
 finally:
 if paths:
 run('systemctl','stop',*[p.name for p in paths])
 run('systemctl','clean','--what=state',name+'.timer')
 for path in paths:path.unlink(missing_ok=True)
 run('systemctl','daemon-reload')

The preview asks for nine in the morning in New York, starting from a fixed date before the March 2026 daylight-saving transition. The first occurrence is 14:00 UTC; the following two are 13:00 UTC. The local clock time remains nine. This is a schedule interpretation check, not a request to change the host’s time zone or travel to those dates.

The persistent timer is started and stopped before the first laboratory event. Starting it establishes its timestamp state. While it is inactive, all three specified events pass. On reactivation it launches its service exactly once, producing one line in the result file. The nonpersistent timer, activated after the same events, produces no result file.

This experiment observes an actual inactive period rather than editing a timestamp file to impersonate a missed run. It demonstrates the catch-up decision without rebooting the machine. Three elapsed events become one activation; an application that needs to process three distinct accounting periods must discover and track those periods itself.

Stopping and starting the persistent timer again produces no extra line because no new event was missed. Cleanup stops the runtime units and removes the persistent timer’s state through systemctl clean --what=state. The nonpersistent timer has no matching persistent state to clean. Configuration files and the harmless result directory are then removed.

AccuracySec=1ms and zero randomized delay keep the short experiment interpretable. Neither setting promises exact process-start timing under arbitrary load. The code waits for observable completion within a bounded interval and asserts the resulting invocation count, rather than equating a predicted deadline with actual execution.

Decide which missed work matters

For a backup, one fresh snapshot after downtime may satisfy the recovery policy. For billing, each missing period might require separate reconciliation. Put that distinction into application state and idempotency rules. A service already active when a timer elapses is not automatically duplicated, and retained active oneshot state can prevent the command from running again.

Use the timer’s loaded configuration and next/last trigger properties to inspect scheduling, and the target service’s result to inspect execution. Calendar ordering after time-related targets does not by itself prove that a remote time source has synchronized the clock; the actual time-service configuration must supply that promise when it is required.

Test representative dates and lifecycle transitions before relying on a schedule. Include time-zone boundaries, timer disablement, and the task’s expected duration. Choose a monotonic reference for elapsed-delay requirements and a named calendar zone for civil-time requirements. Preserve evidence of completed useful work separately from evidence that the manager issued a trigger.

Sources

systemd 252 systemd.timer, clock references, calendar persistence, accuracy, and randomization, checked 2026-09-12.
systemd 252 systemd.time, calendar expressions and named time-zone syntax, checked 2026-09-12.
systemd 252 systemd-analyze, calendar previews from a fixed base time and multiple iterations, checked 2026-09-12.
systemd 252 systemctl, timer activation and removal of persistent timer state, checked 2026-09-12.

Scheduled jobs need to be safe to rerun

Sat, 15 Aug 2026 00:00:00 +0000

A scheduler starts attempts, while the business usually cares about completing a logical task. One daily credit may be attempted several times after an error, a restart, or an operator retry. Giving every attempt a new identifier makes those duplicates indistinguishable from new work. Use a key that identifies the intended task, such as its name and nominal scheduled time, and define what a repeated key means before changing business state.

Scheduling and execution are separate

Do not assume every scheduler catches up missed work. Ordinary cron behavior varies by implementation and does not provide a universal catch-up guarantee. A systemd calendar timer with Persistent=true can trigger after an inactive interval when at least one activation was missed; that does not mean replaying every missed occurrence. Kubernetes documents that a CronJob may create duplicate Jobs or miss an occurrence under some circumstances.

Choose the required business rule explicitly: skip stale work, process each missing period, or reconcile the current state. Use the nominal logical period in the deduplication key, with an unambiguous time zone. An actual start timestamp changes across retries and therefore cannot identify the same intended occurrence. Keep an attempt identifier separately for diagnostics without confusing it with business identity.

When it breaks

A check followed by an unrelated update has a race. Two workers can both observe that a key is absent and both apply the effect. A process lock may prevent overlap on one machine but does not make a business update atomic with a marker file. If the process fails after the update and before writing the marker, a retry repeats the effect. Writing the marker first can instead lose the work.

When both state and completion record live in one transactional database, commit them together under a uniqueness constraint. A failure before commit should preserve neither change; a retry after commit should see the completed logical key. In SQLite, BEGIN IMMEDIATE acquires the write transaction early, and competing writers may wait or fail with a busy error. A timeout bounds that wait; it does not make every workload contention-free.

This boundary does not cover an external email, payment, or HTTP call. A local rollback cannot undo an effect already accepted elsewhere. Such workflows need an appropriate remote idempotency contract or a durable delivery design with deduplication at the receiver. Reusing a key with a different payload also requires a defined rejection policy; the example below always applies the same fixed credit.

The fixed credit and unique completion key commit within one SQLite transaction. The historical eight-worker fixture passed at this local boundary; scheduler delivery, process-kill recovery and external exactly-once effects remain unverified. No example was rerun for this drawing.

Check it quickly

Save the code as idempotent-jobs.py and run python3 idempotent-jobs.py. The earlier source records Python 3.12.14 and SQLite 3.53.1; the separate dated rerun below records this review. Python must include sqlite3 linked to SQLite 3.24.0 or later for UPSERT, with support for separate connections in multiple threads. It uses a temporary SQLite database and eight worker threads, each with its own database connection. The business effect adds ten units to a balance. A non-null primary key records which logical daily credit has completed within the same transaction.

from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from tempfile import TemporaryDirectory
from threading import Barrier
import json
import sqlite3

with TemporaryDirectory() as directory:
 database = str(Path(directory) / 'jobs.db')
 with sqlite3.connect(database) as connection:
 connection.executescript('''
CREATE TABLE balance (id INTEGER PRIMARY KEY, units INTEGER NOT NULL);
INSERT INTO balance VALUES (1, 0);
CREATE TABLE completed (logical_key TEXT PRIMARY KEY NOT NULL);
''')

 def run(key, fail=False, gate=None):
 connection = sqlite3.connect(database, timeout=10, isolation_level=None)
 try:
 if gate:
 gate.wait(timeout=5)
 connection.execute('BEGIN IMMEDIATE')
 inserted = connection.execute(
 'INSERT INTO completed VALUES (?) ON CONFLICT(logical_key) DO NOTHING', (key,))
 if not inserted.rowcount:
 connection.execute('COMMIT')
 return 'duplicate'
 connection.execute('UPDATE balance SET units = units + 10 WHERE id = 1')
 if fail:
 raise RuntimeError('injected failure before commit')
 connection.execute('COMMIT')
 return 'applied'
 except Exception:
 if connection.in_transaction:
 connection.execute('ROLLBACK')
 raise
 finally:
 connection.close()

 key = 'daily-credit:2026-09-13T00:00:00Z'
 try:
 run(key, fail=True)
 except RuntimeError as error:
 assert str(error) == 'injected failure before commit'
 with sqlite3.connect(database) as connection:
 assert connection.execute('SELECT units FROM balance').fetchone() == (0,)
 assert connection.execute('SELECT count(*) FROM completed').fetchone() == (0,)
 gate = Barrier(8)
 with ThreadPoolExecutor(max_workers=8) as pool:
 results = list(pool.map(lambda _: run(key, gate=gate), range(8)))
 assert results.count('applied') == 1 and results.count('duplicate') == 7
 assert run(key) == 'duplicate'
 assert run('daily-credit:2026-09-14T00:00:00Z') == 'applied'
 with sqlite3.connect(database) as connection:
 units = connection.execute('SELECT units FROM balance').fetchone()[0]
 completed = connection.execute('SELECT count(*) FROM completed').fetchone()[0]
 assert (units, completed) == (20, 2)
 print(json.dumps({'rolled_back_attempt_left_no_state': True,
 'concurrent_applied': results.count('applied'),
 'concurrent_duplicates': results.count('duplicate'),
 'final_units': units, 'completed_logical_jobs': completed}))

The first attempt intentionally raises after both inserts and updates but before commit. The script confirms that rollback leaves zero units and no completed key. Eight workers then start together for the same logical occurrence. Exactly one reports applied; seven report duplicate. Their unique-key decision and balance mutation are serialized by the actual database, not a mocked lock.

A further replay after successful commit remains a duplicate. A different nominal date is a new logical task and applies another ten units. The final result is twenty units and two completed keys. This verifies rollback and replay within one SQLite database, not scheduler delivery guarantees, process-kill recovery, or external exactly-once execution.

A separate 2026-10-19 rerun passed the unchanged fixture with Python 3.12.14 and SQLite 3.53.1 on Linux 6.18.55. It confirms the same local transaction assertions, without executing any scheduler or testing process-kill recovery.

Keep failure and completion visible

Record the logical key, attempt outcome, duration, and last successful business period. Distinguish a harmless duplicate from a database error; catching every exception and labeling it completed would hide lost work. Define how long completion records must remain available before deleting them, since a retry after deletion can apply again. Operational evidence should tell an operator whether the intended period completed, rather than merely whether a timer launched a process.

Sources

SQLite: transactions, writer serialization, commit, and rollback, checked 2026-10-15.
SQLite: UPSERT, targeted handling of a uniqueness conflict, checked 2026-10-15.
systemd.timer, the scope of persistent calendar catch-up, checked 2026-10-15.
Kubernetes: CronJob, approximate scheduling and duplicate or missed jobs, checked 2026-10-15.

Automation · XB Field Notes

Cron repeats on the calendar

Read fields as calendar selections

When it breaks

Check it quickly

Sources

GitHub Actions workflows gate the build

Separate execution from enforcement

Make the dependency result explicit

Permissions describe the token, not all credentials

When it breaks

Check it quickly

Further reading

Sources

A systemd timer schedules a service, not its success

When it breaks

Check it quickly

Migrate the execution contract too

Further reading

Sources

Ansible tasks can declare repeatable target states

Say the end, not each step

Modules make reruns converge

Inventory is the map

When it breaks

Check it quickly

Further reading

Sources

Choose the clock and catch-up rule for a systemd timer

When it breaks

Check it quickly

Decide which missed work matters

Further reading

Sources

Scheduled jobs need to be safe to rerun

Scheduling and execution are separate

When it breaks

Check it quickly

Keep failure and completion visible

Further reading

Sources