Workloads · XB Field Notes

ConfigMaps separate configuration from the image

Fri, 20 Aug 2027 00:00:00 +0000

A ConfigMap stores non-confidential configuration separately from an image. That lets a release change an environment-specific value without rebuilding the application artifact. The useful boundary is explicit: the image supplies code, while the workload definition identifies which configuration that code should consume. A release is therefore more than an image tag; its configuration reference is part of the behavior being deployed.

Configuration reference is part of a release. The diagram explains historical selection and retained-input rollback, without proving application compatibility, rollout availability or reversal of external effects. No example was rerun.

Give configuration a reviewable identity

ConfigMap data values are strings, with binaryData available for binary values represented through the API’s encoding. The object is intended for small configuration, with a one-MiB data limit. It is not a general file repository or a secret-protection mechanism. The API can validate object structure without understanding whether a particular application setting is meaningful or safe for the business.

A Pod can consume values through environment references or mounted files. Ordinary Pod references resolve the ConfigMap in that Pod’s namespace. Choose names and required keys deliberately, then validate the application’s interpretation. An accepted Deployment object does not prove that every referenced key exists or that a new container can start successfully with the selected values.

Define units, accepted values, and defaults in the application’s configuration schema. A stored string resembling a boolean or number does not choose its parsing rules. Review those rules with the consumer before promoting a configuration between environments.

Versioned immutable ConfigMaps make release identity easier to inspect. A name such as settings-v2 can be retained alongside settings-v1, with immutable preventing data changes to the existing object. This protection does not prohibit an authorized actor from deleting and recreating a name. Keep object lifecycle permissions and release records aligned with the intended immutability promise.

When it breaks

Editing configuration and replacing processes are different actions. Environment values already injected into a running container are not rewritten by a ConfigMap update. Normal projected files can update eventually, but the application must read the new contents; subPath and previously opened files introduce additional lifetime distinctions. Do not report a configuration rollout merely because the API now shows a new value.

Changing a Deployment’s Pod-template reference to a new ConfigMap is a concrete rollout trigger. Changing the ConfigMap itself does not automatically rewrite that template. Likewise, a checksum annotation is useful only when a tool actually updates it in the Pod template. Kubernetes does not discover arbitrary external configuration changes and generate that checksum on the application’s behalf.

Rollback depends on retained inputs. Repointing a workload to settings-v1 fails if the old object has been deleted, and restoring an image alone cannot restore a changed configuration. Keep needed versions available and check consumers before cleanup. Configuration changes also need business validation; a syntactically valid threshold can still disable a service or change its behavior incorrectly.

Check it quickly

Use a disposable Kubernetes cluster with kubectl access and python:3.11-slim-bookworm available to its nodes. Save this as config-releases.py and run python3 config-releases.py KUBECONFIG_PATH CONTEXT, substituting that cluster’s dedicated configuration path and context name. The test needs permission to create and delete its own namespace. The recorded server version was Kubernetes 1.34.11, with kubectl 1.34.1.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}


def deployment(namespace, name, podspec):
 return apply(namespace, {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': {'name': name},
 'spec': {'replicas': 1, 'selector': {'matchLabels': {'app': name}},
 'template': {'metadata': {'labels': {'app': name}}, 'spec': podspec}}})


def deployment_pod(namespace, name):
 wait_for(lambda: obj(namespace, 'deployment', name), lambda d:
 d.get('status', {}).get('observedGeneration', 0) >= d['metadata']['generation']
 and d.get('status', {}).get('updatedReplicas') == 1
 and d.get('status', {}).get('availableReplicas') == 1
 and d.get('status', {}).get('replicas') == 1)
 def pods():
 return [p for p in obj(namespace, 'pods', selector='app=' + name)['items']
 if not p['metadata'].get('deletionTimestamp')]
 return wait_for(pods, lambda p: len(p) == 1 and pod_ready(p[0]))[0]

namespace = 'notes-config-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 for version, mode in [('v1', 'blue'), ('v2', 'green')]:
 apply(namespace, {'apiVersion': 'v1', 'kind': 'ConfigMap',
 'metadata': {'name': 'settings-' + version}, 'immutable': True, 'data': {'mode': mode}})
 podspec = spec()
 podspec['containers'][0]['env'] = [{'name': 'MODE', 'valueFrom':
 {'configMapKeyRef': {'name': 'settings-v1', 'key': 'mode'}}}]
 deployment(namespace, 'web', podspec)
 first = deployment_pod(namespace, 'web')
 def mode(pod):
 return k('exec', '-n', namespace, pod['metadata']['name'], '--', 'python3', '-c',
 'import os; print(os.environ["MODE"])').stdout.strip()
 assert mode(first) == 'blue'
 rejected = k('patch', 'configmap', 'settings-v1', '-n', namespace, '--type=merge',
 '-p', json.dumps({'data': {'mode': 'changed'}}), expected=None)
 assert rejected.returncode != 0 and 'immutable' in rejected.stderr
 assert obj(namespace, 'configmap', 'settings-v1')['data']['mode'] == 'blue'
 def select(version):
 patch = [{'op': 'replace',
 'path': '/spec/template/spec/containers/0/env/0/valueFrom/configMapKeyRef/name',
 'value': 'settings-' + version}]
 k('patch', 'deployment', 'web', '-n', namespace, '--type=json', '-p', json.dumps(patch))
 return deployment_pod(namespace, 'web')
 second = select('v2')
 assert mode(second) == 'green' and second['metadata']['uid'] != first['metadata']['uid']
 third = select('v1')
 assert mode(third) == 'blue' and third['metadata']['uid'] != second['metadata']['uid']
 image_ids = [p['status']['containerStatuses'][0]['imageID'] for p in (first, second, third)]
 assert len(set(image_ids)) == 1
 assert len({p['metadata']['uid'] for p in (first, second, third)}) == 3
 print(json.dumps({'modes': ['blue', 'green', 'blue'], 'distinct_pod_uids': 3,
 'same_image_id': image_ids[0], 'immutable_update_rejected': True,
 'retained_config_rollback_verified': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script creates two immutable ConfigMaps and a single-replica Deployment referencing settings-v1. The first Pod reads blue from its environment. Attempting to overwrite that ConfigMap’s data is rejected, and reading the object confirms the original value remains blue. This checks actual server enforcement rather than just the manifest’s immutable field.

Next, a Pod-template patch selects settings-v2. After controller reconciliation and readiness, a different Pod reads green. Selecting the retained first version causes another replacement that reads blue again. The three Pod UIDs are distinct, while all three observed image identities match. The original run reports all assertions passed and namespace cleanup; no cluster run was repeated here. The image tag remains mutable: matching observed identities prove this run, not future reproducibility. Pin a digest when that is required.

This verifies configuration selection, replacement, and a retained-input rollback using a simple worker. It does not establish application compatibility, production availability during rollout, or rollback of external side effects. Carry the image, configuration version, workload definition, and observed application result together in the release evidence so each part of the change can be reviewed.

Sources

Kubernetes: ConfigMaps, non-confidential data, consumption and immutability, checked 2026-09-13.
Kubernetes: Deployments, Pod-template changes and rolling replacement, checked 2026-09-13.
Kubernetes: Images, mutable tags and digest-pinned identity, checked 2026-10-15.

Kubernetes Deployments describe desired state

Thu, 19 Aug 2027 00:00:00 +0000

You describe the end state, not the steps. That idea underlies replicas, updates, and rollback: the cluster keeps working to match your description.

The count sequences describe the article’s local arithmetic model. Real controllers have separate responsibilities and asynchronous failure modes. Historical model passes are unchanged; no example was rerun.

Spec beats imperative steps

A Deployment is a statement of the end state, not a list of steps: how many replicas, which image, what update strategy, which labels. The operator publishes the what, and the controller, a reconcile loop, is responsible for the how. The distinction is resource ownership, not whether a shell command was used. kubectl run normally creates a standalone Pod; its kubelet can restart containers, but no ReplicaSet maintains a replacement population after that Pod disappears. A Deployment can itself be created with a command, yet its stored specification continues guiding controllers after the client exits. Interrupting the client does not reliably indicate whether the API accepted the request: inspect the stored object before retrying or diagnosing convergence.

A controller converges the gap

The Deployment controller manages ReplicaSets, and ReplicaSet controllers maintain their Pod populations. Kubelet can restart a crashed container within the same Pod; replacing a deleted or lost Pod is a different reconciliation path. These mechanisms cooperate, but should not be collapsed into one controller that creates a Pod every few seconds.

Reconciliation is asynchronous and depends on working control-plane components and sufficient capacity. Image pull failures, quotas, admission denial or unavailable storage can prevent convergence indefinitely. Observe conditions and events. A desired replica count is a target to pursue, not proof of ready capacity or a bound on recovery time.

Updates happen through new Revision

Changing the Pod template triggers a rollout and revision; scaling alone does not. The controller adjusts old and new ReplicaSet sizes according to the rollout strategy, readiness and availability constraints. Services and their datapaths handle traffic separately, so the Deployment controller itself does not gradually switch individual connections.

A rollback restores a previous Pod template, not every field of the Deployment or external state such as a database schema. Retained ReplicaSets supply rollback history, with a default revisionHistoryLimit of ten; cleanup starts after completion. Reapplying an unchanged template does not generate a fresh revision merely because a pipeline ran.

When it breaks

In apps/v1, a Deployment selector is immutable and the Pod template must match it. A template that violates the selector is rejected by API validation, rather than creating an accepted Deployment that endlessly produces unowned Pods. Plan selector changes as an explicit migration. Overlapping controllers deserve scrutiny, but matching labels alone do not erase controlling owner references.

Deleting a managed Pod or directly scaling an owned ReplicaSet can be corrected by its controllers. In contrast, kubectl scale deployment updates the Deployment’s desired replicas and persists until another actor changes that field. HPA, a later apply or a GitOps reconciler can overwrite it; the Deployment does not secretly remember an earlier Git value on its own.

A stalled rollout can retain more ReplicaSets than the intended historical limit because cleanup has completion conditions. Inspect rollout status before declaring a retention leak. Keep external change history and a tested rollback plan; zero retained history removes normal rollback targets, while a large number alone does not prove control-plane performance degradation.

Check it quickly

Save the following as desired_state_loop.py and run python3 desired_state_loop.py. It uses only the standard library and illustrates a simplified count-reconciliation loop: compare desired against observed, emit one corrective action per tick, and repeat until the gap closes.

"""Simulate a Deployment-style desired-state reconcile loop (stdlib only)."""

def reconcile_step(desired, observed):
 if observed < desired:
 return "create-one-pod"
 if observed > desired:
 return "delete-one-pod"
 return "in-sync"


def converge(desired, observed, log):
 while observed != desired:
 action = reconcile_step(desired, observed)
 log.append(action)
 if action == "create-one-pod":
 observed += 1
 else:
 observed -= 1
 log.append("in-sync")
 return observed


def main():
 desired = 3
 observed = 1 # one survivor after a node loss
 actions = []
 observed = converge(desired, observed, actions)
 assert observed == desired
 assert actions == ["create-one-pod", "create-one-pod", "in-sync"], actions

 # Manual drift: someone adds two pods without changing desired.
 observed = 5
 drift = []
 observed = converge(desired, observed, drift)
 assert observed == desired
 assert drift == ["delete-one-pod", "delete-one-pod", "in-sync"], drift

 # Model a selector/template validation check before accepting desired state.
 selector = {"app": "api"}
 template_labels = {"app": "api-v2"} # deliberate mismatch, a negative control
 matched = all(template_labels.get(k) == v for k, v in selector.items())
 assert not matched, "mismatched template must fail this validation model"

 print(f"desired={desired} final-observed={observed}")
 print(f"recovery-actions={actions}")
 print(f"observed-count-drift-corrected={drift}")
 print("PASS: controller closes the desired-vs-observed gap; mismatched template is rejected by the local validation check")


if __name__ == "__main__":
 main()

The two integer-count scenarios end in sync. The separate label check rejects the mismatched template as a negative control, illustrating the contract checked before API acceptance. It does not submit a real Deployment or model adoption, watches, scheduling and readiness.

Both language fixtures passed this review with Python 3.12.14 on Linux. The model checks convergence arithmetic, not scheduling or storage. Before release day, confirm the live selector, replicas, and revision history.

Sources

Kubernetes: Deployments, the Deployment resource explained, checked 2026-10-15.
Kubernetes: Controllers, the platform and its model, checked 2026-10-15.

A DaemonSet puts one Pod on every eligible node

Mon, 29 Mar 2027 00:00:00 +0000

A Deployment targets a replica count; a DaemonSet targets coverage of eligible nodes. That is useful for node-level log collectors, network agents, and device plugins. A newly eligible node creates reconciliation work, not an immediate guarantee that its agent is running or ready. Selection rules, scheduling, image pulls, startup, and node health can all delay useful coverage.

The controller creates and reconciles Pods from a shared template. Its desired state is normally one managed Pod per eligible node, rather than an operator-maintained replica count. Desired coverage and observed readiness are separate measurements. Keep an independently defined intended node population so a mistaken selector cannot make excluded machines disappear from the audit.

One per node, by design

The controller evaluates node selectors and required affinity, creates a Pod targeted to each eligible node, and normally lets the scheduler bind it. Nodes joining or leaving change that desired population. DaemonSet Pods automatically tolerate the unschedulable taint, so cordoning a node alone does not exclude them. Registration does not make scheduling, image retrieval, or initialization instantaneous.

Priority can help an important daemon preempt lower-priority workloads, but it does not create capacity or satisfy incompatible placement constraints. Choose priority and resource requests according to actual operational importance and workload impact. A Pod can remain Pending or fail after scheduling; neither a high PriorityClass nor a toleration proves that it is ready or immune to node-pressure eviction.

Coverage over count

RollingUpdate replaces Pods in a controlled sequence; OnDelete waits for old Pods to be deleted before replacing them from the changed template. With maxSurge zero, maxUnavailable one permits an unavailable node during rollout, not uninterrupted per-node coverage. A nonzero maxSurge can temporarily place old and new Pods on one node and needs extra resources. Version skew during updates is expected, and blocked rollouts can prolong it. kubectl rollout status ds/agent reports controller rollout progress, not continuity of delivered logs.

Selectors and affinity can divide heterogeneous fleets into deliberate subsets. A GPU device plugin and an edge log forwarder need not share one DaemonSet. Check the intended membership of each subset, and distinguish a node that is intentionally excluded from one whose expected agent is missing.

Rides the node lifecycle

The controller adds documented tolerations for conditions including not-ready, unreachable, several pressure taints, and unschedulable. Custom control-plane or hardware taints may still require explicit tolerations. A toleration permits placement past its matching taint; it does not select that node or overcome every other constraint. Label changes can alter membership, so observe the actual reconciliation rather than assuming an atomic handover between daemon fleets.

Ordinary kubectl drain refuses DaemonSet-managed Pods unless –ignore-daemonsets is supplied; that option skips them rather than deleting them. Cordoning and draining are therefore different from stopping these node services. Configure graceful termination for actual Pod deletion, shutdown, or other supported maintenance paths, and test the intended node-lifecycle procedure.

When it breaks

A new tainted pool can be silently excluded while desiredNumberScheduled still equals numberReady. Those fields describe the DaemonSet controller’s population, not every machine the operator intended to cover, and they are not per-pool counters. Compare node identities, selectors, taints, owned Pods, readiness, and update revision. A mismatch identifies a gap; events and Pod conditions are needed to distinguish missing tolerance, insufficient resources, image failure, or application startup failure.

A bad daemon update can remove observability on several nodes. Choose disruption and surge budgets, test a selector-scoped canary, and verify collection continuity and resource use before widening the rollout. Keep a usable previous revision and confirm rollback actually converges; a command returning successfully does not recover missing logs or guarantee that an old image remains available.

DaemonSet coverage audit: mechanism and evidence boundaries.

Check it quickly

Use Bash, Python 3, an authorized context/namespace/DaemonSet, and INTENDED_NODES: an independently maintained JSON map of intended node names to UIDs. Do not derive that roster from this DaemonSet’s selector. The script reads JSON rather than table columns:

(
set -euo pipefail
: "${CONTEXT:?Set the authorized kubectl context}"
: "${NAMESPACE:?Set the DaemonSet namespace}"
: "${DAEMONSET:?Set the DaemonSet name}"
: "${INTENDED_NODES:?Set the independent node-name to UID JSON file}"
umask 077
tmp=$(mktemp -d)
trap 'rm -rf -- "$tmp"' EXIT
k=(kubectl --context "$CONTEXT" --request-timeout=15s)
"${k[@]}" -n "$NAMESPACE" get ds "$DAEMONSET" -o json > "$tmp/before.json"
"${k[@]}" get nodes -o json > "$tmp/nodes.json"
"${k[@]}" -n "$NAMESPACE" get pods -o json > "$tmp/pods.json"
"${k[@]}" -n "$NAMESPACE" get ds "$DAEMONSET" -o json > "$tmp/after.json"
python3 - "$tmp" "$INTENDED_NODES" "$NAMESPACE" "$DAEMONSET" <<'PY'
import collections, json, pathlib, sys
root, roster, namespace, name = sys.argv[1:]
def require(ok, message):
 if not ok:
 raise SystemExit(message)
def unique(pairs):
 result = {}
 for key, value in pairs:
 require(key not in result, "Duplicate JSON key")
 result[key] = value
 return result
def load(path):
 return json.loads(pathlib.Path(path).read_text(), object_pairs_hook=unique)
def ready(obj):
 return any(c.get("type") == "Ready" and c.get("status") == "True"
 for c in obj.get("status", {}).get("conditions", []))
a, b = (load(pathlib.Path(root) / (x + ".json")) for x in ("before", "after"))
for ds in (a, b):
 require(ds["kind"] == "DaemonSet" and ds["apiVersion"] == "apps/v1",
 "Wrong resource type")
 require(ds["metadata"]["namespace"] == namespace and
 ds["metadata"]["name"] == name, "Wrong DaemonSet identity")
 require(ds["metadata"].get("uid") and not ds["metadata"].get("deletionTimestamp"),
 "Missing UID or deleting DaemonSet")
require(all(a["metadata"][k] == b["metadata"][k] for k in ("uid", "generation")),
 "DaemonSet changed during collection; retry")
uid = a["metadata"]["uid"]
expected = load(roster)
require(isinstance(expected, dict) and expected and
 all(isinstance(k, str) and k and isinstance(v, str) and v
 for k, v in expected.items()), "Use a nonempty node-name to UID map")
node_items = load(pathlib.Path(root) / "nodes.json")["items"]
nodes = {n["metadata"]["name"]: n for n in node_items}
require(len(nodes) == len(node_items), "Duplicate node names")
wrong_nodes = sorted(n for n in expected if n not in nodes or
 nodes[n]["metadata"]["uid"] != expected[n])
unready_nodes = sorted(n for n in expected if n in nodes and not ready(nodes[n]))
rows, seen = [], set()
for pod in load(pathlib.Path(root) / "pods.json")["items"]:
 m = pod["metadata"]
 require(m["namespace"] == namespace, "Wrong Pod namespace")
 if not any(o.get("controller") is True and o.get("uid") == uid and
 o.get("kind") == "DaemonSet" and o.get("apiVersion") == "apps/v1"
 and o.get("name") == name for o in m.get("ownerReferences", [])):
 continue
 require(m.get("uid") and m["uid"] not in seen, "Missing or duplicate Pod UID")
 seen.add(m["uid"])
 node = pod["spec"].get("nodeName", "")
 phase = pod.get("status", {}).get("phase", "Unknown")
 terminating = bool(m.get("deletionTimestamp"))
 rows.append(dict(pod=m["name"], uid=m["uid"], node=node, phase=phase,
 ready=ready(pod) and phase == "Running" and not terminating,
 terminating=terminating,
 revision=m.get("labels", {}).get("controller-revision-hash")))
counts = collections.Counter(r["node"] for r in rows if r["node"])
ready_nodes = {r["node"] for r in rows if r["ready"] and r["node"]}
issues = dict(node_identity_mismatch=wrong_nodes, node_not_ready=unready_nodes,
 missing_ready=sorted(set(expected) - ready_nodes),
 unexpected_owned_nodes=sorted(set(counts) - set(expected)),
 multiple_pods=sorted(n for n, count in counts.items() if count > 1),
 unassigned=[r["uid"] for r in rows if not r["node"]],
 unready_pods=[r["uid"] for r in rows if not r["ready"]])
print(json.dumps(dict(namespace=namespace, daemonset=name, uid=uid,
 intended=expected, pods=rows, issues=issues), indent=2))
sys.exit(1 if any(issues.values()) else 0)
PY
)

Zero means this collected snapshot has one nonterminating Running/Ready owned Pod per intended, Ready node with matching UID. Nonzero requires investigation: surge duplicates can be legitimate; unassigned Pods remain unresolved. Pod UIDs and revisions are reported, but target revision, placement eligibility and log delivery are not proved. Separate API reads are not atomic; recheck during change. Node UIDs do not prove current physical execution. Only syntax and synthetic file controls were exercised here. No cluster, rollout or drain ran; technical and runtime validation remain held.

Sources

Kubernetes node-pressure eviction, DaemonSet priority does not confer eviction immunity, checked 2026-10-18.
Kubernetes DaemonSet API, desired and ready status plus surge and unavailable budgets, checked 2026-10-18.
Kubernetes node drain, drain skips DaemonSet Pods when ignore-daemonsets is selected, checked 2026-10-18.
Kubernetes DaemonSet rolling updates, update strategies, disruption budget and surge configuration, checked 2026-10-18.
Kubernetes DaemonSet, documents daemonset behavior, checked 2026-06-01.
Kubernetes taints and tolerations, documents reaching tainted nodes, checked 2026-09-05.

Regular init containers establish startup prerequisites

Sun, 03 Jan 2027 00:00:00 +0000

A regular init container performs setup before a Pod’s application containers start. It has its own command, image and filesystem view, and can mount a volume that the application later reads. This separates a prerequisite from the main process, but does not make arbitrary setup safe or automatically bounded. Design the step around its actual retry behavior and the lifetime of its output.

Historical source fixture, not rerun; execution remains blocked. Regular init ordering is local to one Pod. The explicit emptyDir outlives container restarts but disappears with the Pod; no durable storage or fleet serialization is implied.

Sequence is local to one Pod

Kubelet runs ordinary init containers in their declared order. Each must finish successfully before the next begins, and the application containers wait for that sequence. Inspect initContainerStatuses to distinguish a waiting step, a running step and a terminated attempt. A Pod’s phase alone does not explain which command failed or whether another attempt is underway.

Shared storage requires explicit mounts. An emptyDir can carry a generated file between init and application containers, and it survives a container restart within that Pod. It disappears with the Pod. The separate containers do not share their image root filesystems, and a file written only inside an init container’s writable layer is not automatically delivered to the application.

Ordinary init containers must complete. Kubernetes also supports restartable sidecars represented in initContainers with container-level restartPolicy set to Always; those have different lifecycle rules. Do not apply the ordinary run-to-completion description to every entry merely because it appears in that field. This article’s example uses only regular init containers.

When it breaks

A nonzero exit does not always fail the whole Pod immediately. With an ordinary application Pod’s Always policy, a failed regular init is retried until it succeeds; successful init steps do not restart merely because the application remains running. Backoff slows repeated attempts, but is not a fixed retry-count limit. With Pod restartPolicy Never, a failed initialization instead makes that Pod fail.

Write setup so another attempt can encounter existing output safely. An interrupted download may leave a partial file, and an emptyDir marker may survive the failed process. Validate the completed artifact and publish it deliberately. The counter used below is diagnostic state; it is not a production strategy for deciding that an external prerequisite must have become valid.

Ordering inside one Pod also does not serialize a fleet. Putting a database migration in every replica’s init sequence can run several migrations concurrently. Use the migration system’s actual locking and compatibility contract. Waiting for a dependency once cannot prove that dependency will remain available after initialization; continuing request handling and readiness need their own design.

Initialization affects resource placement too. For ordinary init containers, scheduling accounts for peak initialization requirements alongside the summed application requirements, with overhead where applicable. A brief but large setup request can prevent scheduling even when the steady application is small. Sidecars and Pod-level resource features require checking their additional accounting rules.

Check it quickly

Use a disposable Kubernetes cluster, kubectl access and python:3.11-slim-bookworm on its nodes. Save the code as init-order.py and run python3 init-order.py KUBECONFIG_PATH CONTEXT, replacing the arguments with explicit test-cluster details. The account must create and delete namespaces. The original article records Kubernetes 1.34.11; allow time for one failed initialization and its retry. This review did not rerun the cluster fixture.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-init-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 prepare = r'''from pathlib import Path
import sys,time
root=Path('/work'); counter=root/'attempts'
attempt=int(counter.read_text())+1 if counter.exists() else 1
counter.write_text(str(attempt))
with (root/'order').open('a') as f: f.write('prepare'+str(attempt)+'\n')
if attempt == 1: sys.exit(7)
print('waiting for release',flush=True)
while not (root/'go').exists(): time.sleep(0.1)
(root/'input').write_text('ready')
'''
 transform = r'''from pathlib import Path
root=Path('/work'); assert (root/'input').read_text()=='ready'
(root/'output').write_text('READY')
with (root/'order').open('a') as f: f.write('transform\n')
'''
 main = r'''from pathlib import Path
import json,time
root=Path('/work'); assert (root/'output').read_text()=='READY'
with (root/'order').open('a') as f: f.write('app\n')
print(json.dumps({'order':(root/'order').read_text().splitlines(),
 'artifact':(root/'output').read_text()}),flush=True)
time.sleep(600)
'''
 podspec = spec(main)
 mount = [{'name': 'work', 'mountPath': '/work'}]
 podspec['volumes'] = [{'name': 'work', 'emptyDir': {}}]
 podspec['containers'][0]['volumeMounts'] = mount
 podspec['initContainers'] = []
 for name, script in [('prepare', prepare), ('transform', transform)]:
 container = spec(script)['containers'][0]
 container['name'] = name
 container['volumeMounts'] = mount
 podspec['initContainers'].append(container)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'ordered'}, 'spec': podspec})
 def retry_running(p):
 statuses = p.get('status', {}).get('initContainerStatuses', [])
 return statuses and statuses[0]['restartCount'] >= 1 and 'running' in statuses[0]['state']
 waiting = wait_for(lambda: obj(namespace, 'pod', 'ordered'), retry_running)
 first = waiting['status']['initContainerStatuses'][0]
 assert first['lastState']['terminated']['exitCode'] == 7
 assert waiting['status']['phase'] == 'Pending'
 assert all('running' not in c['state'] for c in waiting['status'].get('containerStatuses', []))
 assert 'running' not in waiting['status']['initContainerStatuses'][1]['state']
 k('exec', '-n', namespace, 'ordered', '-c', 'prepare', '--', 'python3', '-c',
 'from pathlib import Path; Path("/work/go").touch()')
 ready = wait_for(lambda: obj(namespace, 'pod', 'ordered'), pod_ready)
 assert ready['metadata']['uid'] == waiting['metadata']['uid']
 output = wait_for(lambda: k('logs', '-n', namespace, 'ordered', '-c', 'app').stdout.strip(), bool)
 observed = json.loads(output)
 assert observed == {'order': ['prepare1', 'prepare2', 'transform', 'app'], 'artifact': 'READY'}
 assert ready['status']['initContainerStatuses'][0]['restartCount'] == 1
 assert all(c['state']['terminated']['exitCode'] == 0 for c in ready['status']['initContainerStatuses'])
 never = spec()
 never['restartPolicy'] = 'Never'
 failed_init = spec('raise SystemExit(7)')['containers'][0]
 failed_init['name'] = 'fails'
 never['initContainers'] = [failed_init]
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'never'}, 'spec': never})
 failed = wait_for(lambda: obj(namespace, 'pod', 'never'), lambda p: p.get('status', {}).get('phase') == 'Failed')
 status = failed['status']['initContainerStatuses'][0]
 assert status['restartCount'] == 0 and status['state']['terminated']['exitCode'] == 7
 assert not any('running' in c['state'] for c in failed['status'].get('containerStatuses', []))
 print(json.dumps({'ordered_artifact': observed, 'retried_init_count': 1,
 'app_waited_for_both_regular_init_containers': True,
 'never_policy_failed_pod_without_retry': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The preparation container records its first attempt and exits seven. During its second attempt it waits for a release file. The script confirms the Pod is Pending, the first failure is recorded, and neither the next init step nor the application is running. It then releases preparation through that container’s shared volume.

The second init transforms ready into READY, and the application validates the artifact. Its log records prepare1, prepare2, transform and app in that order; the first init’s restart count is one. A separate Pod with Never policy exits initialization unsuccessfully and reaches Failed with no init retry. The original article reports passing assertions and removal of both Pods with their namespace; this is historical execution attribution. The experiment verifies local sequencing and failure policy, not distributed migration safety or durable storage.

Sources

Kubernetes: Init Containers, regular init ordering, retries, resource accounting and repeatable setup, checked 2026-10-15.
Kubernetes: Pod Lifecycle, Pod phases, container states and restart policies, checked 2026-10-15.

Workloads · XB Field Notes

ConfigMaps separate configuration from the image

Give configuration a reviewable identity

When it breaks

Check it quickly

Sources

Kubernetes Deployments describe desired state

Spec beats imperative steps

A controller converges the gap

Updates happen through new Revision

When it breaks

Check it quickly

Further reading

Sources

A DaemonSet puts one Pod on every eligible node

One per node, by design

Coverage over count

Rides the node lifecycle

When it breaks

Check it quickly

Sources

Regular init containers establish startup prerequisites

Sequence is local to one Pod

When it breaks

Check it quickly

Sources