Kubernetes · XB Field Notes

The scheduler accounts for requests, not momentary utilization

Sun, 02 Apr 2028 00:00:00 +0000

Kubernetes scheduling answers whether a Pod can fit a node under the applicable resource requests and placement constraints. It does not simply look at a current CPU utilization chart and place work wherever the line is low. Requests provide accounting inputs for future demand; actual usage is a separate observation. Diagnose both before deciding that an idle-looking cluster has capacity for a particular Pod.

Read the admitted requests. API rejection, an accepted Pending Pod, and successful placement are different stages.

Account for the admitted specification

For ordinary application containers, the scheduler accounts for their requested resources when testing node fit. Initialization requirements, Pod overhead and supported Pod-level resource configuration can also affect the effective request. The final admitted Pod is the relevant input, including defaults and mutations, rather than just the source manifest that an operator remembers submitting.

Node allocatable is also different from the hardware’s total capacity. It represents the resource amount available to Pods under the node’s configuration. Other admitted workloads consume scheduling headroom through their requests even when their processes happen to be idle. A request does not mean that physical memory has already been allocated to the application or that CPU cores are exclusively assigned.

Limits constrain running workloads through the execution stack; they are not generally substituted for an explicitly smaller request in scheduler fit checks. A container can therefore have a CPU limit larger than one node’s CPU capacity and still schedule with a small request. That accepted limit does not manufacture additional processors or promise the process can reach the declared ceiling.

When it breaks

A Pending Pod may be blocked by insufficient requested resources, but also by taints, affinity, storage, topology or other constraints. Read the PodScheduled condition and scheduling events before changing resource values. Lowering requests cannot repair an incompatible node selector or unavailable volume, and blindly lowering an honest request can create runtime contention instead of a sound placement.

Defaulting makes omission easy to misread. When a limit is specified without a request, and no admission-time mechanism supplies another request, Kubernetes copies that resource’s limit into its request. Setting only a large limit can therefore produce a large scheduling demand. Inspect both maps on the stored object, including any LimitRange effects, before interpreting a failed placement.

The API also rejects a request greater than its corresponding limit. That is an admission failure, not a scheduler decision about cluster capacity. Keeping those failure stages separate helps explain why one manifest never creates a Pod while another produces an accepted but unscheduled Pod that can be inspected.

QoS is not an immunity promise. Under conventional per-container rules, Guaranteed requires CPU and memory requests and limits to be present and equal for every relevant container. Matching only memory is insufficient. Even a Guaranteed Pod can encounter resource failure; the class does not replace priority, capacity planning, application correctness or actual node-pressure analysis.

Check it quickly

The fixture and observations below are retained from the original article. This review did not rerun the Kubernetes integration; reported success is historical, not a new execution result.

Use a disposable Kubernetes cluster with at least one Ready, untainted worker, kubectl and python:3.11-slim-bookworm on its nodes. Save this as request-placement.py and run python3 request-placement.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11, and the selected worker reported ten allocatable CPUs.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

from decimal import Decimal, ROUND_CEILING

def cpu_quantity(value):
 return Decimal(value[:-1]) / 1000 if value.endswith('m') else Decimal(value)

namespace = 'notes-requests-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 workers = [n for n in obj(namespace, 'nodes')['items'] if
 not n['spec'].get('unschedulable') and not n['spec'].get('taints')
 and any(c['type'] == 'Ready' and c['status'] == 'True' for c in n['status']['conditions'])]
 assert workers, 'a Ready worker without taints is required'
 worker = workers[0]
 allocatable = cpu_quantity(worker['status']['allocatable']['cpu'])
 too_large = str(int(allocatable.to_integral_value(rounding=ROUND_CEILING)) + 1)
 for name, request in [('burst', '10m'), ('oversized', too_large)]:
 podspec = spec()
 podspec['nodeSelector'] = {'kubernetes.io/hostname': worker['metadata']['labels']['kubernetes.io/hostname']}
 podspec['containers'][0]['resources'] = {
 'requests': {'cpu': request, 'memory': '24Mi'}, 'limits': {'cpu': too_large, 'memory': '96Mi'}}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': name}, 'spec': podspec})
 burst = wait_for(lambda: obj(namespace, 'pod', 'burst'), pod_ready)
 assert burst['spec']['nodeName'] == worker['metadata']['name']
 def insufficient(p):
 return any(c['type'] == 'PodScheduled' and c['status'] == 'False'
 and c.get('reason') == 'Unschedulable' and 'Insufficient cpu' in c.get('message', '')
 for c in p.get('status', {}).get('conditions', []))
 pending = wait_for(lambda: obj(namespace, 'pod', 'oversized'), insufficient)
 assert pending['status']['phase'] == 'Pending' and not pending['spec'].get('nodeName')
 only_limit = spec()
 only_limit['containers'][0]['resources'] = {'limits': {'cpu': '200m', 'memory': '64Mi'}}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'defaulted'}, 'spec': only_limit})
 defaulted = wait_for(lambda: obj(namespace, 'pod', 'defaulted'), pod_ready)
 resources = defaulted['spec']['containers'][0]['resources']
 assert resources['requests'] == resources['limits'] == {'cpu': '200m', 'memory': '64Mi'}
 assert defaulted['status']['qosClass'] == 'Guaranteed'
 invalid = spec()
 invalid['containers'][0]['resources'] = {'requests': {'cpu': '200m'}, 'limits': {'cpu': '100m'}}
 denied = k('create', '-n', namespace, '-f', '-', payload={
 'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'invalid'}, 'spec': invalid}, expected=None)
 assert denied.returncode != 0 and 'must be less than or equal to cpu limit' in denied.stderr
 print(json.dumps({'node_allocatable_cpu': str(allocatable), 'limit_cpu': too_large,
 'small_request_with_large_limit_scheduled': True,
 'larger_than_node_request_pending': True,
 'limit_only_defaulted_resources': resources,
 'defaulted_qos': defaulted['status']['qosClass'],
 'request_above_limit_rejected': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script uses nodeSelector to target that worker while leaving the scheduler responsible for assignment. Both comparison Pods have an eleven-CPU limit. The first requests only ten millicpu and becomes Ready on the selected node. The second requests eleven CPUs, remains Pending without nodeName, and reports Unschedulable with Insufficient cpu. Neither Pod tries to consume eleven CPUs; their processes are idle.

A third Pod specifies only 200m CPU and 64Mi memory limits. The stored object contains identical requests, and its observed QoS class is Guaranteed. A fourth attempted Pod requests 200m while limiting CPU to 100m; the API rejects it. The original article reports that all assertions passed, and the namespace was removed.

This separates successful placement, scheduler rejection, defaulting and API validation with real objects. It does not test actual peak capacity, resource-manager exclusivity or every scheduling plugin. Size requests from workload evidence and review actual admitted demand alongside usage, instead of turning an idle chart or a familiar ratio into a placement guarantee.

Sources

Kubernetes: Resource Management, request-based placement, limits and admission defaulting, checked 2026-10-15.
Kubernetes: Pod Quality of Service Classes, Guaranteed requirements and the limits of QoS protection, checked 2026-10-15.

Kubernetes CronJobs need a history policy

Fri, 25 Feb 2028 00:00:00 +0000

A CronJob creates Jobs on a schedule, and each Job manages the Pods that perform the work. Finished objects can preserve useful status, but retention is bounded by several independent mechanisms. In particular, CronJobs already have default history limits. Omitting explicit history settings does not make ordinary completed runs accumulate forever.

Count and TTL do not guarantee minimum retention when the other permits earlier deletion. Cleanup is asynchronous, and active-job limits do not prove a business operation is safe to retry. The retained diagnostics were not rerun.

Separate history from log retention

The default CronJob history keeps up to three successful Jobs and one failed Job. The controller reconciles those categories independently. A zero limit requests no retained Jobs in that category, but cleanup is asynchronous rather than an exact completion-time deletion. Active Jobs are not completed-history entries and need separate attention.

Thirty days of hourly scheduling means 720 scheduled opportunities, not necessarily 720 retained Jobs or exactly 720 Pods. Missed schedules, concurrency policy, retries and parallelism change those relationships. A Job can own several Pods. State the assumptions before using run counts as storage or API-capacity estimates.

Keeping a Pod object also does not guarantee its logs remain available. Node failure, rotation, container cleanup and eviction affect log access. Export the operational evidence required by your retention policy to a suitable logging system. Kubernetes objects are useful investigation records, but they are not an archival guarantee.

Large retained histories still have costs: object storage, list sizes and investigation noise. A watch receives relevant changes rather than repeatedly emitting every unchanged object merely because it exists. Measure actual object counts and control-plane behavior instead of describing retained status as continuous activity.

Count and age are independent policies

The CronJob history limits constrain finished Job counts, not their minimum age. A busy schedule can replace the last failed record quickly, while a quiet schedule may keep it much longer. Choose counts from the inspection interval and failure frequency, rather than treating one retained failure as enough evidence for every workload.

TTL-after-finished is configured on the Job. For future CronJob-created Jobs, place ttlSecondsAfterFinished inside the Job template’s spec. The timer starts when a Job reaches Complete or Failed; expiry makes it eligible for cascading deletion. A zero TTL makes it eligible immediately, while an unset TTL supplies no age-based cleanup.

History cleanup and TTL can each remove a Job. Consequently, neither a large history count nor a long TTL guarantees minimum retention when the other mechanism permits earlier deletion. Finalizers and controller progress can delay physical removal. Editing a CronJob template affects future Jobs; it does not rewrite the policy of every existing Job.

Ownership determines cascading behavior

CronJob-created Jobs reference their owner, and Jobs reference their Pods through dependent ownership. Normal cascading deletion can remove that hierarchy. Explicit orphan deletion instead preserves dependents and removes the relevant ownership link; it does not intentionally leave an ordinary dangling reference for the collector to follow.

After orphaning Jobs, the former CronJob no longer trims their history. A configured TTL can still apply to each finished Job. Directly created Jobs also need an explicit retention decision. Check the actual deletion propagation policy and ownership rather than assuming every client uses identical defaults or that every orphan will soon disappear.

When it breaks

A Job that never reaches a terminal condition does not become eligible through finished-job TTL. History limits do not bound active runs either. An appropriate activeDeadlineSeconds can bound a Job’s runtime, while concurrencyPolicy controls overlapping schedules. These settings address different failure modes and do not prove the underlying operation is safe to retry.

Zero history can remove the failure record before anyone investigates it. Preserve required logs and result artifacts elsewhere before shortening retention. Conversely, a missing owner relationship or an absent TTL deserves investigation, not automatic deletion: another controller or retention requirement may deliberately own the cleanup decision.

Check it quickly

The following retained read-only diagnostics were not rerun in this review. Select a disposable test context and namespace. Compare the stored CronJob with the Jobs it actually owns, their completion conditions and creation times; namespace-wide totals below include unrelated workloads and active Jobs.

kubectl get cronjob hourly-rollup -o jsonpath='{.spec.successfulJobsHistoryLimit} {.spec.failedJobsHistoryLimit} {.spec.jobTemplate.spec.ttlSecondsAfterFinished}{"\n"}'
kubectl get jobs --sort-by=.metadata.creationTimestamp --no-headers | wc -l
kubectl get pods --field-selector=status.phase=Succeeded --no-headers | wc -l
kubectl get jobs -o jsonpath='{range .items[*]}{.metadata.name} ttl={.spec.ttlSecondsAfterFinished} owner={.metadata.ownerReferences[0].name}{"\n"}{end}'

The first line usually shows API-defaulted history values even when the manifest omitted them. Blank output is not proof of unlimited retention; check lookup errors and the complete object. The last line displays only the first owner reference. An absent TTL and owner mean these two particular mechanisms provide no demonstrated cleanup, not that deletion is impossible. Repeat inspection across completed runs, expecting bounded owned histories after reconciliation while active runs and unrelated objects are assessed separately.

Sources

Kubernetes Job, “Clean up finished jobs automatically” and the TTL mechanism for finished Jobs, checked 2026-10-15.
Kubernetes CronJob, default history limits and the job template, checked 2026-10-15.
Kubernetes garbage collection, owner references and cascading deletion, checked 2026-10-15.
Primary reference, TTL eligibility and finalizers, checked 2026-10-15.
Primary reference, log rotation and node-level retention, checked 2026-10-15.

cgroups enforce container budgets within a hierarchy

Fri, 11 Feb 2028 00:00:00 +0000

Kubernetes resource fields become useful constraints only when the node’s execution stack applies them. On Linux, kubelet and the container runtime use cgroups to organize processes and configure resource controllers. The result is a hierarchy, not merely one number attached to a Pod. Check the relevant container group and its inherited constraints before deciding what budget a process can actually consume.

Effective budgets follow the cgroup hierarchy. Local caps, relative CPU weights and memory accounting are different controls; the displayed numerical observations come from the retained historical source, not a new run.

Read the controller that enforces the resource

In cgroup v2, cpu.max expresses CPU bandwidth as quota and period in microseconds. A value of 10000 100000 permits one tenth of a CPU’s time per period for that group. It does not pin the process to one core or reduce the processor’s clock speed. Several runnable threads can consume a group’s allowance faster within the same period.

CPU weight serves a different purpose: it influences relative distribution under contention among eligible groups at the applicable hierarchy level. A Kubernetes CPU request can inform that weight, as well as scheduling placement. A request is not itself a hard runtime ceiling, and a weight does not guarantee an application’s wall-clock latency or exclusive ownership of a CPU.

The memory controller exposes limits and usage separately. memory.max describes the group’s hard memory boundary, while memory.current reports accounted use. Reclaim and OOM behavior are memory mechanisms, not CPU-style periodic throttling. Application heap size alone does not explain all accounted memory, so compare runtime memory metrics with the controller’s view when investigating a limit.

When it breaks

Hierarchical constraints can be tighter than one local file suggests. A child group with no local CPU cap can still be constrained by an ancestor. Container cgroup namespaces may hide parts of that hierarchy from the container’s view. Do not infer unrestricted node capacity from a local max value without checking the node-level context and effective parent constraints.

A multi-container Pod can have separate container limits. One container’s unused allowance is not automatically a transferable promise that another container may exceed its own cap. PodLevelResources is beta and enabled by default in Kubernetes 1.34, subject to component support and feature-gate configuration. Its shared Pod budget does not erase explicit container caps; it must not be silently substituted for per-container settings. The experiment below deliberately uses only container resources and confirms that distinction in the admitted Pod.

High throttling can explain latency even when a host-wide CPU average looks modest. Conversely, one throttling counter cannot identify every source of delay. Read changes over a relevant interval, correlate workload demand and request latency, and remember that ancestor contention and other resources can also matter. A short synthetic loop establishes enforcement, not an application capacity plan.

Controller files also depend on the environment. cgroup v1 uses different interfaces, and CPU-manager policies or runtime configuration can alter placement and enforcement details. Verify the actual node setup rather than copying a pathname and assuming its meaning. The fixture requires the cgroup v2 interface and reads settings without changing node controller files.

Check it quickly

The fixture and observations below are retained from the original article. This review did not rerun the Kubernetes integration; reported success is historical, not a new execution result.

Use a disposable Linux Kubernetes cluster with cgroup v2, kubectl and python:3.11-slim-bookworm available on the nodes. Save the script as cgroup-enforcement.py and run python3 cgroup-enforcement.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. Namespace creation and deletion permissions are required. The recorded server was Kubernetes 1.34.11 using the kind container runtime setup.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-cgroup-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 podspec = spec()
 podspec['containers'] = []
 for name, request, limit, memory in [('small', '25m', '100m', '64Mi'), ('large', '50m', '200m', '96Mi')]:
 container = spec()['containers'][0]
 container['name'] = name
 container['resources'] = {'requests': {'cpu': request, 'memory': '24Mi'},
 'limits': {'cpu': limit, 'memory': memory}}
 podspec['containers'].append(container)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'budgets'}, 'spec': podspec})
 wait_for(lambda: obj(namespace, 'pod', 'budgets'), pod_ready)
 snapshot = '''from pathlib import Path
import json,os
p=Path('/sys/fs/cgroup')
assert (p/'cgroup.controllers').exists(), 'this example requires cgroup v2'
print(json.dumps({'cpu_max':(p/'cpu.max').read_text().strip(),
 'cpu_weight':int((p/'cpu.weight').read_text()),
 'memory_max':int((p/'memory.max').read_text()),
 'group_inode':os.stat(p).st_ino}))
'''
 observations = {}
 for name, expected_cpu, expected_memory in [('small', 0.1, 64), ('large', 0.2, 96)]:
 data = json.loads(k('exec', '-n', namespace, 'budgets', '-c', name, '--', 'python3', '-c', snapshot).stdout)
 quota, period = map(int, data['cpu_max'].split())
 assert quota / period == expected_cpu
 assert data['memory_max'] == expected_memory * 1024 * 1024
 observations[name] = data
 assert observations['small']['group_inode'] != observations['large']['group_inode']
 load = '''from pathlib import Path
import json,time
p=Path('/sys/fs/cgroup/cpu.stat')
def counters(): return {k:int(v) for k,v in (line.split() for line in p.read_text().splitlines())}
before=counters(); begin=time.monotonic(); cpu=time.process_time(); value=0
while time.monotonic()-begin < 3: value=(value+1)%1000003
after=counters()
print(json.dumps({'wall_seconds':time.monotonic()-begin,'process_cpu_seconds':time.process_time()-cpu,
 'nr_throttled_delta':after['nr_throttled']-before['nr_throttled'],
 'throttled_usec_delta':after['throttled_usec']-before['throttled_usec']}))
'''
 throttling = json.loads(k('exec', '-n', namespace, 'budgets', '-c', 'small', '--', 'python3', '-c', load).stdout)
 assert throttling['nr_throttled_delta'] > 0 and throttling['throttled_usec_delta'] > 0
 final = obj(namespace, 'pod', 'budgets')
 assert pod_ready(final) and all(c['restartCount'] == 0 for c in final['status']['containerStatuses'])
 assert 'resources' not in final['spec']
 print(json.dumps({'container_groups': observations, 'measured_throttling': throttling,
 'container_restarts': 0, 'pod_level_resource_field_used': False}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

Two containers share one Pod but expose different cgroup directory identities. The smaller container reports cpu.max of 10000 100000 and memory.max of 67108864. The larger reports 20000 100000 and 100663296. Observed CPU weights are six and eleven; those values are runtime mappings, not portable constants that every installation must reproduce.

A three-second busy loop in the smaller container records thirty additional throttled periods and about 2.707 seconds of throttled time. Its process CPU time is about 0.298 seconds over 3.007 seconds elapsed. Both containers remain Ready with zero restarts. The original article reports that all assertions passed, and cleanup removed the namespace.

The test verifies distinct container settings and actual CPU throttling. It does not measure fairness between contending workloads, expose every ancestor setting, or exercise Pod-level resource allocation. Keep those boundaries explicit when turning low-level controller observations into operational conclusions.

Sources

Kubernetes: About cgroup v2, Linux resource-controller integration and prerequisites, checked 2026-10-15.
Kubernetes: Resource Management, container requests and limits passed to the runtime, checked 2026-10-15.
Linux: Control Group v2, hierarchy, CPU bandwidth, weights and memory accounting, checked 2026-10-15.
Kubernetes 1.34 Pod-level resources, feature state, support and shared Pod budgets, checked 2026-10-18.

Cordon and drain coordinate node maintenance

Thu, 03 Feb 2028 00:00:00 +0000

Cordon changes scheduling eligibility; drain coordinates eviction and waits for Pod termination. Neither command guarantees that work survives maintenance. Application shutdown, replacement capacity, storage and disruption budgets determine that outcome. Treat a drained node as an observed state with documented exceptions, rather than a promise inferred from one command name.

Cordon is a placement restriction, not an empty-node certificate. Drain is not atomic and does not prove successful writes or replacement. The retained diagnostics were not run in this review.

Cordon changes future placement

Cordon sets the node’s unschedulable field. Existing Pods continue running, and ordinary new scheduling avoids that node. Drain performs cordoning itself, so a separate cordon can create an earlier observation window but is not mandatory. Uncordoning permits scheduling again; it does not force immediate placement or repair an unhealthy kubelet.

The scheduling restriction has exceptions. DaemonSet Pods tolerate the unschedulable condition, and a Pod assigned directly through nodeName bypasses normal scheduler selection. Other Pods with the relevant toleration can also qualify. Check actual remaining workloads and avoid describing cordon as a firewall against every possible Pod arrival.

Drain has filters and independent operations

Kubectl first identifies Pods and checks deletion filters. Bare Pods or missing managing controllers can require force; emptyDir data requires acknowledgement of its loss; DaemonSet Pods require the ignore option. Mirror Pods are skipped because deleting their API representation cannot stop the underlying static workload managed by kubelet.

These preliminary checks are not an atomic transaction. Once eligible evictions begin, some can succeed while others wait or fail. Drain can also leave the node cordoned after failure. Re-read node and Pod state before retrying, rather than assuming an unsuccessful command means that nothing changed.

Ignore-daemonsets leaves DaemonSet Pods running. Delete-emptydir-data permits loss of those ephemeral contents; it does not migrate them. Force allows handling unmanaged Pods and does not bypass PodDisruptionBudgets. Confirm ownership, data needs and replacement behavior before choosing any of these options. Statelessness alone does not establish that removing a lone instance is harmless.

Budgets govern voluntary eviction

When supported, drain uses the Eviction API, which checks applicable disruption budgets. Direct deletion through disable-eviction bypasses that protection. The distinction matters: adding force cannot cure a budget rejection, while bypassing eviction can remove replicas whose continued operation is needed for quorum or availability.

A healthy Pod can be blocked when its budget allows no disruption. The default IfHealthyBudget policy also constrains eviction of Running but unhealthy Pods when the guarded application is already disrupted; it does not invariably require every unhealthy Pod to become Ready. AlwaysAllow changes that unhealthy-Pod decision. Select the policy deliberately from application recovery behavior.

Drain waits for termination and retries some rejected evictions. Its timeout defaults to unlimited, but authentication errors, server failures and other conditions can still end the command. Graceful termination gives the application a window to stop; it cannot prove completed writes or successful replacement. A finite operational deadline helps expose blocked maintenance.

When it breaks

A zero-disruption budget is a useful lead, not permission to weaken the budget immediately. Inspect the selected Pods, desired and current healthy counts, replacement capacity and storage topology. Restoring an unhealthy replica or adding safe capacity may be preferable to reducing protection. Concurrent drains compete for the same budgets and can intensify the problem.

If maintenance cannot proceed, identify the exact remaining Pod and reason. Do not substitute disable-eviction simply to obtain a successful terminal result. Check application health and remaining static or DaemonSet workloads before powering down; a successful ordinary drain deliberately leaves some categories behind.

Check it quickly

These retained commands were not run in this review. Use a disposable, explicitly selected cluster and the intended namespace. The client dry run still reads cluster state, but neither performs evictions nor validates whether the server will admit them. Its ignore and emptyDir flags suppress those particular preflight refusals.

kubectl get node node-1 -o jsonpath='{.spec.unschedulable}{"\n"}'
kubectl drain node-1 --dry-run=client --ignore-daemonsets --delete-emptydir-data
kubectl get pods --field-selector spec.nodeName=node-1 -o custom-columns=NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind
kubectl get pdb -A -o custom-columns=NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,ALLOWED:.status.disruptionsAllowed

The node field shows cordon state. The Pod listing covers only the current namespace and displays only the first owner reference; neither an empty owner column nor a displayed owner proves the complete controller relationship. Compare all namespaces and the controlling reference before acting. The budget listing is cluster-wide but omits namespaces and selectors. Match each budget to the actual workload, then verify a real, authorized drain separately; this inspection cannot certify its success.

Sources

Safely Drain a Node, the drain workflow and its interaction with PodDisruptionBudgets, checked 2026-10-15.
kubectl drain reference, the exact refusal conditions and flags, checked 2026-10-15.
Pod disruptions, voluntary versus involuntary disruption and eviction retries, checked 2026-10-15.
Primary reference, unhealthy-Pod eviction policies, checked 2026-10-15.

The Horizontal Pod Autoscaler depends on metric semantics

Wed, 22 Dec 2027 00:00:00 +0000

A HorizontalPodAutoscaler turns configured metrics into replica recommendations. Choosing the metric and its aggregation is therefore part of application design. Increasing replicas should reduce the measured pressure or increase useful throughput; otherwise the controller may spend more money without relieving the bottleneck. Set explicit minimum and maximum counts, but do not confuse a cost bound with a performance guarantee.

Arithmetic model from the source, not a live observation. Its historical Python-only pass is unchanged. The model omits real-controller tolerance, readiness, stabilization and rate policies; no cluster behavior or availability is established.

Separate measurement from control

HPA normally reconciles periodically, with a default controller sync period of fifteen seconds. For per-Pod average targets, its basic calculation is the ceiling of current replicas multiplied by current metric divided by target metric. CPU utilization is relative to resource requests, rather than the CPU limit or the entire node’s capacity.

Averaging across Pods is not the same as averaging a time series. Metrics Server reports CPU usage measured over a window; custom adapters define their own measurement and query semantics. HPA does not universally smooth every input over a configurable measurement window. Tolerance, readiness handling, stabilization, and rate policies are separate controller mechanisms layered around metric recommendations.

Check the unit before the target number. Per-Pod custom metrics use AverageValue; object and external metrics can use Value or AverageValue with different scaling meaning. A total queue depth and an average backlog per replica are not interchangeable. Resource utilization additionally needs the relevant resource requests. An unsuitable metric can return a perfectly valid number while still measuring the wrong bottleneck.

Pick a signal replicas can improve

CPU often works for CPU-bound services, but availability of the resource metrics API is a prerequisite, not a universal cluster property. Workers blocked on a database can have low CPU and high latency. More workers might make the database problem worse. Test whether the chosen target corresponds to sustainable work per replica before relying on autoscaling.

Request rate, in-flight work, and consumer lag each require explicit scope. For a per-Pod metric, expose the necessary per-Pod observations rather than collapsing everything into one workload series. For an object or external signal, aggregate to the intended workload or queue and exclude unrelated traffic. Avoid unnecessary per-user labels that increase adapter query cost and complicate selection.

Give replica ownership one home

HPA reads the target’s scale information and adjusts its replica count. A Deployment’s initial replicas value is not the HPA minimum; minReplicas is the configured floor. Reapplying a manifest that continually writes a fixed replica count can compete with HPA. Arrange configuration ownership deliberately, including what static count should apply if autoscaling is removed.

When it breaks

An unavailable custom metric does not automatically fall back to CPU. With multiple configured metrics, HPA chooses the largest usable recommendation. If one metric fails and the usable metrics recommend shrinking, it skips that scale-down. A valid recommendation above the current count can still permit scale-up. Thus explicitly adding CPU offers another input, not unconditional failover in both directions.

Distinguish a failed metric source from missing measurements for individual Pods. The latter also causes conservative adjustments in the resource algorithm. Inspect conditions and events rather than asserting that every missing sample must set ScalingActive=False. Successful readings, failed sources, and current bounds can coexist.

Stale adapter data can drive scaling after the relevant work has drained. Inspect timestamps, query scope, caching, and collection latency before retuning the target. A metric API error and a valid-but-irrelevant memory reading are different failures; choosing memory as a poor latency proxy does not itself make the reading Unknown.

Check it quickly

Save this as hpa_metric_sim.py and run python3 hpa_metric_sim.py. The standard-library model uses already-computed per-replica averages and starts inside its bounds. It demonstrates maximum selection and the failed-source scale-down guard. It is not an implementation of every HPA metric type.

import math
import sys


def desired_for(current, value, target):
 assert current > 0 and target > 0
 if value is None:
 return None
 assert value >= 0
 return math.ceil(current * value / target)


def decide(current, metrics, min_replicas, max_replicas):
 assert 0 < min_replicas <= current <= max_replicas
 recommendations = [desired_for(current, value, target)
 for _, value, target in metrics]
 available = [want for want in recommendations if want is not None]
 if not available:
 return current
 want = max(available)
 if None in recommendations and want < current:
 return current
 return max(min_replicas, min(max_replicas, want))


def check(actual, expected, message):
 assert actual == expected, (message, actual, expected)
 print(f"OK: {message} -> {actual}")


def main():
 current = 4
 cpu = ("cpu", 80, 50)
 queue = ("average_queue", 900, 300)
 missing_queue = ("average_queue", None, 300)
 cool_cpu = ("cpu", 10, 50)
 check(decide(current, [cpu, queue], 1, 20), 12, "maximum wins")
 check(decide(current, [cpu, missing_queue], 1, 20), 7, "missing allows scale-up")
 check(decide(current, [cool_cpu, missing_queue], 1, 20), 4, "missing blocks scale-down")
 check(decide(current, [("cpu", None, 50), missing_queue], 1, 20), 4, "all missing holds")
 check(decide(current, [cpu, queue], 1, 10), 10, "upper bound")
 check(decide(current, [cool_cpu, ("average_queue", 30, 300)], 1, 20), 1, "valid low metrics shrink")
 check(decide(current, [cool_cpu], 2, 20), 2, "lower bound")
 print("PASS: 7 model scenarios; no Kubernetes controller was exercised")
 print(sys.version.split()[0])


if __name__ == "__main__":
 main()

The seven assertions produce recommendations of twelve and seven for the hot inputs, hold four when a failed source accompanies a shrink request, and hold four when all sources fail. They also check both bounds and a valid shrink to one. Both language copies passed locally with Python 3.12.14 during this review.

No cluster controller, metrics adapter, tolerance, readiness adjustment, stabilization window, or scaling-rate policy is exercised. The model’s explicit missing-source guard is its learning target. Validate real timestamps, conditions, and scaling events separately before treating arithmetic success as evidence that production autoscaling works.

Sources

Kubernetes Horizontal Pod Autoscaling, documents algorithm and metric sources, checked 2026-10-15.
Resource Metrics Pipeline, documents metrics-server and the resource metrics API, checked 2026-10-15.
HorizontalPodAutoscaler walkthrough, Pods metrics use AverageValue; object/external types differ, checked 2026-10-15.

A CronJob turns a schedule into Jobs

Fri, 17 Dec 2027 00:00:00 +0000

A CronJob combines a Job template with a recurring schedule. The CronJob controller creates Jobs; each Job manages the Pods that perform the work. A run can involve retries, replacement Pods, or parallel workers, depending on the Job specification. There need not be a dedicated application process waiting between runs, although completed Jobs and Pods may remain for inspection.

Scheduling is approximate. Duplicate Jobs and missed occurrences are both possible, so do not promise unconditional at-least-once delivery for every scheduled time. Make repeated execution safe and record which business periods actually completed. Idempotency prevents duplicate effects; a separate completeness check detects work that never ran.

Scheduling is approximate, not a durable queue or a promise that every tick completes. startingDeadlineSeconds limits start lateness, not run duration. Source commands were not rerun; example execution remains blocked.

A schedule makes a Job

The five schedule fields describe minute, hour, day of month, month, and day of week. Ranges, steps, and macros such as @daily are supported. */10 * * * * requests ten-minute intervals; it does not guarantee that a Pod starts precisely on those boundaries. Resource shortages and image pulls can delay execution after Job creation.

Set spec.timeZone explicitly when wall-clock interpretation matters. Without it, the controller manager’s local time zone applies; moving machines only changes behavior if that configuration changes. Use the timeZone field rather than putting TZ or CRON_TZ inside the schedule. Daylight-saving transitions and application reporting periods still need a deliberate policy.

Keep the CronJob name within 52 characters because generated Job names need an appended suffix. Since Kubernetes 1.32, generated Jobs carry batch.kubernetes.io/cronjob-scheduled-timestamp. Compare that intended time with creation and start timestamps to investigate delays. The annotation helps diagnosis; it is not the sole possible audit trail or evidence that the business operation finished.

Concurrency is scoped to one CronJob

The default Allow policy permits overlapping Jobs. Forbid declines a new start while an earlier Job remains active; the skipped time counts as missed, and a later eligible start still depends on its deadline. Replace requests deletion before creating a replacement; asynchronous deletion does not prove the old Pods have stopped. These policies do not coordinate separate CronJobs or external workers touching the same data.

A task consistently longer than its interval can fail to complete under Replace. Forbid trades overlap avoidance for missed starts, rather than providing a durable queue. Use application-level deduplication or coordination where overlapping effects would be harmful, including interruption during replacement. A lock alone does not prove that a partially committed operation can safely resume.

Deadlines, suspension and history

startingDeadlineSeconds limits the lateness of a start, not execution duration. Without it, occurrences have no individual starting deadline. The concept guide describes refusing catch-up beyond 100 missed schedules. However, the reviewed 1.34.0 scheduling helper emits a warning and returns the most recent eligible time; it does not reject solely for that count. Verify the server version instead of inferring a permanent stop from the warning. A configured deadline limits the counting window, and values below ten seconds can miss controller checks.

Suspension stops new Jobs without stopping existing ones. Suspended schedule times count as missed, so resuming can trigger immediate catch-up subject to the other rules. Do not treat that as a promise to replay one Job for every missed tick. Keep a durable work ledger if every reporting period must be processed.

History limits retain completed Job objects, with defaults of three successes and one failure. They do not cap concurrent running Jobs or replace execution deadlines. Updating the template affects future Jobs; already-started Jobs retain their own specifications.

When it breaks

A twenty-five-minute task on a ten-minute schedule can overlap under Allow and duplicate writes. Inspect actual ownership, active Pods, and completion records before attributing duplicate business effects to a schedule. A missed-start deadline should reflect how late the result remains useful, rather than an arbitrary attempt to hide controller warnings.

A long gap in lastScheduleTime can also be intentional: suspension, Forbid, a sparse calendar expression, or a changed time zone may explain it. Events expire, and successful Job creation does not establish successful backups. Alert on the last verified usable result as well as scheduling health.

Check it quickly

In the intended read-only context and namespace, inspect an existing nightly-report. These commands were retained from the earlier article and were not rerun against a cluster in this review.

kubectl get cronjob nightly-report -o jsonpath='{.spec.schedule} {.spec.concurrencyPolicy} {.spec.startingDeadlineSeconds}{"\n"}'
kubectl get cronjob nightly-report -o jsonpath='{.status.lastScheduleTime}{"\n"}'
kubectl get jobs --sort-by=.metadata.creationTimestamp -o custom-columns=NAME:.metadata.name,START:.status.startTime,SUCCEEDED:.status.succeeded
kubectl get events --field-selector involvedObject.kind=CronJob --sort-by=.lastTimestamp | tail -5

The Jobs listing includes unrelated Jobs in the namespace; correlate owner references before drawing conclusions. START timestamps alone cannot prove overlap, and an empty SUCCEEDED field does not prove a Job is still running. Check active/failed counts and conditions alongside each scheduled timestamp. Interpret gaps against the actual calendar and policy, and use controller logs when retained Events lack the explanation.

Sources

Kubernetes CronJob, schedule syntax, concurrency policy, the 100-missed-schedule rule and history limits, checked 2026-10-15.
CronJob v1 API reference, field-level definition of the CronJob spec, checked 2026-10-15.
Kubernetes 1.34.0 CronJob scheduling helper, mostRecentScheduleTime and nextScheduleTime select the latest missed time and warn above 100, checked 2026-10-15.
CronJob job control (v1.34.0), DeleteJob uses background propagation; replacement is not a Pod-termination barrier, checked 2026-10-18.

Node affinity places workloads on matching nodes

Thu, 09 Dec 2027 00:00:00 +0000

Node affinity matches workload requirements to node labels. It can require NVMe storage or prefer a newer instance family, but it does not replace the scheduler’s other checks. Even without affinity, scheduling filters resource requests, taints and volume constraints before scoring feasible nodes. It is not simply a comparison of idle capacity.

GPU placement illustrates the distinction. A device-plugin GPU resource request accounts for available devices; an accelerator label can select a particular model. Labels alone allocate no GPU, and a toleration alone supplies neither the device nor a placement requirement.

Required affinity filters by labels. Preferred affinity contributes to ranking among feasible nodes; neither alone guarantees placement.

Match nodes by label

Labels such as accelerator=nvidia-a100 and storage=nvme describe properties that provisioning must establish and maintain. A replacement node does not automatically inherit arbitrary labels from its predecessor. Verify the new hardware before applying the old label; otherwise a formally matching rule can select an unsuitable machine.

Treat the label vocabulary as an API. Renaming a key affects every referring manifest, while stale values can misrepresent hardware. Validate manifests against the intended node inventory, including capacity planned but not yet online. Absence of a label may reflect provisioning or a temporarily missing pool, rather than a typo alone.

For isolation, label integrity matters. Kubernetes 1.34 documents the Node authorizer plus NodeRestriction admission plugin and protected node-restriction.kubernetes.io/ label keys. An ordinary self-reported compliance label is not an independent security boundary. Confirm who can change labels and workload placement requirements.

Prefer or require

requiredDuringSchedulingIgnoredDuringExecution filters out nodes that fail the rule. preferredDuringSchedulingIgnoredDuringExecution adds weighted preferences among feasible nodes; other scores can win. Neither guarantees spare capacity, a healthy application or resilience. A hard requirement may leave a Pod Pending, while a soft one permits a nonmatching node only if all hard constraints pass.

The names also specify when rules apply. A later label change does not evict an already-bound Pod under IgnoredDuringExecution, and changing a preference does not automatically redistribute existing replicas. Inspect new scheduling decisions separately from the current placement inventory.

When both nodeSelector and required node affinity exist, both must match. Alternative nodeSelectorTerms are ORed; matchExpressions inside one term are ANDed. Review that grouping before blaming capacity, because an accidental conjunction can eliminate the entire intended pool.

Anti-affinity splits replicas

Pod anti-affinity matches other Pods within a selected namespace scope and topology domain. kubernetes.io/hostname represents nodes; the standard zone key is topology.kubernetes.io/zone. All relevant nodes need consistent labels. Admission policy can restrict required anti-affinity topology keys, so a zone rule is not universally accepted.

With three mutually selected replicas, required hostname separation and only two otherwise eligible nodes, two replicas may run while the third remains Pending. That is different from preventing the whole workload from running. Preferred anti-affinity can allow co-location and does not promise nearly the same resilience. Choose the rule from the application’s failure tolerance, eligible domains and replacement capacity.

When it breaks

For a Pending Pod, inspect the complete scheduler event and admitted specification. A label mismatch is one possibility; insufficient requested resources, untolerated taints, volume topology or scheduler-profile constraints are others. Low observed CPU use does not establish request headroom. Fix the constraint actually reported rather than immediately weakening affinity.

For an unexpected bound Pod, compare its node’s labels with the rule that applied when it was scheduled. Check whether the rule was preferred, whether labels subsequently changed, and whether the Pod bypassed ordinary scheduling through nodeName. Do not infer that the scheduler migrated an existing GPU workload because the current inventory differs from intent.

Check it quickly

The retained commands below are an unexecuted, abbreviated starting point:

kubectl get nodes -L accelerator,workload-class --no-headers
kubectl describe pod api-0 | grep -A 8 "Node-Selectors\|Affinity"
kubectl get pods -o wide -l app=api | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head

The grep view can omit affinity details. The final pipeline includes unique Pod names and a header, so it does not aggregate replicas per node; fixed column positions are also fragile. Inspect the full admitted affinity and nodeSelector fields, actual nodeName values and selected peer Pods, then count topology domains explicitly. These commands do not prove placement or resilience. The diagram records an earlier graphics-only review. Live scheduling remains unverified.

Sources

Kubernetes node affinity docs, node affinity semantics, checked 2026-06-10.
Kubernetes taints and tolerations, documents repelling pods from specialized nodes, checked 2026-09-05.
Kubernetes 1.34 placement rules, label trust, grouping and scheduling-time affinity, checked 2026-10-18.
Kubernetes scheduler, filtering and scoring feasible nodes, checked 2026-10-18.
Schedule GPUs, device resource requests versus GPU model labels, checked 2026-10-18.

Kubernetes rolling updates replace pods gradually

Sat, 27 Nov 2027 00:00:00 +0000

Choose the release shape before its speed. Can old and new versions process requests together? Can they share storage and ownership safely? What happens if the new version cannot start? Those answers determine whether a Deployment should overlap versions with RollingUpdate or stop the old version before replacement with Recreate. The companion article handles rollout pacing and live observation.

Choose version overlap only after compatibility and ownership checks. Recreate changes upgrade ordering; rollback still needs a tested image, configuration and data recovery plan.

Replace gradually or stop first

RollingUpdate is the default Deployment strategy. Its surge and unavailable budgets control replacement while allowing old and new ReplicaSets to coexist. Readiness contributes to availability accounting, but does not establish application correctness or guarantee that every request succeeds.

Recreate waits for old Pods to terminate during a Deployment upgrade before creating the new version. Plan an interval with no serving replicas. Remove the rollingUpdate configuration when selecting this strategy; those fields belong only to RollingUpdate. Measure shutdown and startup rather than assuming a thirty-second window.

Compatibility is the deciding constraint

Overlap requires compatible APIs, stored data, and background work. A frontend can look healthy while an old worker misreads records written by the new worker. Test mixed-version reads and writes, job ownership, and retry behavior before adjusting replica counts. A rollout budget cannot repair an incompatible schema change.

Stateful does not automatically mean Recreate: a properly coordinated replicated application may support overlapping versions. Conversely, Recreate is not a distributed lock. Its ordering applies to Deployment upgrades; a ReplicaSet may replace a manually deleted Pod while that Pod is still terminating. Singleton correctness needs application coordination and, where required, fencing that prevents an obsolete owner from writing.

Storage terminology also matters. ReadWriteOnce restricts writable mounting to one node, not necessarily one Pod. Multiple Pods on that node may access the volume. ReadWriteOncePod provides single-Pod access for supported CSI volumes. Check the actual driver and workload requirements rather than using a storage label as proof of exclusive ownership.

Rollback is another change

Keep a restartable previous image and revision history. kubectl rollout undo restores a recorded Pod template; changing only the image may miss changed arguments or environment references. Neither operation reverses a database migration or restores the contents of an independently edited ConfigMap. An old template referencing the same ConfigMap name can still receive new data.

Before release, record which image, configuration, and data versions work together. Prefer backward-compatible migrations when overlap or rollback is required. Rehearse the recovery sequence and its downtime; retaining a ReplicaSet alone does not make that sequence safe. With Recreate, returning to the old version can entail another stop/start interval.

When it breaks

Uncoordinated workers can duplicate jobs or corrupt data while every Pod is Ready. Investigate ownership and compatibility, not only container health. A slow image pull or storage attachment can stretch a Recreate outage long beyond the measured application startup time. Include a genuinely cold start and shutdown tail in the rehearsal.

For four replicas, maxUnavailable: 50% permits a reduction to two available replicas, not automatically one. A failed replacement can stall progress; an independent failure can reduce capacity further. Determine whether two instances can carry the expected load before accepting that budget.

A zero unavailable budget still cannot prevent unrelated failures. At the other extreme, allowing the whole replica count to be unavailable is legal, although often undesirable. Treat rejecting that setting as an organizational policy, not a Kubernetes API rule. Both surge and unavailable cannot be zero.

Check it quickly

Save this as strategy_window.py and run python3 strategy_window.py. This standard-library model accepts positive replica counts and nonnegative integer budgets only. It calculates planning quantities, not a controller simulation or a complete API validator. A scaled-to-zero Deployment is legal Kubernetes state, deliberately outside this active-rollout exercise.

"""Integer rollout-budget arithmetic; no cluster or traffic is exercised."""


def rolling_budget(replicas, surge, unavailable):
 values = (replicas, surge, unavailable)
 if any(type(value) is not int for value in values):
 raise ValueError("this model accepts integers only")
 if replicas <= 0 or surge < 0 or unavailable < 0:
 raise ValueError("positive replicas and nonnegative budgets required")
 if surge == 0 and unavailable == 0:
 raise ValueError("surge and unavailable cannot both be zero")
 planned_floor = max(0, replicas - unavailable)
 target_ceiling = replicas + surge
 return planned_floor, target_ceiling


def main():
 assert rolling_budget(6, 1, 1) == (5, 7)
 assert rolling_budget(4, 0, 2) == (2, 4)
 assert rolling_budget(4, 1, 4) == (0, 5)
 for invalid in [(4, 0, 0), (4, -1, 1), (0, 1, 0)]:
 try:
 rolling_budget(*invalid)
 except ValueError:
 pass
 else:
 raise AssertionError(f"invalid model input accepted: {invalid}")
 print("rolling: planned availability floor=5, replica target ceiling=7")
 print("full-unavailability budget: planned floor=0 (permitted)")
 print("recreate: expect a zero-serving interval during replacement")
 print("PASS: budget arithmetic only; runtime availability untested")


if __name__ == "__main__":
 main()

Expect four output lines ending in the arithmetic-only PASS message. Six/one/one yields five and seven; the full-unavailability case deliberately yields zero. Terminating Pods, independent failures, scheduling, and request handling are outside the model, so neither number is an observed runtime bound.

The assertions were executed with Python 3.12.14 in Linux during this revision. No Kubernetes integration exercise was executed. Before adopting the strategy, rehearse an upgrade and rollback in staging with continuous requests, storage checks, and mixed-version business tests. Record their results separately from this local arithmetic check.

Sources

Kubernetes Deployments, upgrade ordering and revision rollback, checked 2026-10-15.
Deployment API, strategy fields and rollout budgets, checked 2026-10-15.
Kubernetes persistent volumes, node-level and Pod-level access modes, checked 2026-10-15.
Kubernetes ConfigMaps, configuration references and propagation, checked 2026-10-15.
Kubernetes StatefulSet deletion considerations, identity and duplicate-process risks, checked 2026-10-15.

Security context defines constraints that need runtime evidence

Sat, 20 Nov 2027 00:00:00 +0000

A Kubernetes securityContext configures process identity and selected operating-system constraints. It can reduce what a compromised application is permitted to do, but it does not establish a complete security boundary by itself. Start with the operations the application actually needs, configure the appropriate fields, and test both a permitted operation and a forbidden one in the resulting container.

Source-described Kubernetes 1.34.11 fixture, not rerun. Results reflect the combined settings; RuntimeDefault is runtime-dependent. No container escape resistance, exhaustive seccomp coverage, hostile multi-tenant isolation or application authorization was tested. Execution remains blocked.

Put each setting at the right scope

Pod-level settings can supply user and group identity, shared-volume group ownership behavior and a seccomp profile. Container-level settings can override applicable defaults and configure capabilities, privilege escalation and a read-only root filesystem. Check the field’s actual API scope; placing a familiar name under the wrong securityContext does not create an equivalent constraint.

runAsUser selects a numeric user, while runAsNonRoot requires the container to satisfy a non-root policy. These settings complement image metadata rather than trusting a username’s appearance. Volume permissions must also permit the selected identity to perform required work. For an emptyDir used below, fsGroup supplies group access so the application can write its designated working directory.

Linux capabilities divide privileges that would otherwise be associated broadly with root. Dropping ALL removes those capabilities from the container’s configured set. Standard containers do not inherit every Linux capability by default, and a container running as root is not automatically proof of a successful host escape. Review the actual effective privileges rather than using either shortcut.

When it breaks

allowPrivilegeEscalation false sets the no_new_privs constraint for the process. It restricts gaining privileges through executing programs, such as set-user-ID or file-capability transitions. It does not revoke every privilege the process already has or replace application authorization. Inspect the process flag and effective capabilities; do not attribute every denied operation to this one field.

readOnlyRootFilesystem makes the container’s root filesystem read-only. Other mounts can remain writable, including explicitly supplied working volumes. This does not imply a blanket no-execution policy: an interpreter can read and run code stored in a writable volume. It also does not stop network requests or remove access to already mounted data. Read-only storage is one constrained operation, not proof that a compromise cannot progress.

Seccomp filters system calls, but RuntimeDefault depends on the runtime’s supplied profile. A process reporting seccomp filter mode confirms an active filtering mechanism, not that every undesirable syscall or attack path is blocked. Ordinary Linux runtimes can enforce these fields; a special sandbox is not required merely for the settings to have an effect, though stronger isolation has separate design considerations.

Conflicting identity settings can pass API admission but prevent container startup. An image may also expect to write caches or temporary files under its root filesystem. Diagnose startup state and denied paths, then provide only the working storage the application requires. Avoid solving every permission failure by restoring root, privilege escalation or a writable root filesystem.

Check it quickly

The fixture and observations below are retained from the original article. This review did not rerun the Kubernetes integration; reported success is historical, not a new execution result.

Use a disposable Linux Kubernetes cluster, kubectl and python:3.11-slim-bookworm available on its nodes. Save the script as security-context.py and run python3 security-context.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11 using kind’s standard runtime configuration.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-security-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 probe = '''from pathlib import Path
import errno,json,os,subprocess,time
status=dict(line.split(':',1) for line in Path('/proc/self/status').read_text().splitlines() if ':' in line)
try:
 Path('/tmp/forbidden-write').write_text('x')
 raise AssertionError('root filesystem unexpectedly writable')
except OSError as error:
 assert error.errno==errno.EROFS
try:
 os.setuid(0)
 raise AssertionError('unexpected UID escalation')
except OSError as error:
 assert error.errno==errno.EPERM
work=Path('/work'); (work/'data').write_text('writable')
(work/'example.py').write_text("print('interpreted from the writable volume')")
ran=subprocess.check_output(['python3',str(work/'example.py')],text=True).strip()
result={'uid':os.getuid(),'gid':os.getgid(),'cap_eff':int(status['CapEff'].strip(),16),
 'no_new_privs':int(status['NoNewPrivs']),'seccomp':int(status['Seccomp']),
 'root_write_errno':errno.EROFS,'uid_zero_denied':True,
 'volume_value':(work/'data').read_text(),'writable_volume_code':ran}
assert result['uid']==result['gid']==10001
assert result['cap_eff']==0 and result['no_new_privs']==1 and result['seccomp']==2
print(json.dumps(result),flush=True); time.sleep(600)
'''
 podspec = spec(probe)
 podspec['securityContext'] = {'runAsNonRoot': True, 'runAsUser': 10001,
 'runAsGroup': 10001, 'fsGroup': 10001, 'seccompProfile': {'type': 'RuntimeDefault'}}
 podspec['volumes'] = [{'name': 'work', 'emptyDir': {}}]
 c = podspec['containers'][0]
 c['securityContext'] = {'allowPrivilegeEscalation': False, 'readOnlyRootFilesystem': True,
 'capabilities': {'drop': ['ALL']}}
 c['volumeMounts'] = [{'name': 'work', 'mountPath': '/work'}]
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'restricted'}, 'spec': podspec})
 wait_for(lambda: obj(namespace, 'pod', 'restricted'), pod_ready)
 observed = json.loads(wait_for(lambda: k('logs', '-n', namespace, 'restricted').stdout.strip(), bool))
 conflicting = spec()
 conflicting['securityContext'] = {'runAsNonRoot': True, 'runAsUser': 0}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'conflict'}, 'spec': conflicting})
 refused = wait_for(lambda: obj(namespace, 'pod', 'conflict'), lambda p:
 any(c.get('state', {}).get('waiting', {}).get('reason') == 'CreateContainerConfigError'
 for c in p.get('status', {}).get('containerStatuses', [])))
 waiting = refused['status']['containerStatuses'][0]
 assert 'runAsUser breaks non-root policy' in waiting['state']['waiting']['message']
 assert waiting['restartCount'] == 0
 print(json.dumps({'runtime_observation': observed,
 'conflicting_uid_accepted_by_api_but_container_start_refused': True,
 'container_escape_resistance_tested': False}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The restricted Pod reports UID and GID 10001, effective capabilities zero, NoNewPrivs one and seccomp mode two. Writing under the root filesystem’s /tmp fails with EROFS, and changing to UID zero fails with EPERM. Those results reflect the combined identity and privilege settings, not an isolated test of each field’s individual effect.

The process successfully writes a value under /work and invokes Python on a file it wrote there. This deliberately demonstrates the allowed working-volume behavior and the limit of the read-only-root claim. A second Pod combines runAsNonRoot with runAsUser zero: the API accepts it, but kubelet reports CreateContainerConfigError before the container starts.

The original article reports that all assertions passed, and the namespace was removed. The fixture verifies these concrete runtime constraints and startup failure. It does not test container escape resistance, every seccomp rule, hostile multi-tenancy or the application’s authorization model.

Sources

Kubernetes: Security Context, Pod and container security fields and runtime effects, checked 2026-10-15.
Linux: No New Privileges Flag, restrictions on privilege gains through exec, checked 2026-10-15.
Linux: capabilities(7), separate capability privileges and process capability sets, checked 2026-10-15.

The horizontal autoscaler scales on demand

Thu, 04 Nov 2027 00:00:00 +0000

The HorizontalPodAutoscaler adjusts a scalable workload toward configured metric targets. Its output is a desired replica count, not a guarantee that additional Pods are already serving traffic. Scheduling, image startup, initialization, readiness, and available node capacity determine how quickly that recommendation becomes useful capacity. Design the control loop together with those delays.

Source arithmetic and list-state model only. Historical Python-only execution remains passed; it did not exercise tolerance, rate policies, per-Pod handling, Kubernetes reconciliation or useful ready capacity. No new run was performed.

Metrics produce recommendations

For a simple per-Pod average, desired replicas are the ceiling of current replicas multiplied by current metric divided by target metric. A ratio of two suggests doubling before additional controls apply. A ratio below one can suggest shrinking, but tolerance, missing measurements, readiness handling, bounds, and scaling behavior can prevent or limit the action.

CPU utilization uses resource requests as its denominator. Changing requests can change utilization without changing consumed CPU, so coordinating vertical and horizontal scaling matters. The metrics API must be available and the target must support the scale subresource. HPA changes a Deployment’s count; the Deployment controller manages its ReplicaSets and Pods.

The default reconciliation interval is fifteen seconds, but that is not an end-to-end response promise. Metrics have their own collection windows and delays. Distinguish average usage across Pods from temporal averaging in the measurement pipeline, and distinguish both from stabilization of recommendations. A stale but valid number can mislead an otherwise healthy controller.

Stabilize recommendations, not the last resize

Downscale stabilization retains recent replica recommendations and selects the highest within the configured window. The default downscale window is five minutes; autoscaling/v2 behavior can customize it. This is not a simple cooldown measured from the last replica change. A high recommendation that left the count unchanged can still postpone a later shrink.

Rate policies separately bound how many Pods or what percentage may change over a period. Scale-up can therefore be limited too; it is not necessarily immediate or unlimited. Set a realistic floor for startup latency and a maximum consistent with capacity and cost. Stabilization reduces churn but does not compensate for a target that tracks the wrong bottleneck.

Choose pressure that extra replicas relieve

CPU can be useful for compute-bound requests, while queue depth or work per replica may better describe workers. Neither latency nor an arbitrary resource graph automatically makes a good control input. If all replicas wait on a saturated shared database, multiplying them can increase contention instead of reducing delay.

Evaluate what happens after one replica is added: can work move to it, does per-replica pressure fall, and does downstream capacity remain sufficient? Long-lived connections, uneven partitions, and cache warmup can all delay the intended effect. A controller cannot redistribute an application’s work simply by changing a number.

When it breaks

Missing metrics are unknown demand, not zero demand. With multiple metrics, a valid scale-up recommendation can still be used even if another source fails; a failed source prevents a shrink suggested by the remaining sources. Missing samples for individual Pods also receive conservative treatment. Use conditions and events to distinguish these cases instead of assuming every missing sample stops every action.

A maxReplicas ceiling can leave queues growing while ScalingLimited reports that the calculated recommendation was constrained. A high minimum can hide low demand. Repeated fixed replicas writes from deployment automation can fight HPA. Also inspect CPU request changes and startup readiness before blaming normal rolling updates: HPA supports Deployment rollouts and does not simply count every initializing Pod as zero-load capacity for unrestricted shrink.

Check it quickly

Save this as hpa_loop.py and run python3 hpa_loop.py. It is a standard-library model of ratio arithmetic, bounds, and a highest-recent-recommendation downscale window. It deliberately omits tolerance, rate policies, individual Pod samples, and live Kubernetes reconciliation.

import math

TARGET = 50.0
MIN_REPLICAS = 2
MAX_REPLICAS = 10
WINDOW_S = 300


def desired(replicas, value):
 raw = math.ceil(replicas * value / TARGET)
 return max(MIN_REPLICAS, min(MAX_REPLICAS, raw))


def step(replicas, history, now, value):
 if value is None:
 print(f"t={now}s metric=missing hold={replicas}")
 return replicas
 recommendation = desired(replicas, value)
 history[:] = [(t, count) for t, count in history if now - t <= WINDOW_S]
 history.append((now, recommendation))
 if recommendation < replicas:
 result = min(replicas, max(count for _, count in history))
 else:
 result = recommendation
 print(f"t={now}s raw={recommendation} {replicas}->{result}")
 return result


def main():
 history = []
 replicas = step(3, history, 0, 100)
 assert replicas == 6
 replicas = step(replicas, history, 240, 50)
 assert replicas == 6
 replicas = step(replicas, history, 400, 15)
 assert replicas == 6 # A recent recommendation, not a recent scale event.
 replicas = step(replicas, history, 541, 15)
 assert replicas == 2
 replicas = step(replicas, history, 600, None)
 assert replicas == 2
 assert desired(2, 500) == 10
 assert desired(2, 5) == 2
 print("PASS: recommendation history holds 6 at t=400, then shrinks to 2")


if __name__ == "__main__":
 main()

At time zero the model grows from three to six. At 240 seconds it recommends six again without resizing. At 400 seconds a new recommendation of two is held at six because the recent recommendation remains in the window, even though the last actual resize was more than five minutes ago. At 541 seconds that high recommendation has expired and the model shrinks to two.

Both language copies passed with Python 3.12.14 in this review, including missing-input hold and both bounds. Those assertions establish the model’s stated behavior only. For cluster tuning, compare real HPA conditions, recommendations, replica transitions, Pod readiness, and application latency over time. No live controller or metrics pipeline was exercised here.

Sources

Kubernetes horizontal pod autoscaling, how HPA derives replica counts, checked 2026-10-15.
HorizontalPodAutoscaler walkthrough, worked example with multiple metrics and status conditions, checked 2026-10-15.

Kubernetes labels define selection, not authorization

Thu, 09 Sep 2027 00:00:00 +0000

Labels are key-value metadata used to select Kubernetes objects. A Service selector, an operational query, and a workload controller can all rely on them, but each consumer has its own API and behavior. A label change can therefore alter group membership while leaving the underlying process healthy. Review labels as inputs to actual consumers rather than as purely descriptive stickers.

Historical fixture observations, not a new cluster run. Missing-key inequality changes query membership; Service selection does not enforce network authorization. The fixture uses ordinary readiness and does not enforce NetworkPolicy.

Know which selector language is in use

Label keys use a qualified-name syntax, and values have format and length constraints. A value may be empty; otherwise its permitted characters and endpoints are restricted, with a maximum of sixty-three characters. Labels are not a place for arbitrary paragraphs or unconstrained JSON. Use annotations for descriptive metadata, while still respecting annotation validation and consumer contracts.

Equality requirements can test that a key has a value, while set-based requirements support membership and existence tests. Multiple requirements in a selector are combined as conditions that must all hold. An important detail is inequality: track!=canary also matches an object with no track label. If absence is unacceptable, require the key to exist as well instead of assuming inequality implies its presence.

Selector forms differ between resources. A normal selector-based Service uses a map of equality requirements; it does not accept the complete set-expression language as arbitrary map values. Other APIs expose LabelSelector structures, and kubectl list queries accept selector expressions. Check the exact field’s schema before translating a convenient command-line query into a manifest.

When it breaks

A Service’s selector identifies matching Pods in its namespace, and the endpoint machinery tracks addresses and readiness. Matching labels alone do not prove that a server is ready or that its port is correct. With normal readiness behavior, available routing targets depend on endpoint conditions too. Inspect the relevant EndpointSlices and application checks rather than stopping at a successful label query.

The publishNotReadyAddresses setting changes how readiness is treated for Service discovery. Do not assume an endpoint’s ready flag always proves the Pod passed its application probe under that setting. The fixture below uses the ordinary default and does not exercise this override.

Relabelling a healthy Pod can remove it from a Service without terminating it. Conversely, an overly broad selector can include a workload that was never intended as a backend. Changing labels on controller-managed Pods can also interact with ownership and reconciliation. Update the intended desired state through the responsible controller, and check all important consumers of a shared label before changing its meaning.

Selectors are not network authorization. A Pod excluded from one Service can remain reachable through its own address or another Service unless other controls prevent that access. NetworkPolicy, application authorization, and routing each solve different parts of the problem. Do not describe a selector as a guest list that denies all direct traffic to unselected Pods.

Check it quickly

The fixture and observations below are retained from the original article. This review did not rerun the Kubernetes integration; reported success is historical, not a new execution result.

Use a disposable Kubernetes cluster with kubectl and python:3.11-slim-bookworm available on the nodes. Save this as label-selectors.py and run python3 label-selectors.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The script creates a unique namespace containing three standalone HTTP Pods and a Service, then removes it. The recorded server version was Kubernetes 1.34.11.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-labels-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 for name, role, track in [('a', 'reader', 'stable'), ('b', 'reader', 'canary'), ('c', 'writer', None)]:
 script = '''from http.server import BaseHTTPRequestHandler,HTTPServer
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 data=LABEL.encode();self.send_response(200);self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''.replace('LABEL', repr(name))
 podspec = spec(script)
 podspec['containers'][0]['readinessProbe'] = {'httpGet': {'path': '/', 'port': 8080}, 'periodSeconds': 1}
 labels = {'app': 'catalog', 'role': role}
 if track:
 labels['track'] = track
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': name, 'labels': labels}, 'spec': podspec})
 for name in ('a', 'b', 'c'):
 wait_for(lambda name=name: obj(namespace, 'pod', name), pod_ready)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'readers'},
 'spec': {'selector': {'app': 'catalog', 'role': 'reader'}, 'ports': [{'port': 80, 'targetPort': 8080}]}})
 def selected():
 slices = obj(namespace, 'endpointslices', selector='kubernetes.io/service-name=readers')['items']
 return {e['targetRef']['name'] for item in slices for e in item.get('endpoints', [])
 if e.get('conditions', {}).get('ready') and e.get('targetRef')}
 before = wait_for(selected, lambda names: names == {'a', 'b'})
 inequality = {p['metadata']['name'] for p in obj(namespace, 'pods', selector='app=catalog,track!=canary')['items']}
 assert inequality == {'a', 'c'}
 expression = {p['metadata']['name'] for p in obj(namespace, 'pods', selector='app in (catalog),role=reader')['items']}
 assert expression == {'a', 'b'}
 writer_ip = obj(namespace, 'pod', 'c')['status']['podIP']
 probe = 'from urllib.request import urlopen; print(urlopen(' + repr('http://'+writer_ip+':8080/') + ',timeout=3).read().decode())'
 assert k('exec', '-n', namespace, 'a', '--', 'python3', '-c', probe).stdout.strip() == 'c'
 k('label', 'pod', 'b', '-n', namespace, 'role=writer', '--overwrite')
 after = wait_for(selected, lambda names: names == {'a'})
 assert pod_ready(obj(namespace, 'pod', 'b'))
 invalid = k('patch', 'pod', 'a', '-n', namespace, '--type=merge',
 '-p', json.dumps({'metadata': {'labels': {'notes.example/value': 'has spaces'}}}), expected=None)
 assert invalid.returncode != 0 and 'Invalid' in invalid.stderr
 print(json.dumps({'service_targets_before': sorted(before), 'service_targets_after': sorted(after),
 'inequality_also_matches_missing_key': sorted(inequality),
 'set_selector_matches': sorted(expression), 'excluded_pod_directly_reachable': True,
 'invalid_label_value_rejected': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

Pods a and b have role=reader; c has role=writer. All share app=catalog and become ready through HTTP probes. Pod a has a stable track, b has canary, and c lacks track. The reader Service initially has ready targets a and b. A set-based query finds those readers, while track!=canary finds a and c, confirming the missing-key behavior.

An actual request from a to c’s Pod address returns c, even though c is excluded from the reader Service. Changing b’s role to writer then leaves only a in the Service’s ready endpoint set. Pod b remains ready. A separate patch containing spaces in a label value is rejected by the API. The original article reports that all assertions passed.

This verifies API selection, EndpointSlice changes, and direct reachability in the isolated test network. It does not test every controller’s adoption rules, enforce a NetworkPolicy, or benchmark Service traffic. Keep selection intent, endpoint readiness, and access policy independently observable when reviewing routing changes.

Sources

Kubernetes: Labels and Selectors, valid values, equality and set-based requirements, checked 2026-10-15.
Kubernetes: Services, selector-based backend discovery, checked 2026-10-15.
Kubernetes: EndpointSlices, endpoint identity and readiness conditions, checked 2026-10-15.

Namespaces scope objects; policies establish other boundaries

Wed, 25 Aug 2027 00:00:00 +0000

A namespace scopes many Kubernetes objects within one cluster. Two teams can each create a ConfigMap named settings because those objects have different namespace-qualified identities. This organizational boundary is useful, but it does not automatically provide network isolation, a dedicated kernel, reserved node capacity, or complete tenant separation. Identify which additional policies enforce the boundaries your workload actually requires.

Namespace identity is one boundary. The historical fixture does not prove NetworkPolicy enforcement, CPU fairness or hostile-tenant isolation. ResourceQuota constrains admission rather than running-process bandwidth or physical reservations. No example was rerun.

Names are scoped by resource type

Pods, Services, ConfigMaps, and many workload controllers are namespaced. Nodes, namespaces themselves, and other cluster-level resources are not divided into separate copies merely because a client specifies a namespace. Check the resource’s API scope rather than assuming every object lives inside a room. Namespaces also do not form a nested hierarchy of independent child clusters.

Within a Pod, a short Service name normally resolves in the Pod’s namespace. A qualified name can target a Service elsewhere, such as api.other.svc.cluster.local in a cluster using the default domain. This naming convention locates a destination; it does not authorize or prohibit the connection. The cluster domain can be configured differently, so do not mistake the example suffix for a universal constant.

Namespace choice must remain explicit in automation. A correct object name combined with the wrong active namespace still addresses the wrong resource. Use deliberate kubeconfig, context, and namespace inputs, especially for mutation. Namespace scoping makes identity clearer, but an administrator with broad permissions can still modify objects across those scopes.

When it breaks

A namespace without an applicable NetworkPolicy remains connected according to the cluster network’s behavior. Merely creating a namespace or qualifying a DNS name does not deny other Pods. Network isolation requires suitable policies and a network implementation that enforces them. Application authentication remains another layer; successful name resolution says nothing about who should be allowed to read a response.

RBAC controls API operations separately. A RoleBinding can grant an account access to ConfigMaps in one namespace without granting the same access elsewhere. Permissions are additive, so another broad binding can still grant additional access. Test actual allowed and denied operations for the intended identity rather than assuming that adding a narrow Role cancels wider grants.

ResourceQuota primarily enforces admission limits, including object counts and aggregate resource requests or limits. It does not itself throttle a running process or reserve physical capacity. LimitRange can supply defaults and bounds for admitted workloads, while runtime resource enforcement has other mechanisms. These policies must be configured; the namespace alone does not make a noisy neighbor harmless.

Check it quickly

The unchanged historical cluster integration was not rerun in this review because Kubernetes is unavailable.

Use a disposable Kubernetes cluster with the default cluster.local DNS domain, kubectl administrator access for impersonation, and python:3.11-slim-bookworm available on its nodes. Save this as namespaces.py and run python3 namespaces.py KUBECONFIG_PATH CONTEXT with explicit test configuration. The recorded cluster ran Kubernetes 1.34.11 with kind networking. The script creates two unique namespaces and removes both afterward.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

suffix = uuid4().hex[:8]
first, second = 'notes-a-' + suffix, 'notes-b-' + suffix
created = []
try:
 for namespace, value in [(first, 'A'), (second, 'B')]:
 k('create', 'namespace', namespace)
 created.append(namespace)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': {'name': 'settings'}, 'data': {'scope': value}})
 script = '''from http.server import BaseHTTPRequestHandler,HTTPServer
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 data=VALUE.encode();self.send_response(200);self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''.replace('VALUE', repr(value))
 podspec = spec(script)
 podspec['containers'][0]['readinessProbe'] = {'httpGet': {'path': '/', 'port': 8080}, 'periodSeconds': 1}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'web', 'labels': {'app': 'web'}}, 'spec': podspec})
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'api'},
 'spec': {'selector': {'app': 'web'}, 'ports': [{'port': 80, 'targetPort': 8080}]}})
 wait_for(lambda namespace=namespace: obj(namespace, 'pod', 'web'), pod_ready)
 wait_for(lambda namespace=namespace: obj(namespace, 'endpointslices', selector='kubernetes.io/service-name=api')['items'],
 lambda slices: any(e.get('conditions', {}).get('ready') for s in slices for e in s.get('endpoints', [])))
 assert obj(first, 'configmap', 'settings')['data']['scope'] == 'A'
 assert obj(second, 'configmap', 'settings')['data']['scope'] == 'B'
 nodes_a = {n['metadata']['uid'] for n in obj(first, 'nodes')['items']}
 nodes_b = {n['metadata']['uid'] for n in obj(second, 'nodes')['items']}
 assert nodes_a == nodes_b and nodes_a
 apply(first, {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'reader'}, 'automountServiceAccountToken': False})
 apply(first, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'Role', 'metadata': {'name': 'read-config'},
 'rules': [{'apiGroups': [''], 'resources': ['configmaps'], 'verbs': ['get']}]})
 apply(first, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'RoleBinding', 'metadata': {'name': 'read-config'},
 'subjects': [{'kind': 'ServiceAccount', 'name': 'reader', 'namespace': first}],
 'roleRef': {'apiGroup': 'rbac.authorization.k8s.io', 'kind': 'Role', 'name': 'read-config'}})
 identity = 'system:serviceaccount:' + first + ':reader'
 assert k('--as', identity, 'get', 'configmap', 'settings', '-n', first).returncode == 0
 denied = k('--as', identity, 'get', 'configmap', 'settings', '-n', second, expected=None)
 assert denied.returncode != 0 and 'Forbidden' in denied.stderr
 apply(first, {'apiVersion': 'v1', 'kind': 'ResourceQuota', 'metadata': {'name': 'one-service'},
 'spec': {'hard': {'services': '1'}}})
 wait_for(lambda: obj(first, 'resourcequota', 'one-service'), lambda q: q.get('status', {}).get('used', {}).get('services') == '1')
 extra = {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'extra', 'namespace': first},
 'spec': {'ports': [{'port': 80}]}}
 quota_denial = k('create', '-f', '-', payload=extra, expected=None)
 assert quota_denial.returncode != 0 and 'exceeded quota' in quota_denial.stderr
 assert not obj(first, 'networkpolicies')['items'] and not obj(second, 'networkpolicies')['items']
 remote = 'api.' + second + '.svc.cluster.local'
 probe = '''import json,socket
from urllib.request import urlopen
print(json.dumps({'short_name':socket.gethostbyname('api'),
 'qualified_name':socket.gethostbyname(REMOTE),
 'cross_namespace_body':urlopen('http://'+REMOTE+'/',timeout=3).read().decode()}))'''.replace('REMOTE', repr(remote))
 network = json.loads(k('exec', '-n', first, 'web', '--', 'python3', '-c', probe).stdout)
 assert network['short_name'] == obj(first, 'service', 'api')['spec']['clusterIP']
 assert network['qualified_name'] == obj(second, 'service', 'api')['spec']['clusterIP']
 assert network['cross_namespace_body'] == 'B'
 print(json.dumps({'same_names_different_configmaps': True, 'nodes_are_cluster_scoped': True,
 'rbac_allows_local_and_denies_other_namespace': True,
 'quota_rejects_second_service': True, 'short_and_qualified_dns_match_services': True,
 'cross_namespace_http_succeeds_without_network_policy': True}))
finally:
 for namespace in reversed(created):
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

Each namespace receives a ConfigMap named settings, a Pod named web, and a Service named api. The ConfigMaps retain different A and B values. Node queries made with either namespace return the same cluster-scoped node identities. This verifies object-name scope without inventing separate node pools.

A test service account is granted ConfigMap reads only in the first namespace. Requests impersonating it succeed there and return Forbidden in the second. A ResourceQuota then permits one Service in the first namespace; creating another is rejected after the existing usage is observed. This exercises a real admission decision rather than only checking that a quota object exists.

Finally, a request from the first Pod resolves api to its local Service, resolves the qualified second Service to that Service’s address, and receives B over HTTP. Neither namespace has a NetworkPolicy. The original run reports all assertions passed. The experiment verifies naming, API authorization, one quota dimension, and network reachability. It does not demonstrate NetworkPolicy enforcement, CPU fairness, or full hostile-tenant isolation. Review and test each required boundary explicitly instead of making namespace membership stand in for all of them.

Sources

Kubernetes: Namespaces, object-name scope and cluster-scoped resources, checked 2026-09-13.
Kubernetes: Resource Quotas, namespace admission limits and object counts, checked 2026-09-13.
Kubernetes: RBAC authorization, namespace bindings and additive permissions, checked 2026-09-13.
Kubernetes: Network Policies, network isolation requires policy and an enforcing implementation, checked 2026-09-13.
Kubernetes: DNS for Services and Pods, short names, namespace-qualified Service names and cluster domains, checked 2026-09-13.

Kubernetes Services give pods a stable home

Mon, 23 Aug 2027 00:00:00 +0000

A Service separates a stable discovery interface from replaceable backends.

The before/after sets are model results, not a transport test. Stable discovery does not prove every request succeeds. Endpoint conditions, ports and datapaths need separate checks. The exact local model passed again; no Kubernetes or transport test was run.

One address in front of many

An ordinary ClusterIP Service keeps a virtual address during that object’s lifetime, and cluster DNS provides a discovery name. Callers can depend on that name while interchangeable Pods are replaced. This separates a stable client-facing contract from the lifecycle of individual backends, without requiring a config push for every replacement.

Headless and ExternalName Services have different discovery behavior, and Services without selectors can use separately managed EndpointSlices. Scope the stable-VIP claim to the ordinary case. A Deployment manages workloads rather than acting as a network client itself; applications, gateways and other components use the resulting service discovery.

Selection drives which pods respond

A selector-based Service identifies matching Pods in its namespace. EndpointSlice controllers reconcile their addresses and conditions asynchronously. Readiness normally determines eligibility for ordinary traffic, but an unready endpoint may remain present with a false readiness condition rather than being erased from the API object.

The selector does not authorize network access, and a label match does not prove a correct targetPort or useful application response. publishNotReadyAddresses and terminating-endpoint behavior also affect interpretation. Check the selected set and its conditions, then inspect the actual datapath; these are related observations, not interchangeable evidence.

Traffic goes to whatever matches

Scaling and replacement can happen behind one name, but request continuity is not guaranteed by that name. A connection already established to a backend can fail when the backend terminates. Updates take time to reach proxies, and an empty eligible set cannot serve traffic merely because DNS still resolves.

Plan readiness, rollout capacity, termination draining and client retries together. Retries also need application-level safety for non-idempotent operations. Observe availability during replacement rather than asserting that a precise selector prevents every interruption. The local model below demonstrates set selection only, not those timing and transport properties.

The stable name is a promise to callers

Treat the Service name as the supported discovery interface, rather than caching a Pod IP seen once. A caller using an ordinary Service VIP does not need to resolve again for every backend change, but resolving only once forever fails when the Service is recreated with a new IP. Respect resolution and connection lifetimes appropriate to the client.

Headless clients need their own discovery and balancing behavior because answers can contain changing endpoint addresses. Test how the library refreshes DNS, replaces connections and handles stale results. Stability is an API contract with lifecycle conditions, not an instruction to disable refresh or ignore failures.

When it breaks

The quietest failure is a selector that matches nothing. A typo such as app: webs instead of app: web, or a Deployment that stopped setting the expected label, leaves the Service with zero endpoints. The cluster IP and DNS name still resolve, so the outage looks like refused connections rather than a missing name. When a Service answers lookups but carries no traffic, inspect its endpoint set first; an empty eligible set can reflect labels, readiness, Pod addresses or reconciliation; it does not by itself rule out a separate network problem.

A headless Service chosen by habit is the next trap. Setting clusterIP: None removes the virtual IP, so DNS returns the changing pod addresses and each client must balance across them. Code written for one stable endpoint suddenly faces many shifting ones, with uneven retries and broken connection reuse. Choose headless only when the caller explicitly wants discovery.

Ports break the promise from the other side. A Service mapped to an incorrect container targetPort cannot deliver requests to the intended listener, and an already-allocated NodePort makes the API operation fail. ClientIP affinity can concentrate traffic from clients sharing an address. It does not promise permanent attachment to a removed endpoint or automatically turn every backend failure into a total outage. Choose affinity from the application requirement and test failover.

Check it quickly

Save the following as service_endpoints.py in a scratch directory and run python3 service_endpoints.py. It models endpoint convergence with plain dictionaries and changes nothing outside the process: one function derives the endpoint set from a selector plus pod labels and readiness, and four scenarios exercise it, including a deliberate empty selector as the negative control.

"""Simulate Service endpoint convergence from selector plus readiness."""

PODS = [
 {"name": "web-1", "labels": {"app": "web"}, "ready": True},
 {"name": "web-2", "labels": {"app": "web"}, "ready": True},
 {"name": "web-3", "labels": {"app": "web"}, "ready": False},
 {"name": "worker-1", "labels": {"app": "worker"}, "ready": True},
]


def select_endpoints(pods, selector):
 """Derive the endpoint set; an empty selector warns instead of matching."""
 if not selector:
 return [], "warning: empty selector matches nothing"
 matched = sorted(
 pod["name"]
 for pod in pods
 if pod.get("ready")
 and all(pod.get("labels", {}).get(key) == value for key, value in selector.items())
 )
 return matched, ""


def check(condition, message):
 if not condition:
 raise AssertionError(message)


def main():
 endpoints, warning = select_endpoints(PODS, {"app": "web"})
 check(endpoints == ["web-1", "web-2"], "ready web pods should form the set")
 check(warning == "", "a precise selector must not warn")
 print("steady state:", endpoints)

 rolling = [dict(pod, ready=True) if pod["name"] == "web-3" else pod for pod in PODS]
 rolling = [pod for pod in rolling if pod["name"] != "web-1"]
 endpoints, _ = select_endpoints(rolling, {"app": "web"})
 check(endpoints == ["web-2", "web-3"], "rollover must swap one ready pod for another")
 print("after rollover:", endpoints)

 endpoints, _ = select_endpoints(PODS, {"app": "web", "track": "stable"})
 check(endpoints == [], "an over-narrow selector should converge to empty, not error")
 print("over-narrow selector:", endpoints)

 endpoints, warning = select_endpoints(PODS, {})
 check(endpoints == [] and "empty selector" in warning, "empty selector must warn loudly")
 print("empty selector:", repr(warning))

 print("PASS: four convergences; readiness-gated selection, rollover, and empty warning hold")


if __name__ == "__main__":
 main()

The four scenarios verify this local selection model: the steady state holds the two ready web pods while the unready one stays out, and the rollover swaps one ready pod for another without changing the selector. The empty selector is the negative control: it warns loudly instead of silently matching everything or nothing. Both language fixtures passed with Python 3.12.14 on Linux in this review. The empty-selector warning is a local safeguard; Kubernetes can serve selectorless Services through explicitly managed EndpointSlices.

The simulation checks label-plus-readiness convergence only. It does not run kube-proxy, cluster DNS, or real probe timing. Before deploying, inspect the live endpoint set alongside pod labels and readiness, and repeat the lookup against the real Service name.

Sources

Kubernetes: Services, the Service abstraction, checked 2026-10-15.
Kubernetes: Pods, the pods a Service selects, checked 2026-10-15.
Kubernetes: Deployments, managing the pod set, checked 2026-10-15.
Primary reference, endpoint conditions and readiness exceptions, checked 2026-10-15.
Primary reference, ordinary and headless DNS answers, checked 2026-10-15.

Startup probes delay other probes until initialization succeeds

Sun, 22 Aug 2027 00:00:00 +0000

A startup probe gives a container a separate initialization contract. While that configured probe has not succeeded, Kubernetes does not execute the container’s liveness or readiness probes. This is stronger and more specific than ignoring some liveness failures during a grace period: those other checks have not begun. Once startup succeeds, ordinary ongoing health and readiness behavior takes over.

The threshold-times-period product helps planning but is not an exact restart time. Startup success does not force readiness or guarantee continuing health. The retained Kubernetes integration was not rerun.

Give each stage an observable condition

The application process starts before its startup probe can succeed. A Running container may therefore still have started reported as false in container status and may not make its Pod Ready. Running describes execution; successful startup describes completion of the configured startup check. An HTTP listener that opens early can expose progress without yet claiming that initialization is complete.

Choose a startup condition that represents necessary local preparation, such as loading an index or validating required initial state. A port opening is sufficient only if that is the actual readiness prerequisite you intend to observe. If the listener appears before expensive setup finishes, checking only that listener can release liveness protection too early and recreate the original restart loop.

After a successful startup check, kubelet stops that startup probe for the current container instance. It runs liveness and readiness according to their separate configuration. Startup success is not a permanent health certificate: the application can later become unready or require a restart. A new container instance must complete its own startup check again.

When it breaks

A startup probe that never succeeds eventually reaches its failure threshold. Kubelet then terminates the container and applies its restart policy. Startup protection is therefore not permission to initialize forever. Set period, timeout and failure threshold from measured initialization behavior, allowing for the environment’s realistic resource contention and expected recovery time.

The product of failureThreshold and periodSeconds is useful when planning the startup allowance, but do not present it as an exact stopwatch for every observed restart. Initial delay, probe execution and scheduling, termination grace and restart backoff also affect the timeline. Record the actual attempts and state changes when investigating a slow deployment.

Do not make successful readiness a prerequisite for liveness unless that dependency is explicitly represented by startup configuration. Readiness and liveness otherwise answer separate ongoing questions. Equally, passing startup does not force readiness to succeed; a later readiness check can still reject traffic eligibility for a different reason, while the process remains alive.

Changing checks requires application support. An endpoint that always returns success cannot verify a missing initialization step merely because it is named startup. Keep its state transition deliberate, expose enough diagnostic information to explain waiting, and avoid placing credentials or sensitive initialization details in responses readable by unintended clients. The fixture below uses only counters and synthetic state.

Check it quickly

The fixture and observations below are retained from the original article. This review did not rerun the Kubernetes integration; reported success is historical, not a new execution result.

Use a disposable Kubernetes cluster, kubectl and python:3.11-slim-bookworm available to its nodes. Save the script as startup-gate.py and run python3 startup-gate.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. Namespace creation and deletion permissions are required. The recorded server was Kubernetes 1.34.11. The startup probe allows sixty failed attempts at one-second periods for this short controlled exercise.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-startup-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 server = '''from http.server import BaseHTTPRequestHandler,HTTPServer
from pathlib import Path
import json
counts={'startup':0,'live':0,'ready':0}
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 key=self.path.removeprefix('/'); code=200
 if key in counts:
 counts[key]+=1
 if key=='startup' and not Path('/tmp/start').exists(): code=503
 body=json.dumps(counts).encode(); self.send_response(code)
 self.send_header('Content-Length',str(len(body))); self.end_headers(); self.wfile.write(body)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''
 podspec = spec(server)
 c = podspec['containers'][0]
 c['startupProbe'] = {'httpGet': {'path': '/startup', 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 60}
 for field, path in [('livenessProbe', '/live'), ('readinessProbe', '/ready')]:
 c[field] = {'httpGet': {'path': path, 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 2}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'slow'}, 'spec': podspec})
 wait_for(lambda: obj(namespace, 'pod', 'slow'), lambda p:
 any('running' in c['state'] for c in p.get('status', {}).get('containerStatuses', [])))
 read = 'from urllib.request import urlopen; print(urlopen("http://127.0.0.1:8080/stats",timeout=3).read().decode())'
 def counts():
 result = k('exec', '-n', namespace, 'slow', '--', 'python3', '-c', read, expected=None)
 return json.loads(result.stdout) if result.returncode == 0 else None
 before = wait_for(counts, lambda d: d and d['startup'] >= 3)
 assert before['live'] == before['ready'] == 0
 starting = obj(namespace, 'pod', 'slow')
 assert not pod_ready(starting)
 assert starting['status']['containerStatuses'][0]['started'] is False
 assert starting['status']['containerStatuses'][0]['restartCount'] == 0
 k('exec', '-n', namespace, 'slow', '--', 'python3', '-c',
 'from pathlib import Path; Path("/tmp/start").touch()')
 ready = wait_for(lambda: obj(namespace, 'pod', 'slow'), pod_ready)
 assert ready['status']['containerStatuses'][0]['started'] is True
 enabled = wait_for(counts, lambda d: d and d['live'] > 0 and d['ready'] > 0)
 time.sleep(3)
 after = counts()
 assert after['startup'] == enabled['startup']
 assert after['live'] > enabled['live'] and after['ready'] > enabled['ready']
 assert obj(namespace, 'pod', 'slow')['status']['containerStatuses'][0]['restartCount'] == 0
 print(json.dumps({'before_startup_success': before, 'after_startup_success': after,
 'other_probes_suppressed_before_startup': True,
 'startup_probe_stops_after_success': True, 'container_restarts': 0}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The HTTP server counts requests separately for startup, live and ready paths. Its statistics path is observational and does not increment those counters. Startup initially returns 503 until a local release file appears. After three observed startup requests, both other counters are zero, the container has not restarted, and its started field is false.

Creating the release file makes the next startup request succeed. The Pod becomes Ready, started becomes true, and the live and ready counters begin increasing. After another three seconds, the startup counter stays fixed while both ongoing counters increase. The recorded final counts were four, four and five; exact ongoing counts can vary with timing.

The original article reports that all assertions passed without a restart, and the namespace was removed. This verifies suppression and transition using actual HTTP calls. It does not measure a production boot-time percentile or exercise startup-budget exhaustion. Use those additional measurements when choosing real thresholds, while keeping startup completion distinct from continuing service health.

Sources

Kubernetes: Configure Probes, startup failure budgets and probe configuration, checked 2026-10-15.
Kubernetes: Pod Probes, suppression of liveness and readiness until startup succeeds, checked 2026-10-15.

Kubernetes Secrets require explicit access boundaries

Sat, 21 Aug 2027 00:00:00 +0000

A Kubernetes Secret is an API object for confidential values that workloads need. It gives operators a dedicated object type, consumption paths, and access-control targets. It does not make data unreadable to every other actor. Design the boundary around who can retrieve the object, create consumers, inspect their output, and administer the nodes that run them.

A Secret’s boundary includes both direct readers and actors who can create consumers. A narrow projection limits one delivery, not necessarily its author. Rotation also needs delivery, reload and issuer verification. Conceptual diagram; no new cluster test.

Encoding and protection are different layers

In the JSON API representation, Secret data values use base64; stringData is a write convenience for supplying strings. Base64 can be reversed without a key and is not encryption. The representation does not prove how a particular cluster protects its backing storage. Kubernetes documentation describes Secrets as unencrypted at rest by default unless encryption is configured. Verify the actual API-server and storage setup rather than inferring it from encoded output.

Storage encryption also does not prevent an authorized API reader from obtaining the data. RBAC rules for get, list, and watch can expose Secret contents, so a broad listing permission is not merely access to object names. Protect backups and diagnostic output too. A working application eventually receives usable values, which means a compromised authorized consumer remains inside the data’s trust boundary.

Secrets are intended for small values. Kubernetes v1.34.0 limits the sum of decoded data values to one MiB; that check does not count the base64 representation or metadata. Separate request and metadata limits still apply. Mount only required keys and restrict access to the relevant containers. Unix file modes can narrow access within a workload, but do not replace API authorization or control of workload creation. Environment injection also has an exposure and lifecycle tradeoff; choose it from the application’s needs rather than assuming it is automatically safer than a file.

When it breaks

Denying get on Secrets does not necessarily deny indirect access. A subject allowed to create Pods in a namespace can normally name a Secret for a volume in its Pod specification. Kubelet then supplies the data to that workload. Permissions to create suitable controllers can have similar consequences. Review workload-creation rights and applicable admission controls alongside direct data-reading rules.

Broadly mounting every key expands the set of values a process can read. Selecting one key through volume items limits that particular projection, but a privileged creator may be able to request a different projection. Do not mistake a narrow manifest for proof that the actor writing it has equally narrow authority. The intended consumer and the authority to define consumers are separate parts of the boundary.

Rotation requires more than editing an API object. Delivery, application reload, credential issuance, and revocation have their own timing. Environment snapshots and subPath mounts do not automatically refresh like a normal Secret projection. An updated file also does not force an application to discard a value it already cached. Test the issuing system and consumers together before claiming that old credentials are no longer accepted.

Check it quickly

The fixture and observations below are retained from the original article. This review did not rerun the Kubernetes integration; reported success is historical, not a new execution result.

Use a disposable Kubernetes cluster with administrator access for test impersonation and python:3.11-slim-bookworm available on its nodes. Save this as secret-access.py and run python3 secret-access.py KUBECONFIG_PATH CONTEXT against its dedicated configuration. Kubernetes 1.34.11 was used for the recorded run. Every value in this fixture is deliberately synthetic, and output reports checks without printing the stored value.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

import base64
namespace = 'notes-secrets-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 demo = 'DEMO_ONLY_NOT_A_CREDENTIAL'
 k('create', '-n', namespace, '-f', '-', payload={'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'example'},
 'type': 'Opaque', 'stringData': {'needed': demo, 'unused': 'UNUSED_DEMO'}})
 stored = obj(namespace, 'secret', 'example')
 assert base64.b64decode(stored['data']['needed']).decode() == demo
 apply(namespace, {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'creator'},
 'automountServiceAccountToken': False})
 apply(namespace, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'Role',
 'metadata': {'name': 'pod-creator'},
 'rules': [{'apiGroups': [''], 'resources': ['pods'], 'verbs': ['create', 'get']},
 {'apiGroups': [''], 'resources': ['pods/log'], 'verbs': ['get']}]})
 apply(namespace, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'RoleBinding',
 'metadata': {'name': 'pod-creator'},
 'subjects': [{'kind': 'ServiceAccount', 'name': 'creator', 'namespace': namespace}],
 'roleRef': {'apiGroup': 'rbac.authorization.k8s.io', 'kind': 'Role', 'name': 'pod-creator'}})
 identity = 'system:serviceaccount:' + namespace + ':creator'
 denied = k('--as', identity, 'get', 'secret', 'example', '-n', namespace, expected=None)
 assert denied.returncode != 0 and 'Forbidden' in denied.stderr
 probe = "from pathlib import Path\nimport json,time\nroot=Path('/example')\nprint(json.dumps({'synthetic_value_matches':(root/'one').read_text()=='DEMO_ONLY_NOT_A_CREDENTIAL',\n 'unused_key_absent':not (root/'unused').exists(),\n 'service_account_token_absent':not Path('/var/run/secrets/kubernetes.io/serviceaccount/token').exists()}),flush=True)\ntime.sleep(600)"
 podspec = spec(probe)
 podspec['serviceAccountName'] = 'creator'
 podspec['volumes'] = [{'name': 'example', 'secret': {'secretName': 'example',
 'items': [{'key': 'needed', 'path': 'one'}], 'defaultMode': 0o400}}]
 podspec['containers'][0]['volumeMounts'] = [{'name': 'example', 'mountPath': '/example', 'readOnly': True}]
 definition = {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'reader', 'namespace': namespace},
 'spec': podspec}
 k('--as', identity, 'create', '-f', '-', payload=definition)
 wait_for(lambda: obj(namespace, 'pod', 'reader'), pod_ready)
 output = wait_for(lambda: k('--as', identity, 'logs', '-n', namespace, 'reader').stdout.strip(), bool)
 observed = json.loads(output)
 assert all(observed.values())
 print(json.dumps({'api_base64_decodes_to_synthetic_value': True,
 'direct_secret_get_forbidden': True, 'same_identity_can_create_consuming_pod': True,
 'mount_observation': observed, 'etcd_encryption_tested': False}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script creates a Secret with two dummy keys and confirms that its API base64 representation decodes to the supplied synthetic text. It binds a service account to a Role permitting Pod creation and inspection of Pod logs, without permitting Secret reads. A request impersonating that account to get the Secret returns Forbidden.

The same impersonated subject then creates a Pod that mounts only the needed key. Its own program verifies the expected dummy value, absence of the unused key, and absence of an automatically mounted service-account token. The subject can read those boolean results through its permitted log access. The original article reports that all checks passed, demonstrating the indirect workload path despite the direct denial.

The namespace is removed afterward. No real credential, issued service-account token, backing-store encryption, or rotation process was tested. The experiment establishes authorization and projection behavior, not a complete secret-management system. Assess both direct readers and actors who can create or alter readers before describing a Secret as narrowly scoped.

Sources

Kubernetes: Secrets, API representation, consumption and access implications, checked 2026-10-15.
Kubernetes: Good practices for Secrets, encryption, least privilege and workload access, checked 2026-10-15.
Kubernetes: RBAC authorization, resource verbs, role bindings and namespace scope, checked 2026-10-15.
Secret validation (v1.34.0), ValidateSecret sums decoded data-value bytes before comparing MaxSecretSize, checked 2026-10-18.

ConfigMaps separate configuration from the image

Fri, 20 Aug 2027 00:00:00 +0000

A ConfigMap stores non-confidential configuration separately from an image. That lets a release change an environment-specific value without rebuilding the application artifact. The useful boundary is explicit: the image supplies code, while the workload definition identifies which configuration that code should consume. A release is therefore more than an image tag; its configuration reference is part of the behavior being deployed.

Configuration reference is part of a release. The diagram explains historical selection and retained-input rollback, without proving application compatibility, rollout availability or reversal of external effects. No example was rerun.

Give configuration a reviewable identity

ConfigMap data values are strings, with binaryData available for binary values represented through the API’s encoding. The object is intended for small configuration, with a one-MiB data limit. It is not a general file repository or a secret-protection mechanism. The API can validate object structure without understanding whether a particular application setting is meaningful or safe for the business.

A Pod can consume values through environment references or mounted files. Ordinary Pod references resolve the ConfigMap in that Pod’s namespace. Choose names and required keys deliberately, then validate the application’s interpretation. An accepted Deployment object does not prove that every referenced key exists or that a new container can start successfully with the selected values.

Define units, accepted values, and defaults in the application’s configuration schema. A stored string resembling a boolean or number does not choose its parsing rules. Review those rules with the consumer before promoting a configuration between environments.

Versioned immutable ConfigMaps make release identity easier to inspect. A name such as settings-v2 can be retained alongside settings-v1, with immutable preventing data changes to the existing object. This protection does not prohibit an authorized actor from deleting and recreating a name. Keep object lifecycle permissions and release records aligned with the intended immutability promise.

When it breaks

Editing configuration and replacing processes are different actions. Environment values already injected into a running container are not rewritten by a ConfigMap update. Normal projected files can update eventually, but the application must read the new contents; subPath and previously opened files introduce additional lifetime distinctions. Do not report a configuration rollout merely because the API now shows a new value.

Changing a Deployment’s Pod-template reference to a new ConfigMap is a concrete rollout trigger. Changing the ConfigMap itself does not automatically rewrite that template. Likewise, a checksum annotation is useful only when a tool actually updates it in the Pod template. Kubernetes does not discover arbitrary external configuration changes and generate that checksum on the application’s behalf.

Rollback depends on retained inputs. Repointing a workload to settings-v1 fails if the old object has been deleted, and restoring an image alone cannot restore a changed configuration. Keep needed versions available and check consumers before cleanup. Configuration changes also need business validation; a syntactically valid threshold can still disable a service or change its behavior incorrectly.

Check it quickly

Use a disposable Kubernetes cluster with kubectl access and python:3.11-slim-bookworm available to its nodes. Save this as config-releases.py and run python3 config-releases.py KUBECONFIG_PATH CONTEXT, substituting that cluster’s dedicated configuration path and context name. The test needs permission to create and delete its own namespace. The recorded server version was Kubernetes 1.34.11, with kubectl 1.34.1.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}


def deployment(namespace, name, podspec):
 return apply(namespace, {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': {'name': name},
 'spec': {'replicas': 1, 'selector': {'matchLabels': {'app': name}},
 'template': {'metadata': {'labels': {'app': name}}, 'spec': podspec}}})


def deployment_pod(namespace, name):
 wait_for(lambda: obj(namespace, 'deployment', name), lambda d:
 d.get('status', {}).get('observedGeneration', 0) >= d['metadata']['generation']
 and d.get('status', {}).get('updatedReplicas') == 1
 and d.get('status', {}).get('availableReplicas') == 1
 and d.get('status', {}).get('replicas') == 1)
 def pods():
 return [p for p in obj(namespace, 'pods', selector='app=' + name)['items']
 if not p['metadata'].get('deletionTimestamp')]
 return wait_for(pods, lambda p: len(p) == 1 and pod_ready(p[0]))[0]

namespace = 'notes-config-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 for version, mode in [('v1', 'blue'), ('v2', 'green')]:
 apply(namespace, {'apiVersion': 'v1', 'kind': 'ConfigMap',
 'metadata': {'name': 'settings-' + version}, 'immutable': True, 'data': {'mode': mode}})
 podspec = spec()
 podspec['containers'][0]['env'] = [{'name': 'MODE', 'valueFrom':
 {'configMapKeyRef': {'name': 'settings-v1', 'key': 'mode'}}}]
 deployment(namespace, 'web', podspec)
 first = deployment_pod(namespace, 'web')
 def mode(pod):
 return k('exec', '-n', namespace, pod['metadata']['name'], '--', 'python3', '-c',
 'import os; print(os.environ["MODE"])').stdout.strip()
 assert mode(first) == 'blue'
 rejected = k('patch', 'configmap', 'settings-v1', '-n', namespace, '--type=merge',
 '-p', json.dumps({'data': {'mode': 'changed'}}), expected=None)
 assert rejected.returncode != 0 and 'immutable' in rejected.stderr
 assert obj(namespace, 'configmap', 'settings-v1')['data']['mode'] == 'blue'
 def select(version):
 patch = [{'op': 'replace',
 'path': '/spec/template/spec/containers/0/env/0/valueFrom/configMapKeyRef/name',
 'value': 'settings-' + version}]
 k('patch', 'deployment', 'web', '-n', namespace, '--type=json', '-p', json.dumps(patch))
 return deployment_pod(namespace, 'web')
 second = select('v2')
 assert mode(second) == 'green' and second['metadata']['uid'] != first['metadata']['uid']
 third = select('v1')
 assert mode(third) == 'blue' and third['metadata']['uid'] != second['metadata']['uid']
 image_ids = [p['status']['containerStatuses'][0]['imageID'] for p in (first, second, third)]
 assert len(set(image_ids)) == 1
 assert len({p['metadata']['uid'] for p in (first, second, third)}) == 3
 print(json.dumps({'modes': ['blue', 'green', 'blue'], 'distinct_pod_uids': 3,
 'same_image_id': image_ids[0], 'immutable_update_rejected': True,
 'retained_config_rollback_verified': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script creates two immutable ConfigMaps and a single-replica Deployment referencing settings-v1. The first Pod reads blue from its environment. Attempting to overwrite that ConfigMap’s data is rejected, and reading the object confirms the original value remains blue. This checks actual server enforcement rather than just the manifest’s immutable field.

Next, a Pod-template patch selects settings-v2. After controller reconciliation and readiness, a different Pod reads green. Selecting the retained first version causes another replacement that reads blue again. The three Pod UIDs are distinct, while all three observed image identities match. The original run reports all assertions passed and namespace cleanup; no cluster run was repeated here. The image tag remains mutable: matching observed identities prove this run, not future reproducibility. Pin a digest when that is required.

This verifies configuration selection, replacement, and a retained-input rollback using a simple worker. It does not establish application compatibility, production availability during rollout, or rollback of external side effects. Carry the image, configuration version, workload definition, and observed application result together in the release evidence so each part of the change can be reviewed.

Sources

Kubernetes: ConfigMaps, non-confidential data, consumption and immutability, checked 2026-09-13.
Kubernetes: Deployments, Pod-template changes and rolling replacement, checked 2026-09-13.
Kubernetes: Images, mutable tags and digest-pinned identity, checked 2026-10-15.

Kubernetes Deployments describe desired state

Thu, 19 Aug 2027 00:00:00 +0000

You describe the end state, not the steps. That idea underlies replicas, updates, and rollback: the cluster keeps working to match your description.

The count sequences describe the article’s local arithmetic model. Real controllers have separate responsibilities and asynchronous failure modes. Historical model passes are unchanged; no example was rerun.

Spec beats imperative steps

A Deployment is a statement of the end state, not a list of steps: how many replicas, which image, what update strategy, which labels. The operator publishes the what, and the controller, a reconcile loop, is responsible for the how. The distinction is resource ownership, not whether a shell command was used. kubectl run normally creates a standalone Pod; its kubelet can restart containers, but no ReplicaSet maintains a replacement population after that Pod disappears. A Deployment can itself be created with a command, yet its stored specification continues guiding controllers after the client exits. Interrupting the client does not reliably indicate whether the API accepted the request: inspect the stored object before retrying or diagnosing convergence.

A controller converges the gap

The Deployment controller manages ReplicaSets, and ReplicaSet controllers maintain their Pod populations. Kubelet can restart a crashed container within the same Pod; replacing a deleted or lost Pod is a different reconciliation path. These mechanisms cooperate, but should not be collapsed into one controller that creates a Pod every few seconds.

Reconciliation is asynchronous and depends on working control-plane components and sufficient capacity. Image pull failures, quotas, admission denial or unavailable storage can prevent convergence indefinitely. Observe conditions and events. A desired replica count is a target to pursue, not proof of ready capacity or a bound on recovery time.

Updates happen through new Revision

Changing the Pod template triggers a rollout and revision; scaling alone does not. The controller adjusts old and new ReplicaSet sizes according to the rollout strategy, readiness and availability constraints. Services and their datapaths handle traffic separately, so the Deployment controller itself does not gradually switch individual connections.

A rollback restores a previous Pod template, not every field of the Deployment or external state such as a database schema. Retained ReplicaSets supply rollback history, with a default revisionHistoryLimit of ten; cleanup starts after completion. Reapplying an unchanged template does not generate a fresh revision merely because a pipeline ran.

When it breaks

In apps/v1, a Deployment selector is immutable and the Pod template must match it. A template that violates the selector is rejected by API validation, rather than creating an accepted Deployment that endlessly produces unowned Pods. Plan selector changes as an explicit migration. Overlapping controllers deserve scrutiny, but matching labels alone do not erase controlling owner references.

Deleting a managed Pod or directly scaling an owned ReplicaSet can be corrected by its controllers. In contrast, kubectl scale deployment updates the Deployment’s desired replicas and persists until another actor changes that field. HPA, a later apply or a GitOps reconciler can overwrite it; the Deployment does not secretly remember an earlier Git value on its own.

A stalled rollout can retain more ReplicaSets than the intended historical limit because cleanup has completion conditions. Inspect rollout status before declaring a retention leak. Keep external change history and a tested rollback plan; zero retained history removes normal rollback targets, while a large number alone does not prove control-plane performance degradation.

Check it quickly

Save the following as desired_state_loop.py and run python3 desired_state_loop.py. It uses only the standard library and illustrates a simplified count-reconciliation loop: compare desired against observed, emit one corrective action per tick, and repeat until the gap closes.

"""Simulate a Deployment-style desired-state reconcile loop (stdlib only)."""

def reconcile_step(desired, observed):
 if observed < desired:
 return "create-one-pod"
 if observed > desired:
 return "delete-one-pod"
 return "in-sync"


def converge(desired, observed, log):
 while observed != desired:
 action = reconcile_step(desired, observed)
 log.append(action)
 if action == "create-one-pod":
 observed += 1
 else:
 observed -= 1
 log.append("in-sync")
 return observed


def main():
 desired = 3
 observed = 1 # one survivor after a node loss
 actions = []
 observed = converge(desired, observed, actions)
 assert observed == desired
 assert actions == ["create-one-pod", "create-one-pod", "in-sync"], actions

 # Manual drift: someone adds two pods without changing desired.
 observed = 5
 drift = []
 observed = converge(desired, observed, drift)
 assert observed == desired
 assert drift == ["delete-one-pod", "delete-one-pod", "in-sync"], drift

 # Model a selector/template validation check before accepting desired state.
 selector = {"app": "api"}
 template_labels = {"app": "api-v2"} # deliberate mismatch, a negative control
 matched = all(template_labels.get(k) == v for k, v in selector.items())
 assert not matched, "mismatched template must fail this validation model"

 print(f"desired={desired} final-observed={observed}")
 print(f"recovery-actions={actions}")
 print(f"observed-count-drift-corrected={drift}")
 print("PASS: controller closes the desired-vs-observed gap; mismatched template is rejected by the local validation check")


if __name__ == "__main__":
 main()

The two integer-count scenarios end in sync. The separate label check rejects the mismatched template as a negative control, illustrating the contract checked before API acceptance. It does not submit a real Deployment or model adoption, watches, scheduling and readiness.

Both language fixtures passed this review with Python 3.12.14 on Linux. The model checks convergence arithmetic, not scheduling or storage. Before release day, confirm the live selector, replicas, and revision history.

Sources

Kubernetes: Deployments, the Deployment resource explained, checked 2026-10-15.
Kubernetes: Controllers, the platform and its model, checked 2026-10-15.

A PVC gives a Pod durable storage beyond the container

Fri, 13 Aug 2027 00:00:00 +0000

A PersistentVolumeClaim separates storage lifetime from a recreated Pod; it does not guarantee data recovery. A PersistentVolume describes storage, a claim requests it, and controllers arrange binding or provisioning. Recovery still depends on backing storage, usable topology and backups.

Claims and volumes have separate lifetimes. Finalizers can postpone deletion, while reclaim policy controls the released storage. Review both before maintenance: mistakes can cause persistent Terminating states, unintended data loss or retained storage bills.

Protection delays deletion; it does not establish controller health. Retain requires an ownership, backup and provider-procedure decision. Retained diagnostics remain blocked and were not rerun.

The claim, the volume, and the binding

Ordinary matching checks capacity, access modes, storage class, volumeMode, selectors and existing ownership. An unmatched claim can remain Pending; inspect its events and StorageClass rather than assuming a single cause. Binding is not proof of usable topology: explicitly naming a PV can bind despite conflicting node affinity. Scheduling and mounting still have to succeed.

accessModes has no default. ReadWriteOnce permits read-write mounting on one node, potentially by several Pods there; it is not single-Pod exclusivity. ReadWriteOncePod provides that constraint for supported CSI volumes. Access modes alone do not enforce write protection after mounting. Immediate starts binding or provisioning without waiting for a consumer, risking unusable placement for topology-constrained storage. Supported WaitForFirstConsumer provisioning coordinates storage with a consuming Pod’s scheduling constraints before placement completes.

The deletion order and the finalizer

A usual retirement sequence is Pod, claim, then volume. Deleting an in-use claim records deletionTimestamp, but kubernetes.io/pvc-protection can postpone removal. In Kubernetes v1.34.0, the ordinary PVC-reference check counts Pods assigned to a node; an unscheduled Pod reference alone does not hold protection. This is a controller reference check, not a live mount probe. A stuck Terminating claim therefore needs inspection of scheduled consumers, controller progress and other finalizers.

The same protection exists in the other direction. A PersistentVolume carries kubernetes.io/pv-protection, so deleting a volume that is bound to a claim is postponed until it is no longer bound. That ordering delays an accidental deletion; a deletionTimestamp cannot simply be cancelled. This explains why a volume can seem to ignore a delete for as long as its claim exists.

Reclaim policy: Delete or Retain

When the claim is finally gone the volume is released, and the reclaim policy takes over. With a supporting storage plugin, Delete reclaims the PV and backing asset asynchronously; removal is not an atomic or immediate guarantee. Retain keeps the PV object, marks it Released, and deliberately does not make it available for a new claim, because the previous claimant’s data is still on it — reclamation becomes a manual sequence: verify ownership and backups, then deliberately reclaim, reuse or delete storage according to the provider’s procedure.

Dynamically provisioned volumes inherit the policy from their StorageClass, whose default is Delete. Whether that default fits a database depends on retention and recovery requirements. It is easy to overlook: nothing in the PVC or pod spec mentions the policy, so the decision lives in a cluster-scoped object nobody reviewing the workload ever sees.

When it breaks

The classic failure is the claim that will not die. Someone deletes a PVC, sees Terminating, and waits — while a pod somewhere still mounts it, usually a StatefulSet that was never scaled down or a pod stuck on an unreachable node. Nothing errors, nothing times out, and the PVC blocks the storage behind it. Find the pod rather than deleting the finalizer by hand; bypassing protection can permit premature storage deletion while a workload still uses it.

The second failure is the Retain orphan. The claim is gone, the pods are gone, the namespace may be gone, and a Released PV plus its storage asset remain for deliberate reclamation. Retention may be intentional; without inventory alerts it is easy to miss and the volume does not appear in any workload’s list. The inverse mistake is just as quiet: a class whose default Delete policy was never overridden, discovered only when someone tries to recover data that the reclaim already removed.

Check it quickly

Look at both sides of the binding, and at what holds the claim:

kubectl get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.phase,VOLUME:.spec.volumeName,CLASS:.spec.storageClassName
kubectl get pv -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,POLICY:.spec.persistentVolumeReclaimPolicy,CLAIM:.spec.claimRef.name
kubectl get pvc data -o jsonpath='{.metadata.finalizers}{"\n"}'
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="data") | "\(.metadata.namespace)/\(.metadata.name)"'

These retained diagnostics were not rerun here. Protection indicates cleanup is pending; it does not prove the controller is healthy or a Pod still exists. The fourth command finds all claims named data across namespaces, so match the target namespace before treating a Pod as a consumer. Released plus Retain calls for an ownership and recovery decision, not automatic deletion; the old claimRef may remain.

Sources

Persistent Volumes, binding, access modes, reclaim policies, Released state and the pvc/pv protection finalizers, checked 2026-10-15.
Storage Classes, dynamic provisioning, reclaimPolicy, and volumeBindingMode, checked 2026-10-15.
Persistent Volumes (v1.34), v1.34 prebinding, access-mode limits and supported reclaim behavior, checked 2026-10-18.
PVC protection implementation, v1.34.0 podUsesPVC node-assignment and explicit-reference checks, checked 2026-10-18.

The HPA grows and shrinks a Deployment by load

Wed, 11 Aug 2027 00:00:00 +0000

A HorizontalPodAutoscaler is a feedback controller, not a capacity planner. Its default synchronization interval is fifteen seconds. Depending on its configuration, it reads resource, custom or external metrics and updates a workload’s scale subresource. The basic calculation is desiredReplicas = ceil(currentReplicas * currentMetricValue / desiredMetricValue), followed by handling for missing data, readiness, tolerances, stabilization and bounds.

A plausible HPA object does not prove that capacity will arrive. Metrics availability, requests, replica limits, scheduling and application startup all influence the result. Inspect the recommendation and the actual ready capacity separately; increasing desired replicas cannot manufacture nodes or make a blocked rollout succeed.

Illustrative arithmetic from the article, not an observed scale event. Retained read-only Kubernetes diagnostics remain blocked and were not rerun. A recommendation of four does not establish four ready Pods or sufficient application capacity.

The ratio and the dead band

For a CPU utilization target, observed CPU use is compared with the relevant CPU requests, producing a percentage. The requested target is also a percentage. With identical single-container Pods requesting 200m and targeting 60%, 120m per Pod is a useful simplified equivalent. It is not the general denominator for heterogeneous Pods or every metric type.

The cluster default tolerance is 0.1 around a ratio of one. In Kubernetes 1.34, per-direction tolerance is alpha and disabled by default; check the feature gate and controller configuration before assuming an HPA can override it. Treat that as a control dead band, not a promise that 132m never scales and 133m always does: aggregation, integer utilization, readiness and implementation arithmetic affect boundary cases. In the unadjusted formula, three replicas at 160m against a 120m target recommend four. Rounding, policies and actual scheduling still need separate inspection.

Missing metrics and the veto

Missing Pod metrics are handled conservatively. In Kubernetes 1.34.0, GetResourceReplicas fills a missing downscale sample with that Pod’s resource request multiplied by max(100%, target utilization). The plain per-Pod metric branch, calcPlainMetricReplicas, instead uses the target amount. Both substitute zero for scale-up. A 200m request and 60% utilization target therefore give a 200m downscale substitute, not 120m. Recalculation can cancel the action; displayed utilization still uses the original available samples. For CPU, not-yet-ready Pods receive additional treatment, with configurable readiness-delay and initialization windows. These are rules about startup samples and readiness transitions, not a blanket statement that every Pod is ignored for exactly five minutes.

A CPU utilization calculation needs the relevant resource requests. A missing request can make that metric unusable, even when kubectl top reports usage. With multiple configured metrics, HPA takes the largest proposed replica count; an unusable metric can block a proposed scale-down while another valid metric can still justify scale-up. Missing one source therefore does not mean the entire controller always stops.

Stabilisation and rate limits

By default, scale-down considers a five-minute recommendation window and selects the highest recommendation still relevant there. This reduces oscillation; it is not an exact five-minute delay measured from an external load graph. Recommendations, the current replica count, bounds and configured behavior determine when reduction occurs. Default scale-up stabilization is zero, but users can configure different behavior.

Default scale-up policies allow the larger of four Pods or a doubling over fifteen seconds; default scale-down policies permit a full reduction within their period, subject to stabilization and the minimum replica count. These are permitted rates, not an instruction to grow from one to five to ten automatically. The metric must request that growth and maxReplicas still caps it.

When it breaks

Unknown utilization can come from missing requests, unavailable metrics, authorization or other fetch and calculation errors. Read ScalingActive and its reason rather than treating unknown as proof that no CPU samples exist. Check all relevant containers; a first-container query cannot establish the whole Pod’s denominator.

A workload at maxReplicas can remain overloaded even while HPA works as configured. ScalingLimited explains clipping, but capacity planning also needs node headroom, startup delay and downstream limits. Raising the cap may create Pending Pods or overwhelm a shared dependency. Compare desired replicas with available replicas and observed service latency before declaring autoscaling successful.

Check it quickly

These retained read-only diagnostics were not rerun in this review. Select an authorized test cluster and namespace; inspect the complete HPA and every relevant container alongside this abbreviated view.

Confirm that a metric exists, that a denominator exists, and what the controller decided:

kubectl get hpa web -o wide
kubectl top pods -l app=web
kubectl get deploy web -o jsonpath='{.spec.template.spec.containers[0].resources.requests.cpu}{"\n"}'
kubectl get hpa web -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
kubectl describe hpa web | sed -n '/Events/,$p'

The first line shows current against target utilization; <unknown> means the requested value could not be established, not necessarily that raw usage is absent. The third prints nothing when no CPU request is set, which is the usual reason the second command works while the HPA still refuses to scale. The last two name the condition reason and any FailedGetResourceMetric event, which is where the real explanation lives.

Sources

Horizontal Pod Autoscaling, the algorithm, tolerance, missing-metric handling, stabilisation window and default policies, checked 2026-10-15.
HPA walkthrough, the metrics-server dependency and the requests prerequisite, checked 2026-10-15.
metrics-server, the resource metrics API implementation the HPA reads, checked 2026-10-15.
Kubernetes 1.34 HPA, version-scoped tolerance and conservative metric handling, checked 2026-10-18.
Kubernetes v1.34.0 replica calculator, distinguishes resource-utilization and plain per-Pod missing-sample substitutions, checked 2026-10-18.

Liveness and readiness tell Kubernetes how the Pod feels

Wed, 14 Jul 2027 00:00:00 +0000

A slow start is not a dead process. Readiness and liveness make independent decisions: whether ordinary Service traffic should reach this Pod, and whether terminating this container can help it recover. Readiness failure does not suspend liveness. A startup probe provides that startup protection when configured.

This article focuses on choosing useful health signals and diagnosing bad wiring. Endpoint names are less important than what each check measures. Sharing a URL can work; a shared dependency check that triggers unnecessary restarts can make an outage worse. Test the failure mode and intended recovery, rather than assuming two endpoints automatically make the configuration safe.

Ordinary Service routing converges asynchronously; existing connections and direct Pod access need separate consideration.

Readiness gates traffic

Repeated readiness failures make a Pod ineligible for ordinary Service routing but leave it running, so the process can warm up or recover. Endpoint conditions and the datapath converge asynchronously in both directions; existing connections and direct Pod access need separate consideration. Readiness failure avoids a probe-triggered restart, but can remove all serving capacity: caches can warm, migrations can finish, downstream dependencies can come back, all without losing the Pod’s in-memory state or paying a restart penalty.

Design the readiness endpoint around genuine servability. Check what makes this Pod able to answer correctly right now: migrations applied, required caches populated, critical downstream reachable. A probe that only checks “the HTTP server responds” admits traffic to Pods that will fail every request. But resist stuffing every dependency into it: if a non-critical recommendation service is down, failing readiness sheds real traffic over a decoration.

Liveness guards the process

Repeated liveness failures reach a threshold and terminate the container; restart follows policy. This is a last resort for a hopeless wedge. Use it for what it is a liveness issue—not for containers that are merely slow to become ready. The legitimate targets are narrow: deadlocks where no request will ever succeed again, runaway goroutine or thread leaks that only a fresh process clears, corrupted in-memory state that no retry can heal. In each case the defining property is irreversibility — without a restart, recovery is impossible, not merely slow.

That bar means many services should ship a minimal liveness probe or none at all. A liveness check against an external database turns every database blip into a fleet-wide restart storm. Prefer a cheap local check — a watchdog timestamp refreshed by the main loop — that detects “this process is wedged” without consulting anything outside the process. Container restart policy determines recovery after crashes. Liveness can detect selected unrecoverable local failures before ordinary exit occurs.

Give the probes grace

Set the initialDelaySeconds high enough that a cold start is not counted as a crash, and keep timeoutSeconds consistent with real work. Configure startup protection independently: readiness alone cannot prevent liveness-driven termination. Better still, add a startup probe for slow-booting containers: until it succeeds, both liveness and readiness stay disabled, so version upgrades that warm large caches get minutes instead of seconds without weakening steady-state liveness at all.

Tune thresholds from observed behavior, not round numbers copied between services. periodSeconds times failureThreshold is a planning estimate, not an exact reaction time. Probe phase, timeout, scheduling, termination grace and restart backoff alter the observed timeline. Measure cold-start p99, dependency recovery time, and worst-case pauses, then set each number to react faster than the alert but slower than the noise.

When it breaks

The signature failure is the restart loop during dependency outages: the database slows, readiness would have sufficed, but liveness probes share the same failing endpoint and the whole fleet restarts in waves. Each restart wipes warm caches, making the next wave slower to recover, until the system oscillates indefinitely. Diagnose by correlating restart timestamps with the dependency incident; repair the decision semantics. Keep restart signals local and recovery-relevant. Dependency-aware readiness may help, but withdrawing every replica can be worse than serving degraded responses during a shared outage.

The mirror failure is the zombie Pod: liveness passes because the HTTP server responds, but every request hangs behind a deadlocked worker pool. Traffic keeps flowing to a Pod that answers nothing. A short worker-slot timeout cannot distinguish deadlock from healthy saturation and can trigger overload-driven restarts. Prefer evidence of stalled critical worker progress that accounts for expected pauses and idle periods; validate that restarting improves this specific failure. Handle temporary overload separately.

Check it quickly

Inspect probe wiring and recent kills:

kubectl get pod web-0 -o jsonpath='{.spec.containers[*].livenessProbe}{"\n"}{.spec.containers[*].readinessProbe}{"\n"}'
kubectl get events --field-selector reason=Unhealthy --sort-by=.lastTimestamp | tail -n 10
kubectl describe pod web-0 | grep -A3 -i 'liveness\|readiness'

These unchanged diagnostics were not rerun here. Shared URLs are not inherently wrong if their semantics fit both decisions. Unhealthy events include probe failures, not proof of a kill. Correlate the named Pod, container restart count and termination reason; bounded checks must distinguish deadlock from temporary load and harmless idleness.

Sources

Kubernetes probes, Official docs on liveness and readiness, checked 2026-10-15.
Configure liveness and readiness probes, task guide for probe configuration, checked 2026-10-15.

Kubernetes · XB Field Notes

The scheduler accounts for requests, not momentary utilization

Account for the admitted specification

When it breaks

Check it quickly

Sources

Kubernetes CronJobs need a history policy

Separate history from log retention

Count and age are independent policies

Ownership determines cascading behavior

When it breaks

Check it quickly

Sources

cgroups enforce container budgets within a hierarchy

Read the controller that enforces the resource

When it breaks

Check it quickly

Sources

Cordon and drain coordinate node maintenance

Cordon changes future placement

Drain has filters and independent operations

Budgets govern voluntary eviction

When it breaks

Check it quickly

Sources

The Horizontal Pod Autoscaler depends on metric semantics

Separate measurement from control

Pick a signal replicas can improve

Give replica ownership one home

When it breaks

Check it quickly

Further reading

Sources

A CronJob turns a schedule into Jobs

A schedule makes a Job

Concurrency is scoped to one CronJob

Deadlines, suspension and history

When it breaks

Check it quickly

Sources

Node affinity places workloads on matching nodes

Match nodes by label

Prefer or require

Anti-affinity splits replicas

When it breaks

Check it quickly

Sources

Kubernetes rolling updates replace pods gradually

Replace gradually or stop first

Compatibility is the deciding constraint

Rollback is another change

When it breaks

Check it quickly

Further reading

Sources

Security context defines constraints that need runtime evidence

Put each setting at the right scope

When it breaks

Check it quickly

Sources

The horizontal autoscaler scales on demand

Metrics produce recommendations

Stabilize recommendations, not the last resize

Choose pressure that extra replicas relieve

When it breaks

Check it quickly

Further reading

Sources

Kubernetes labels define selection, not authorization

Know which selector language is in use

When it breaks

Check it quickly

Sources

Namespaces scope objects; policies establish other boundaries

Names are scoped by resource type

When it breaks

Check it quickly

Sources

Kubernetes Services give pods a stable home

One address in front of many