Availability · XB Field Notes

A PodDisruptionBudget limits eviction-based disruption

Sat, 10 Apr 2027 00:00:00 +0000

A PodDisruptionBudget limits eviction-based disruption for a selected group of Pods. It can help preserve serving capacity or quorum during maintenance, but cannot guarantee either. Its protection depends on the caller using the Eviction API, the selector identifying the intended members, and Ready conditions meaning something useful for the application.

Choose the permitted disruption from the application’s failure model. A three-member majority system normally needs two functioning members, but Pod readiness does not itself prove membership, synchronized state, or connectivity between those members. The PDB complements application health and maintenance procedures; it does not replace them.

A PDB constrains a specific eviction path. Verify the selected members, readiness, policy and application health before treating an allowance as safe maintenance capacity.

The boundary is eviction

Node crashes, network partitions, and node-pressure eviction are not prevented by PDBs. Unavailable members can nevertheless reduce the remaining disruption allowance. Directly deleting a Pod or workload bypasses PDB enforcement, as do a workload controller’s normal rolling-update deletions. Voluntary does not automatically mean budget-protected.

A normal drain uses eviction requests and retries when protection blocks them. A caller that chooses direct deletion bypasses that protection. Review the maintenance tool’s behavior before relying on a budget. Deployment and StatefulSet rolling availability is controlled by their own update strategies, so coordinate rollouts and node maintenance instead of assuming one PDB serializes everything.

Read the arithmetic with its denominator

Set either minAvailable or maxUnavailable. For three healthy replicas, an integer minAvailable of two allows one healthy eviction under ordinary settled conditions. maxUnavailable of one expresses an allowance relative to intended scale. CurrentHealthy counts qualifying Ready Pods, excluding members being deleted or already recorded for disruption; Running alone is insufficient.

ExpectedPods is not always the controller’s desired replicas. In the reviewed Kubernetes 1.34.0 implementation, integer minAvailable uses the number of selected Pod objects. Percentage minAvailable and maxUnavailable derive expected scale from owning controllers. Therefore a mismatch with StatefulSet replicas can reflect policy type or reconciliation state, not only a selector typo.

The status allowance is broadly currentHealthy minus desiredHealthy, floored at zero, with additional safety checks and outstanding-eviction accounting. Inspect observedGeneration and conditions when interpreting it. Zero is a current decision, not a diagnosis. It can mean an intentionally strict policy, unavailable Pods, an in-progress eviction, or a controller synchronization problem.

Percentages round upward. Half of seven requires four healthy Pods, while maxUnavailable thirty percent on one desired replica permits losing that single replica. Percentage notation does not automatically make a small workload safe. Check the smallest scale your application can reach, including autoscaling and maintenance changes.

Health policy changes drain behavior

Under the default IfHealthyBudget policy, a Running but unready Pod can be evicted when currentHealthy is at least desiredHealthy. If the application is already below that threshold, such a Pod can block a drain while it has a chance to recover. The PDB sees its unhealthy state; it is not blind to the Pod.

AlwaysAllow permits eviction of those Running-but-unhealthy Pods regardless of that health threshold. It does not authorize unrestricted eviction of healthy Pods. Select the policy with recovery and maintenance needs in mind, especially when a broken readiness check could keep a replacement from ever becoming healthy.

For three healthy voting members, allowing one disruption is a common conservative choice. Recalculate quorum and availability when membership changes; an allowance of one is not suitable for every possible scale or membership protocol. Maintenance may wait for a replacement to become genuinely ready before another healthy eviction fits within the budget.

When it breaks

A minimum equal to all available replicas can deliberately block healthy evictions, rather than necessarily being a mistake. Document how maintenance should obtain room. Do not remove the budget simply because a drain is slow: first check replacement scheduling, readiness, existing disruption, and the application owner’s availability requirement.

A selector matching nothing protects nothing. Conversely, an empty policy/v1 selector matches every Pod in its namespace, which can broaden scope unexpectedly. Compare the complete selector, including matchExpressions, against live labels and owning controllers. Avoid treating a single status number as proof that the right application is protected.

Check it quickly

Inspect existing objects in the intended read-only context and namespace. These original commands were retained and were not rerun on a cluster during this review.

kubectl get pdb -A -o custom-columns=NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,EXPECTED:.status.expectedPods,HEALTHY:.status.currentHealthy,ALLOWED:.status.disruptionsAllowed
kubectl get pdb zk-pdb -o jsonpath='{.spec.selector.matchLabels}{"\n"}'
kubectl get statefulset zookeeper -o jsonpath='{.spec.replicas}{"\n"}'
kubectl get pods -l app=zookeeper --field-selector=status.phase=Running --no-headers | wc -l

The final command counts Running Pods, not Ready Pods. The selector command displays only matchLabels, so it omits any matchExpressions. Use those outputs as initial clues and inspect complete PDB and Pod objects before concluding that HEALTHY or EXPECTED is wrong. A zero allowance does not reveal its cause, and positive allowance does not prove application-level quorum survives an eviction.

Sources

Configure a PodDisruptionBudget, selectors, rounding and unhealthy-Pod policies, checked 2026-10-15.
Pod disruptions, eviction boundaries, involuntary failures and rolling updates, checked 2026-10-15.
Kubernetes 1.34.0 disruption controller, getExpectedPodCount and countHealthyPods distinguish integer minimums, controller scale and outstanding disruptions, checked 2026-10-15.

Pod anti-affinity spreads peers away from each other

Thu, 18 Mar 2027 00:00:00 +0000

Pod anti-affinity expresses where a Pod should avoid selected peers. A hostname topology can reduce shared-node exposure; a zone topology can address a broader failure domain. The rule does not create extra capacity or prove application availability. It shapes placement, while replication, readiness, storage, and failure recovery remain separate responsibilities.

Required rules filter candidates; preferred rules contribute scores. If hard constraints leave no feasible node, the new Pod remains Pending until something relevant changes. That can be the correct outcome for a strict isolation requirement, but it must be included in capacity planning rather than discovered during an incident.

Required anti-affinity excludes selected peers from an occupied topology domain. Topology spread instead bounds a candidate placement’s count difference; both depend on selectors, namespace scope, eligible domains and other constraints.

Select peers and define their domains

The label selector identifies peer Pods, and topologyKey names the node label used to group their locations. kubernetes.io/hostname commonly distinguishes nodes; topology.kubernetes.io/zone groups nodes into zones. Check the actual values rather than assuming every infrastructure provider supplies an identical label surface.

Namespace scope is part of the rule. With neither an explicit namespace list nor a namespace selector, matching defaults to the incoming Pod’s namespace. An empty namespaceSelector object matches all namespaces. Thus cross-namespace influence is possible when configured; the feature is not inherently a tenant-isolation boundary. Review both peer labels and namespace selection together.

All relevant nodes need consistent topology labels. Kubernetes warns that missing labels cause unintended anti-affinity behavior. Do not collapse that warning into a universal claim that every unlabelled node is always accepted or always excluded. Inter-pod rules, existing placements, and topology spread are different mechanisms; inspect the rule and actual candidate set.

A hard rule can cap placement

For replicas that select each other, required hostname anti-affinity generally permits only one matching replica per eligible hostname domain. Nine such replicas across six eligible nodes leave three waiting, even if each node has spare CPU. Rollout surge Pods can hit the same cap before ordinary replica growth does.

Required zone anti-affinity would likewise restrict matching peers per zone, but admission configuration matters. LimitPodHardAntiAffinityTopology restricts required anti-affinity to the hostname key when enabled. Do not assume a zone-level hard rule will be admitted by every cluster. Check the cluster’s permitted topology before designing availability around it.

IgnoredDuringExecution means these scheduling rules do not automatically evict or redistribute already-running Pods after the situation changes. A workload-template update affects replacement Pods; it does not instantly rebalance existing ones. Existing hard anti-affinity can also constrain incoming Pods, so examining only the new Pod’s preferred term is insufficient.

The Kubernetes 1.34.0 scoring plugin also considers existing Pods’ preferred terms; an option can skip that work when the incoming Pod has no inter-pod preferences. Existing preferences are therefore not universally ignored.

Use spread constraints for bounded imbalance

Topology spread expresses a different contract from one-peer-per-domain exclusion. With DoNotSchedule, maxSkew limits a candidate domain’s matching Pod count relative to the global minimum across eligible domains. With ScheduleAnyway, reducing skew is a preference. Multiple constraints combine, so every hard constraint must be satisfiable.

For equally eligible zones with two and one matching Pods and maxSkew one, adding to the fuller zone would make the relevant difference two; adding to the emptier zone preserves balance. If eligible domains fall below minDomains, the global minimum becomes zero. It is not raised. That option is valid with DoNotSchedule and can deliberately prevent further concentration.

Balanced counts depend on labels, selectors, eligibility, and the sequence of placements; they are not an unconditional promise of three-three-three for nine replicas. Inter-pod affinity can impose significant scheduling cost in large clusters. Prefer the mechanism that expresses the intended contract clearly, and measure scheduler performance rather than claiming a universal speed comparison.

When it breaks

Too few eligible domains can leave extra replicas Pending while HPA continues recommending growth. Adding a node in an already occupied zone does not resolve a one-per-zone rule. Relaxing a hard rule changes the availability contract; choose that trade-off deliberately instead of treating it as a harmless scheduling fix.

Missing topology labels and an incorrect peer selector can make intended separation meaningless. A rule selecting no peers proves nothing, while overly broad selection can block unrelated workloads. Inspect namespace scope and labels before attributing every uneven distribution to a scheduler failure.

Check it quickly

Use the intended read-only context and namespace. These existing commands were preserved and were not rerun against a cluster during this review.

kubectl get nodes -L topology.kubernetes.io/zone -L kubernetes.io/hostname
kubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,PHASE:.status.phase
kubectl get events --field-selector reason=FailedScheduling --sort-by=.lastTimestamp | tail -5
kubectl get pods -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort | uniq -c

The Pod listings cover the whole namespace, not necessarily the rule’s peer set. Blank node names include unscheduled Pods; blank topology cells identify missing labels. Filter the observations conceptually by the actual selector and namespace scope, then compare domains and events. One Pod per node does not prove anti-affinity caused the distribution, and uneven counts do not alone prove a spread violation.

Sources

Assigning Pods to Nodes, inter-pod affinity and anti-affinity rules, topology keys and scheduling behaviour, checked 2026-10-15.
Pod Topology Spread Constraints, maxSkew, whenUnsatisfiable, minDomains and the worked skew example, checked 2026-10-15.
Kubernetes 1.34.0 inter-Pod affinity scoring, processExistingPod and IgnorePreferredTermsOfExistingPods, checked 2026-10-15.

Readiness changes routing eligibility; liveness restarts a container

Wed, 13 Jan 2027 00:00:00 +0000

Readiness and liveness answer different operational questions. Readiness asks whether this container should contribute to a Pod that accepts ordinary Service traffic now. Liveness asks whether restarting this container is the configured recovery action. Both can use HTTP, but sharing a protocol does not make their meanings interchangeable. Define the action you want before choosing the condition that triggers it.

Kubernetes 1.34.11 observations are historical and were not rerun. The direct HTTP check was from the same Pod to its own address. Readiness is not network access control; the fixture does not prove graceful draining or production availability. Execution remains blocked.

Observe the action as well as the response

A readiness failure makes the container unready and affects Pod readiness. For a normal selector-based Service, endpoint reconciliation then changes which backends are eligible for ordinary routing. The Pod can remain Running while it is not Ready, and the application process may continue executing. Restart count and Pod UID answer different questions from the Ready condition.

Liveness failures reaching the configured threshold cause kubelet to terminate the container and apply its restart behavior. In a usual Always-policy application Pod this produces a replacement container inside the same Pod. It does not inherently create a new Pod object or deploy another application version. Compare container restart count and termination state alongside the Pod identity.

Use inexpensive checks with meaningful success criteria. A TCP connection proves that a listener accepts connections, not that an application can complete useful work. An HTTP endpoint can answer a richer question, but can also create load if it recursively checks expensive dependencies. Keep the probe’s own execution cost within the workload’s capacity and timeout budget.

When it breaks

Treating a shared database outage as a liveness failure can repeatedly restart every healthy application process without repairing the database. Adding that dependency to readiness also needs judgment: withdrawing every replica may remove useful fallback responses or overload the remaining replicas during a partial failure. Define readiness from the service contract, including any deliberately supported degraded behavior.

Readiness is not a network access control. A Pod address can remain reachable when the Pod has no ready Service target. Existing connections, other routing layers, and Service options such as publishNotReadyAddresses also require their own analysis. Do not describe a failed readiness probe as an instantaneous guarantee that no request can ever reach the process.

Thresholds and timing control sensitivity. A short timeout can turn CPU contention into apparent failure; several consecutive failures can avoid reacting to one brief delay. Startup protection is separate: a configured startup probe suppresses readiness and liveness checks until it succeeds. These settings are sampling and recovery policy, not a proof of zero lost requests during termination.

Check it quickly

Use a disposable Kubernetes cluster with kubectl access and python:3.11-slim-bookworm available on its nodes. Save the code as probe-actions.py, then run python3 probe-actions.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The original article records Kubernetes 1.34.11. This review did not rerun the cluster fixture. The script creates one HTTP Pod and an ordinary Service with default endpoint readiness behavior.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-probes-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 server = '''from http.server import BaseHTTPRequestHandler,HTTPServer
from pathlib import Path
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 flag={'/live':'/tmp/live-fail','/ready':'/tmp/ready-fail'}.get(self.path)
 code=503 if flag and Path(flag).exists() else 200
 body=b'worker'; self.send_response(code); self.send_header('Content-Length',str(len(body)))
 self.end_headers(); self.wfile.write(body)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''
 podspec = spec(server)
 c = podspec['containers'][0]
 c['readinessProbe'] = {'httpGet': {'path': '/ready', 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 2}
 c['livenessProbe'] = {'httpGet': {'path': '/live', 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 3}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'worker', 'labels': {'app': 'worker'}}, 'spec': podspec})
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'web'},
 'spec': {'selector': {'app': 'worker'}, 'ports': [{'port': 80, 'targetPort': 8080}]}})
 def targets():
 return {e['targetRef']['uid'] for s in obj(namespace, 'endpointslices', selector='kubernetes.io/service-name=web')['items']
 for e in s.get('endpoints', []) if e.get('conditions', {}).get('ready') and e.get('targetRef')}
 initial = wait_for(lambda: obj(namespace, 'pod', 'worker'), pod_ready)
 uid = initial['metadata']['uid']
 wait_for(targets, lambda value: value == {uid})
 def change(path, create):
 method = 'touch' if create else 'unlink'
 k('exec', '-n', namespace, 'worker', '--', 'python3', '-c',
 'from pathlib import Path; Path('+repr(path)+').'+method+'()')
 change('/tmp/ready-fail', True)
 wait_for(lambda: obj(namespace, 'pod', 'worker'), lambda p: not pod_ready(p))
 wait_for(targets, lambda value: not value)
 time.sleep(4)
 unready = obj(namespace, 'pod', 'worker')
 assert unready['metadata']['uid'] == uid and unready['status']['containerStatuses'][0]['restartCount'] == 0
 address = 'http://' + unready['status']['podIP'] + ':8080/api'
 probe = 'from urllib.request import urlopen; print(urlopen('+repr(address)+',timeout=3).read().decode())'
 assert k('exec', '-n', namespace, 'worker', '--', 'python3', '-c', probe).stdout.strip() == 'worker'
 change('/tmp/ready-fail', False)
 wait_for(targets, lambda value: value == {uid})
 change('/tmp/live-fail', True)
 restarted = wait_for(lambda: obj(namespace, 'pod', 'worker'), lambda p:
 pod_ready(p) and p['status']['containerStatuses'][0]['restartCount'] >= 1)
 assert restarted['metadata']['uid'] == uid
 assert restarted['status']['containerStatuses'][0]['restartCount'] == 1
 wait_for(targets, lambda value: value == {uid})
 print(json.dumps({'readiness_removed_ready_target_without_restart': True,
 'unready_pod_direct_http_still_succeeds': True,
 'readiness_recovery_restores_target': True,
 'liveness_container_restarts': 1, 'same_pod_uid': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The initial Pod is Ready and its UID appears in the Service’s ready EndpointSlice targets. Creating a local flag makes only the readiness endpoint return 503. After the failure threshold and reconciliation, the ready-target set is empty. Four seconds later the Pod UID is unchanged and the container restart count remains zero.

A direct HTTP request from inside that same Pod to its unready Pod address still returns worker. This does not test reachability from another Pod or an external client. Removing the readiness flag restores its ready target. The script then creates a different flag that makes the liveness endpoint fail. Kubelet restarts the container once, its fresh writable layer removes the flag, and the ready target returns. The Pod UID remains the same throughout.

The original article reports that all assertions passed and the namespace was removed; this is historical execution attribution. This demonstrates distinct controller actions and direct reachability, not graceful draining of existing connections or production availability. For a real rollout, verify endpoint propagation, application shutdown, outstanding work and client behavior together; a green probe alone cannot establish the whole request lifecycle.

Sources

Kubernetes: Configure Probes, failure thresholds and distinct readiness and liveness actions, checked 2026-10-15.
Kubernetes: Pod Lifecycle, container restarts and Pod readiness, checked 2026-10-15.
Kubernetes: EndpointSlices, backend readiness conditions and discovery behavior, checked 2026-10-15.

Passive upstream health follows failed traffic, not a probe timer

Tue, 05 Jan 2027 00:00:00 +0000

nginx passive upstream health uses outcomes of real requests to decide whether a peer should temporarily stop receiving traffic. max_fails sets an unsuccessful-attempt threshold, while fail_timeout supplies both the counting interval and the unavailability period. These settings do not schedule periodic application probes. Active checks require a supported checking capability and configuration beyond this passive request-outcome policy.

When it breaks

Not every error-looking HTTP response counts as an unsuccessful upstream attempt. The proxy retry configuration helps define that classification. In the tested setup, HTTP 503 only contributes to passive failure handling when http_503 is included. Treating every 5xx as an automatic health failure can leave an operator expecting exclusion while nginx continues selecting the peer according to its actual rules.

Another mistake is treating the timeout window as proof of recovery. When the unavailable period ends, a peer can become eligible for real traffic again without a successful independent health probe first. A still-broken backend can make that request fail. least_conn changes selection based on active connections; it is not a gradual recovery ramp or a replacement for a supported slow-start mechanism.

Retries have boundaries as well. Switching peers cannot transparently repair an error after response data has already been sent to the client. Replaying an operation with side effects can create duplicates. The following fixture uses GET requests and failures returned before a successful body, with an explicit attempt limit; it does not establish safe retry semantics for every application method.

The exclusion window is not a recovery probe. Four B-only requests depend on the two-second window; live recovery, active checking and safe retry remain distinct.

Check it quickly

These are historical article reports, not a rerun; raw integration logs were not recovered in the retained validation files. Use locally available pinned images with assertions enabled; cleanup is best-effort.

Save this as check-passive-health.py and run it with Python, Docker, and the pinned images. The recorded server is nginx 1.28.0 with one worker. Two HTTP endpoints in a disposable container report their identities and can be instructed to return 503. A separate control listener exposes counts without generating requests through the balanced groups:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None,extra_ports=()):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
 target.write_bytes(content if isinstance(content,bytes) else content.encode())
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 published=[arg for value in (8080,*extra_ports) for arg in ('-p','127.0.0.1::'+str(value))]
 docker('run','-d','--name',containers[0],'--network',name,*published,tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 request.port=port;request.network=name
 request.ports={int(key.split('/')[0]):int(value[0]['HostPort']) for key,value in details['NetworkSettings']['Ports'].items() if value}
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
backend=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
from urllib.parse import urlsplit,parse_qs
import itertools,json,threading
lock=threading.Lock();ids=itertools.count(1);counts={'A':0,'B':0};active={'A':0,'B':0};faults={'A':200,'B':200};gates={}
class Handler(BaseHTTPRequestHandler):
 protocol_version='HTTP/1.1'
 def setup(self):super().setup();self.ident=next(ids)
 def log_message(self,*args):pass
 def do_GET(self):
 parsed=urlsplit(self.path);label=self.server.label;code=200
 if label=='control':
 query=parse_qs(parsed.query)
 with lock:
 if parsed.path=='/fail':faults[query['peer'][0]]=int(query['status'][0])
 if parsed.path=='/release':gates[query['path'][0]].set()
 data={'counts':dict(counts),'active':dict(active),'faults':dict(faults),'held':list(gates)}
 else:
 with lock:
 counts[label]+=1;active[label]+=1;code=faults[label]
 if parsed.path.startswith('/hold/'):gates[parsed.path]=threading.Event()
 if parsed.path.startswith('/hold/'):assert gates[parsed.path].wait(20)
 data={'peer':label,'path':self.path,'connection':self.ident}
 body=json.dumps(data,sort_keys=True).encode();self.send_response(code)
 self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(body)))
 self.end_headers();self.wfile.write(body)
 if label!='control':
 with lock:active[label]-=1
for port,label in [(8081,'A'),(8082,'B'),(8083,'control')]:
 server=ThreadingHTTPServer(('0.0.0.0',port),Handler);server.label=label
 threading.Thread(target=server.serve_forever,daemon=True).start()
threading.Event().wait()
'''
extra='''
upstream passive { server backend:8081 max_fails=1 fail_timeout=2s; server backend:8082; }
upstream plain { server backend:8081 max_fails=1 fail_timeout=30s; server backend:8082; }
upstream single { server backend:8081 max_fails=1 fail_timeout=30s; }
add_header X-Upstream $upstream_addr always; add_header X-Attempts $upstream_status always;
'''
locations='''
location /passive { proxy_next_upstream error timeout http_503; proxy_next_upstream_tries 2; proxy_pass http://passive; }
location /plain { proxy_next_upstream error timeout; proxy_pass http://plain; }
location /single { proxy_next_upstream error timeout http_503; proxy_pass http://single; }
location /control/ { proxy_pass http://backend:8083/; }
'''
active_config=config('location / { proxy_pass http://peers; health_check; }',
 'upstream peers { zone peers 64k; server backend:8081; server backend:8082; }')
with lab(config(locations,extra),{'active.conf':active_config},backend) as (proxy,request):
 def control(path):
 status,_,body=request('/control/'+path);assert status==200;return json.loads(body)
 def call(path):
 status,h,b=request(path);return {'status':status,'attempts':h['x-attempts'],'upstream':h['x-upstream'],'peer':json.loads(b)['peer']}
 control('fail?peer=A&status=503');results={}
 plain=[call('/plain') for _ in range(6)]
 assert [r['status'] for r in plain].count(503)==3 and all(',' not in r['attempts'] for r in plain)
 results['unlisted_http_503']=plain
 for _ in range(2):
 first=call('/passive')
 if first['attempts']=='503, 200':break
 assert first['status']==200 and first['attempts']=='503, 200' and first['peer']=='B',first
 results['failed_attempt_then_success']=first
 after_failure=control('state')['counts']['A']
 excluded=[call('/passive') for _ in range(4)]
 assert all(r['peer']=='B' and r['attempts']=='200' for r in excluded)
 assert control('state')['counts']['A']==after_failure
 time.sleep(.4);assert control('state')['counts']['A']==after_failure
 results['during_exclusion']=excluded;results['background_probe_count']=0
 control('fail?peer=A&status=200');deadline=time.monotonic()+6;recovered=None
 while time.monotonic()<deadline:
 observed=call('/passive')
 if observed['peer']=='A':recovered=observed;break
 time.sleep(.1)
 assert recovered and recovered['status']==200;results['live_request_after_window']=recovered
 control('fail?peer=A&status=503');before=control('state')['counts']['A']
 single=[call('/single') for _ in range(2)]
 assert all(r['status']==503 and r['attempts']=='503' for r in single)
 assert control('state')['counts']['A']==before+2;results['single_peer_still_attempted']=single
 probe=docker('exec',proxy,'nginx','-t','-c','/srv/active.conf',check=False)
 assert probe.returncode!=0 and 'unknown directive "health_check"' in probe.stderr
 results['active_health_check_directive_supported']=False
 print(json.dumps(results,sort_keys=True))

First, peer A returns actual HTTP 503 while B returns 200. A comparison group retries only connection errors and timeouts. Six requests still select A three times and B three times, with no multi-attempt response. A’s max_fails=1 does not remove it for that unlisted HTTP outcome. The fixture inspects statuses and selected upstream addresses, not only the final response body.

The passive group includes http_503, allows two tries, and uses max_fails=1 fail_timeout=2s for A. A request encounters A’s 503, then succeeds on B with final 200. nginx reports upstream statuses 503, 200 and both addresses. Those values are real backend HTTP responses in this experiment, not synthetic labels inferred from a connection refusal.

Four subsequent requests use only B, provided they finish inside A’s two-second exclusion window. A’s request count remains unchanged, including through a controlled pause, demonstrating that this configuration is not sending background health probes. The backend is then made healthy. After the configured window, a live request reaches A and succeeds; the test waits for that observed selection instead of assuming an exact clock boundary.

A separate group contains only A. Even with max_fails=1 fail_timeout=30s, two requests still reach that failing peer and return 503. nginx ignores those passive-unavailability parameters for a single-server group. There is no alternative peer to select, so the multi-peer exclusion behavior cannot be assumed to create availability where the topology provides none.

Finally, a disposable candidate configuration includes health_check. The tested binary rejects the directive as unknown. This verifies the installed capability boundary directly: the passive settings did not enable active checking. The valid running configuration is retained; no unverified module or commercial capability is represented as available merely because it appears in general documentation.

Design failure detection with the service’s response contract

Choose which outcomes indicate an unusable peer and which are legitimate application responses. An authorization failure or missing object need not mean the server is unhealthy. Passive detection also depends on receiving traffic, so a quiet but broken peer may remain untested until a user request arrives. Active checking, where available, answers a different timing question.

Define a meaningful health endpoint if deploying active checks, and understand which dependencies it exercises. A TCP accept alone does not prove a useful application response. Conversely, an overly broad dependency check can remove every peer for a shared downstream problem. Keep the check’s success condition aligned with the traffic that removing a peer would actually protect.

Observe final client status together with attempt statuses, peer addresses, latency, and exclusion state. A final 200 can conceal a failed first attempt and extra delay. Verify the deployed version’s capabilities, retry limits, and recovery path so the availability claim matches what clients experience rather than what the word “health” suggests.

Sources

nginx upstream module, max_fails, both fail_timeout meanings, single-peer behavior, and slow start, checked 2026-10-15.
nginx proxy module, unsuccessful-attempt classification, retry conditions, and retry limits, checked 2026-10-15.
nginx HTTP health check guide, passive versus active checks and product capabilities, checked 2026-10-15.