Health · XB Field Notes

Consul discovery reflects health policy and freshness

Thu, 06 Apr 2028 00:00:00 +0000

Consul discovery reports registered endpoints and recorded health observations. It cannot promise that every returned endpoint will answer the caller’s next request. A useful operating model separates four things: what is registered, which checks apply, which states the query accepts, and how old those observations may be. A successful lookup establishes none of those guarantees by itself.

Registration and health are separate

The catalog stores registrations; health endpoints add check information. An unfiltered /v1/health/service/web response can include instances with critical checks. A failed check does not automatically erase its service registration. Explicit deregistration and configured critical-service removal are separate lifecycle mechanisms, and ACL filtering can also change which records a reader sees.

Registered checks normally start critical unless an initial status is supplied. That does not mean every registered service necessarily has a meaningful readiness check. For one service instance, consider its own service checks together with node-level checks such as serfHealth. A failed database-service check on the same node does not automatically fail an unrelated web-service check. A critical node check, however, affects services on that node.

Check outcomes are not simply Boolean. Script exit zero means passing, one means warning, and other exit codes mean critical. HTTP checks classify 2xx as passing and 429 as warning after redirect handling; redirects are followed by default unless disable_redirects is set. Other final responses indicate failure. The choice of probe determines what those states actually establish. A TCP handshake or process-existence test does not prove an application can finish a request.

DNS is not always passing-only

The HTTP health endpoint’s passing=true option requires relevant checks to be passing. Consul DNS service lookups use a different default policy: warning instances can remain eligible, while critical instances are excluded. Setting dns_config.only_passing=true excludes warning states too. Therefore, a warning instance appearing in DNS but missing from the passing-only API response need not indicate a broken check.

When comparing outputs, use the same service, datacenter, namespace where applicable, and observation window. Consul normally listens for DNS on port 8600; querying the system resolver without appropriate forwarding is a different path. Querying Consul directly removes that intermediary, but does not turn the answer into a probe of the application from the caller’s network.

Freshness has several stages

Probe intervals, probe timeouts, consecutive-failure thresholds, agent-to-server propagation, read consistency, and client caching can all contribute delay. A 30-second interval is not a guaranteed 30-second end-to-end removal bound. A TTL check expires relative to its most recent update; a failure halfway through its TTL does not necessarily leave a whole additional TTL of eligibility.

Anti-entropy repairs differences between agent state and the catalog. Its documented periodic interval is approximate and depends on cluster size. Registration changes also trigger updates. Failed synchronization can continue across multiple intervals, so the periodic schedule is neither the only update mechanism nor a maximum convergence deadline.

HTTP reads normally use the default consistency mode; DNS normally permits stale reads. A consistent HTTP read verifies leadership with a quorum, at additional cost. Even that stronger catalog read cannot discover a service failure that no check has observed yet. Catalog consistency and real-world readiness remain distinct.

When it breaks

A passing probe may test the wrong dependency, run from a different network, or miss a deadlock. Conversely, coupling every instance’s readiness to one shared dependency can remove the whole pool during a common outage. Choose checks according to useful request capability and the intended degraded behavior, then retain client timeouts and bounded recovery paths.

Do not diagnose an API/DNS mismatch from one screenshot. Check warning policy, relevant check IDs, ACL visibility, deregistration, and cached or stale observations first. A critical instance can remain in the unfiltered health response; its continued presence there is expected, not proof that filtering was ignored.

Consul health selection: mechanism and evidence boundaries.

Check it quickly

Save this as consul_filter.py and run python3 consul_filter.py. This standard-library exercise uses synthetic records, starts no Consul agent, and makes no network requests. Keep assertions enabled, without -O or PYTHONOPTIMIZE. It tests selection and stale snapshots, not Consul itself; this review did not rerun the retained exact fixture.

from copy import deepcopy

nodes = {"n1": "passing", "n2": "critical"}
services = [
 {"ID": "web-1", "Name": "web", "Node": "n1", "Status": "passing"},
 {"ID": "web-2", "Name": "web", "Node": "n1", "Status": "warning"},
 {"ID": "db-1", "Name": "db", "Node": "n1", "Status": "critical"},
 {"ID": "web-3", "Name": "web", "Node": "n2", "Status": "passing"},
]


def health_rows(name):
 # Synthetic snapshot: only the node and THIS service contribute checks.
 return [
 {"ID": item["ID"], "Checks": [nodes[item["Node"]], item["Status"]]}
 for item in services if item["Name"] == name
 ]


def selected(rows, allowed):
 return [row["ID"] for row in rows if all(s in allowed for s in row["Checks"])]


snapshot = health_rows("web")
assert [row["ID"] for row in snapshot] == ["web-1", "web-2", "web-3"]
assert selected(snapshot, {"passing"}) == ["web-1"]
assert selected(snapshot, {"passing", "warning"}) == ["web-1", "web-2"]
print("unfiltered: web-1 web-2 web-3; passing-only: web-1; default DNS policy: web-1 web-2")

previous = deepcopy(snapshot)
services[0]["Status"] = "critical"
current = health_rows("web")
assert selected(previous, {"passing"}) == ["web-1"] # An old snapshot stays old.
assert selected(current, {"passing"}) == []
assert len(current) == 3 # Failing health is not deregistration.
assert selected(current, {"passing", "warning"}) == ["web-2"]
print("service failure changes selection, not registration; an old snapshot still selects web-1")

nodes["n1"] = "critical"
assert selected(health_rows("web"), {"passing", "warning"}) == []
print("node failure excludes both web instances on n1; db-1 never controlled their earlier health")

The assertions keep unrelated database health out of web selection, distinguish warning from critical, and preserve failed registrations in the unfiltered view. Updating current health does not rewrite a previously copied snapshot. Node failure then removes both local web instances from the modeled DNS-eligible set.

For deployment validation, compare actual unfiltered and passing-only health responses with a direct DNS query under the configured warning policy. Record check transitions and client-visible removal separately. That integration was not executed here; the offline model provides no measured detection time, DNS packet evidence, or availability guarantee.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
Consul Health API, service instances, checks, passing filter, and ACL visibility, checked 2026-10-15.
Consul health checks, initial states, script and HTTP outcomes, and TTL expiry, checked 2026-10-15.
Consul DNS configuration, only_passing and relevant checks, checked 2026-10-15.
Consul DNS behavior, port 8600 and caching controls, checked 2026-10-15.
Consul consistency, agent updates and approximate anti-entropy intervals, checked 2026-10-15.
Consul consistency modes, HTTP and DNS defaults and quorum checks, checked 2026-10-15.
Consul health-check reference, timeouts, thresholds, and critical-service deregistration, checked 2026-10-15.

Liveness and readiness tell Kubernetes how the Pod feels

Wed, 14 Jul 2027 00:00:00 +0000

A slow start is not a dead process. Readiness and liveness make independent decisions: whether ordinary Service traffic should reach this Pod, and whether terminating this container can help it recover. Readiness failure does not suspend liveness. A startup probe provides that startup protection when configured.

This article focuses on choosing useful health signals and diagnosing bad wiring. Endpoint names are less important than what each check measures. Sharing a URL can work; a shared dependency check that triggers unnecessary restarts can make an outage worse. Test the failure mode and intended recovery, rather than assuming two endpoints automatically make the configuration safe.

Ordinary Service routing converges asynchronously; existing connections and direct Pod access need separate consideration.

Readiness gates traffic

Repeated readiness failures make a Pod ineligible for ordinary Service routing but leave it running, so the process can warm up or recover. Endpoint conditions and the datapath converge asynchronously in both directions; existing connections and direct Pod access need separate consideration. Readiness failure avoids a probe-triggered restart, but can remove all serving capacity: caches can warm, migrations can finish, downstream dependencies can come back, all without losing the Pod’s in-memory state or paying a restart penalty.

Design the readiness endpoint around genuine servability. Check what makes this Pod able to answer correctly right now: migrations applied, required caches populated, critical downstream reachable. A probe that only checks “the HTTP server responds” admits traffic to Pods that will fail every request. But resist stuffing every dependency into it: if a non-critical recommendation service is down, failing readiness sheds real traffic over a decoration.

Liveness guards the process

Repeated liveness failures reach a threshold and terminate the container; restart follows policy. This is a last resort for a hopeless wedge. Use it for what it is a liveness issue—not for containers that are merely slow to become ready. The legitimate targets are narrow: deadlocks where no request will ever succeed again, runaway goroutine or thread leaks that only a fresh process clears, corrupted in-memory state that no retry can heal. In each case the defining property is irreversibility — without a restart, recovery is impossible, not merely slow.

That bar means many services should ship a minimal liveness probe or none at all. A liveness check against an external database turns every database blip into a fleet-wide restart storm. Prefer a cheap local check — a watchdog timestamp refreshed by the main loop — that detects “this process is wedged” without consulting anything outside the process. Container restart policy determines recovery after crashes. Liveness can detect selected unrecoverable local failures before ordinary exit occurs.

Give the probes grace

Set the initialDelaySeconds high enough that a cold start is not counted as a crash, and keep timeoutSeconds consistent with real work. Configure startup protection independently: readiness alone cannot prevent liveness-driven termination. Better still, add a startup probe for slow-booting containers: until it succeeds, both liveness and readiness stay disabled, so version upgrades that warm large caches get minutes instead of seconds without weakening steady-state liveness at all.

Tune thresholds from observed behavior, not round numbers copied between services. periodSeconds times failureThreshold is a planning estimate, not an exact reaction time. Probe phase, timeout, scheduling, termination grace and restart backoff alter the observed timeline. Measure cold-start p99, dependency recovery time, and worst-case pauses, then set each number to react faster than the alert but slower than the noise.

When it breaks

The signature failure is the restart loop during dependency outages: the database slows, readiness would have sufficed, but liveness probes share the same failing endpoint and the whole fleet restarts in waves. Each restart wipes warm caches, making the next wave slower to recover, until the system oscillates indefinitely. Diagnose by correlating restart timestamps with the dependency incident; repair the decision semantics. Keep restart signals local and recovery-relevant. Dependency-aware readiness may help, but withdrawing every replica can be worse than serving degraded responses during a shared outage.

The mirror failure is the zombie Pod: liveness passes because the HTTP server responds, but every request hangs behind a deadlocked worker pool. Traffic keeps flowing to a Pod that answers nothing. A short worker-slot timeout cannot distinguish deadlock from healthy saturation and can trigger overload-driven restarts. Prefer evidence of stalled critical worker progress that accounts for expected pauses and idle periods; validate that restarting improves this specific failure. Handle temporary overload separately.

Check it quickly

Inspect probe wiring and recent kills:

kubectl get pod web-0 -o jsonpath='{.spec.containers[*].livenessProbe}{"\n"}{.spec.containers[*].readinessProbe}{"\n"}'
kubectl get events --field-selector reason=Unhealthy --sort-by=.lastTimestamp | tail -n 10
kubectl describe pod web-0 | grep -A3 -i 'liveness\|readiness'

These unchanged diagnostics were not rerun here. Shared URLs are not inherently wrong if their semantics fit both decisions. Unhealthy events include probe failures, not proof of a kill. Correlate the named Pod, container restart count and termination reason; bounded checks must distinguish deadlock from temporary load and harmless idleness.

Sources

Kubernetes probes, Official docs on liveness and readiness, checked 2026-10-15.
Configure liveness and readiness probes, task guide for probe configuration, checked 2026-10-15.

Probes tell the platform when to act

Fri, 18 Jun 2027 00:00:00 +0000

A probe result is evidence consumed by a control loop. Kubernetes does not infer your application’s recovery strategy from a route named healthz. The probe type, result, configured thresholds, and container lifecycle determine what happens next. Treating every failure as “restart the Pod” erases distinctions that directly affect availability.

Three questions, different actions

A startup probe gives initialization its own detection policy. When configured, it prevents liveness and readiness probes from running until it succeeds. It stops checking after success for that container lifetime; a newly started container needs to pass startup again.

Liveness is intended to identify conditions for which restarting the affected container can help. After the failure threshold, the kubelet kills that container and applies its restart policy. This is not a promise to replace the whole Pod, change its identity, or repair an unavailable external database.

Readiness controls whether the container is considered ready. A failed readiness check leaves the container running and continues checking for recovery. It affects ordinary readiness-aware Service routing, but is not a firewall and does not cancel requests already using a connection.

Thresholds operate on consecutive observations

A failureThreshold of three is not a three-failures-ever counter. A success breaks the run. Readiness can require multiple consecutive successes to recover; successThreshold must be one for startup and liveness. Know both directions of a transition before interpreting a short trace.

Timeouts, initial delays, periods, and termination grace also matter. Threshold multiplied by period is useful for planning, not an exact end-to-end failover stopwatch. Scheduling, probe execution, and routing propagation intervene. Readiness may be checked more frequently than its configured period while a container is not ready.

The experiment uses four startup failures, three liveness failures, and two readiness failures, with two readiness successes required for recovery. These settings make counters observable; they are not production recommendations.

Match the mechanism to the evidence

An HTTP check uses the documented 200–399 success range. A 200 response containing an error message is not a failed HTTP probe merely because a human dislikes its body. Verify redirects and authentication on the actual probe route rather than assuming the same behavior as an interactive browser.

A TCP check shows that a connection can be opened, not that a query or transaction succeeds. An exec check needs a command present in the container and a meaningful exit status; frequent process creation also costs resources. Select the cheapest mechanism that answers the required question.

Readiness is not a prerequisite for liveness. Without startup gating, a long initialization period can repeatedly fail liveness before the application has time to become ready. Conversely, an excessively generous startup allowance can postpone detection of a genuinely broken launch.

Probe failure and an Unknown result are distinct. Kubernetes documents Unknown as leaving container state unchanged while further checks occur. This binary teaching model intentionally has no third outcome, so it cannot predict transport-error classification. Use kubelet events and the deployed probe handler to establish how a real error was classified.

When it breaks

Using external dependency availability as liveness can restart healthy processes throughout a shared outage. An always-green handler on an unrelated thread can miss a deadlocked business worker. Restarts are not automatically cheap: they can lose in-flight work and repeat expensive initialization.

A readiness gate covering every optional dependency can withdraw useful capacity. Rapid state flapping can transfer load between replicas and worsen saturation. Check false positives and recovery behavior, not just whether a deliberately broken endpoint turns red.

Compare probe outcomes with container restart counts, events, readiness transitions, and actual client traffic. Confirm the deployed configuration and routing consumer. A successful local policy model cannot establish that a real kubelet, network path, or load balancer applies it correctly.

Three probe responsibilities: mechanism and evidence boundaries.

Check it quickly

Save this block as probe_policy_check.py and run python3 probe_policy_check.py with Python 3.10 or newer. Retained records report Python 3.12.14; this review did not rerun it. Keep assertions enabled, without -O or PYTHONOPTIMIZE, because some sample calls occur inside assert. This deterministic model consumes binary observations; it does not deploy a Pod, call Kubernetes, schedule probes, or restart an operating-system process.

class ProbePolicy:
 # (consecutive successes, consecutive failures), chosen for this experiment.
 thresholds = {"startup": (1, 4), "liveness": (1, 3), "readiness": (2, 2)}

 def __init__(self):
 self.restart_requests = 0
 self.start_container()

 def start_container(self):
 self.started = self.ready = self.restart_requested = False
 self.successes = dict.fromkeys(self.thresholds, 0)
 self.failures = dict.fromkeys(self.thresholds, 0)

 def sample(self, kind, passed):
 if self.restart_requested:
 return "awaiting-restart"
 if kind != "startup" and not self.started:
 return "suppressed"
 if kind == "startup" and self.started:
 return "disabled"
 success_limit, failure_limit = self.thresholds[kind]
 self.successes[kind] = self.successes[kind] + 1 if passed else 0
 self.failures[kind] = 0 if passed else self.failures[kind] + 1
 if passed and self.successes[kind] >= success_limit:
 if kind == "startup":
 self.started = True
 elif kind == "readiness":
 self.ready = True
 if not passed and self.failures[kind] >= failure_limit:
 self.ready = False
 if kind != "readiness":
 self.restart_requested = True
 self.restart_requests += 1
 return "restart-requested" # A model output, not a kubelet or OS action.
 return "sampled"


policy = ProbePolicy()
for _ in range(3):
 assert policy.sample("startup", False) == "sampled"
 assert policy.sample("liveness", False) == "suppressed"
 assert policy.sample("readiness", True) == "suppressed"
assert policy.restart_requests == 0 and not policy.ready
assert policy.failures["liveness"] == 0 and policy.successes["readiness"] == 0
policy.sample("startup", True)
assert policy.started and not policy.ready
assert policy.sample("startup", False) == "disabled"
print("three startup failures do not trigger liveness; startup success enables the other probes")

policy.sample("readiness", True)
assert not policy.ready
policy.sample("readiness", True)
assert policy.ready
policy.sample("readiness", False)
assert policy.ready
policy.sample("readiness", True) # Breaks the consecutive-failure run.
policy.sample("readiness", False)
assert policy.ready
policy.sample("readiness", False)
assert not policy.ready and policy.restart_requests == 0
for result in (True, False, True):
 policy.sample("readiness", result)
assert not policy.ready
policy.sample("readiness", True)
assert policy.ready
print("two readiness failures remove readiness without restart; two consecutive successes restore it")

for result in (False, False, True, False, False):
 policy.sample("liveness", result)
assert policy.restart_requests == 0
assert policy.sample("liveness", False) == "restart-requested"
assert policy.restart_requests == 1 and not policy.ready
assert policy.sample("liveness", False) == "awaiting-restart"
assert policy.restart_requests == 1
policy.start_container() # Explicitly model the beginning of another container lifetime.
assert not policy.started and not policy.ready
for _ in range(4):
 policy.sample("startup", False)
assert policy.restart_requests == 2
print("three consecutive liveness failures request a restart; a fresh lifetime runs startup checks again")
print("four startup failures request another restart; no real container, clock, or traffic route was changed")

Three startup failures remain below the fixture’s threshold, and attempted liveness/readiness observations are suppressed. Startup success enables those checks without automatically declaring readiness.

Two consecutive readiness failures remove readiness without requesting a restart; recovery needs two consecutive successes. Intervening opposite outcomes reset the relevant streak. Three consecutive liveness failures produce one restart request, not repeated requests while awaiting restart.

An explicitly modeled new container lifetime clears startup state. Four startup failures then request another restart. Unknown results, transport error classification, real timing, restart policy execution, termination grace, and endpoint propagation remain untested; validate those on the intended cluster before relying on the full operational behavior.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
Kubernetes: Liveness, readiness, and startup probes, probe actions, mechanisms, thresholds, and startup gating, checked 2026-10-15.
Kubernetes: Configure probes, executable probe configuration and observed container restarts, checked 2026-10-15.
Kubernetes: Pod lifecycle, container restart policy and termination versus traffic lifecycle, checked 2026-10-15.

Health · XB Field Notes

Consul discovery reflects health policy and freshness

Registration and health are separate

DNS is not always passing-only

Freshness has several stages

When it breaks

Check it quickly

Sources

Liveness and readiness tell Kubernetes how the Pod feels

Readiness gates traffic

Liveness guards the process

Give the probes grace

When it breaks

Check it quickly

Sources

Probes tell the platform when to act

Three questions, different actions

Thresholds operate on consecutive observations

Match the mechanism to the evidence

When it breaks

Check it quickly

Further reading

Sources