Service · XB Field Notes

Consul discovery reflects health policy and freshness

Thu, 06 Apr 2028 00:00:00 +0000

Consul discovery reports registered endpoints and recorded health observations. It cannot promise that every returned endpoint will answer the caller’s next request. A useful operating model separates four things: what is registered, which checks apply, which states the query accepts, and how old those observations may be. A successful lookup establishes none of those guarantees by itself.

Registration and health are separate

The catalog stores registrations; health endpoints add check information. An unfiltered /v1/health/service/web response can include instances with critical checks. A failed check does not automatically erase its service registration. Explicit deregistration and configured critical-service removal are separate lifecycle mechanisms, and ACL filtering can also change which records a reader sees.

Registered checks normally start critical unless an initial status is supplied. That does not mean every registered service necessarily has a meaningful readiness check. For one service instance, consider its own service checks together with node-level checks such as serfHealth. A failed database-service check on the same node does not automatically fail an unrelated web-service check. A critical node check, however, affects services on that node.

Check outcomes are not simply Boolean. Script exit zero means passing, one means warning, and other exit codes mean critical. HTTP checks classify 2xx as passing and 429 as warning after redirect handling; redirects are followed by default unless disable_redirects is set. Other final responses indicate failure. The choice of probe determines what those states actually establish. A TCP handshake or process-existence test does not prove an application can finish a request.

DNS is not always passing-only

The HTTP health endpoint’s passing=true option requires relevant checks to be passing. Consul DNS service lookups use a different default policy: warning instances can remain eligible, while critical instances are excluded. Setting dns_config.only_passing=true excludes warning states too. Therefore, a warning instance appearing in DNS but missing from the passing-only API response need not indicate a broken check.

When comparing outputs, use the same service, datacenter, namespace where applicable, and observation window. Consul normally listens for DNS on port 8600; querying the system resolver without appropriate forwarding is a different path. Querying Consul directly removes that intermediary, but does not turn the answer into a probe of the application from the caller’s network.

Freshness has several stages

Probe intervals, probe timeouts, consecutive-failure thresholds, agent-to-server propagation, read consistency, and client caching can all contribute delay. A 30-second interval is not a guaranteed 30-second end-to-end removal bound. A TTL check expires relative to its most recent update; a failure halfway through its TTL does not necessarily leave a whole additional TTL of eligibility.

Anti-entropy repairs differences between agent state and the catalog. Its documented periodic interval is approximate and depends on cluster size. Registration changes also trigger updates. Failed synchronization can continue across multiple intervals, so the periodic schedule is neither the only update mechanism nor a maximum convergence deadline.

HTTP reads normally use the default consistency mode; DNS normally permits stale reads. A consistent HTTP read verifies leadership with a quorum, at additional cost. Even that stronger catalog read cannot discover a service failure that no check has observed yet. Catalog consistency and real-world readiness remain distinct.

When it breaks

A passing probe may test the wrong dependency, run from a different network, or miss a deadlock. Conversely, coupling every instance’s readiness to one shared dependency can remove the whole pool during a common outage. Choose checks according to useful request capability and the intended degraded behavior, then retain client timeouts and bounded recovery paths.

Do not diagnose an API/DNS mismatch from one screenshot. Check warning policy, relevant check IDs, ACL visibility, deregistration, and cached or stale observations first. A critical instance can remain in the unfiltered health response; its continued presence there is expected, not proof that filtering was ignored.

Consul health selection: mechanism and evidence boundaries.

Check it quickly

Save this as consul_filter.py and run python3 consul_filter.py. This standard-library exercise uses synthetic records, starts no Consul agent, and makes no network requests. Keep assertions enabled, without -O or PYTHONOPTIMIZE. It tests selection and stale snapshots, not Consul itself; this review did not rerun the retained exact fixture.

from copy import deepcopy

nodes = {"n1": "passing", "n2": "critical"}
services = [
 {"ID": "web-1", "Name": "web", "Node": "n1", "Status": "passing"},
 {"ID": "web-2", "Name": "web", "Node": "n1", "Status": "warning"},
 {"ID": "db-1", "Name": "db", "Node": "n1", "Status": "critical"},
 {"ID": "web-3", "Name": "web", "Node": "n2", "Status": "passing"},
]


def health_rows(name):
 # Synthetic snapshot: only the node and THIS service contribute checks.
 return [
 {"ID": item["ID"], "Checks": [nodes[item["Node"]], item["Status"]]}
 for item in services if item["Name"] == name
 ]


def selected(rows, allowed):
 return [row["ID"] for row in rows if all(s in allowed for s in row["Checks"])]


snapshot = health_rows("web")
assert [row["ID"] for row in snapshot] == ["web-1", "web-2", "web-3"]
assert selected(snapshot, {"passing"}) == ["web-1"]
assert selected(snapshot, {"passing", "warning"}) == ["web-1", "web-2"]
print("unfiltered: web-1 web-2 web-3; passing-only: web-1; default DNS policy: web-1 web-2")

previous = deepcopy(snapshot)
services[0]["Status"] = "critical"
current = health_rows("web")
assert selected(previous, {"passing"}) == ["web-1"] # An old snapshot stays old.
assert selected(current, {"passing"}) == []
assert len(current) == 3 # Failing health is not deregistration.
assert selected(current, {"passing", "warning"}) == ["web-2"]
print("service failure changes selection, not registration; an old snapshot still selects web-1")

nodes["n1"] = "critical"
assert selected(health_rows("web"), {"passing", "warning"}) == []
print("node failure excludes both web instances on n1; db-1 never controlled their earlier health")

The assertions keep unrelated database health out of web selection, distinguish warning from critical, and preserve failed registrations in the unfiltered view. Updating current health does not rewrite a previously copied snapshot. Node failure then removes both local web instances from the modeled DNS-eligible set.

For deployment validation, compare actual unfiltered and passing-only health responses with a direct DNS query under the configured warning policy. Record check transitions and client-visible removal separately. That integration was not executed here; the offline model provides no measured detection time, DNS packet evidence, or availability guarantee.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
Consul Health API, service instances, checks, passing filter, and ACL visibility, checked 2026-10-15.
Consul health checks, initial states, script and HTTP outcomes, and TTL expiry, checked 2026-10-15.
Consul DNS configuration, only_passing and relevant checks, checked 2026-10-15.
Consul DNS behavior, port 8600 and caching controls, checked 2026-10-15.
Consul consistency, agent updates and approximate anti-entropy intervals, checked 2026-10-15.
Consul consistency modes, HTTP and DNS defaults and quorum checks, checked 2026-10-15.
Consul health-check reference, timeouts, thresholds, and critical-service deregistration, checked 2026-10-15.

Socket activation starts a service only when needed

Tue, 01 Feb 2028 00:00:00 +0000

A socket unit can keep a listening endpoint available while its compatible application is stopped. This can reduce the resident memory and periodic work of rarely used services. The socket, kernel queues and manager still consume resources, and installed software still needs security updates. Availability of the listener also depends on successful binding, permissions and the socket unit remaining active. A listening port is an opportunity to request work, not evidence that the application is ready.

The handoff depends on the configured mode. With Accept=no, systemd passes listening descriptors and the daemon accepts connections. With Accept=yes, it accepts a connection and starts a suitable service instance for that connection. Clients may connect before initialization finishes, then wait for an application response or time out. Measure that delay rather than assuming the handoff makes on-demand startup indistinguishable from a warm service.

With the socket unit active, a request can start a compatible service. Explicit application-defined idle exit can stop the process so later demand starts it again; without idle exit it stays running. Measure connect-to-first-response cold-start delay, client deadlines and the resident footprint under the real workload. Monitoring can activate multiple services together, so test combined startup load. Selected mechanisms only; the article remains held/pending and its incorrect general claims are not endorsed. Illustration of selected source mechanisms; no example executed in this round.

The listener waits first

systemd opens the configured ListenStream endpoint when the socket unit starts. With Accept=no, foo.socket normally activates foo.service and passes the listener; Service= can choose another service. With Accept=yes, the matching template is normally foo@.service and each accepted stream connection gets an instance. Datagram sockets do not use this per-connection model. The program must match the selected descriptor contract, rather than simply sharing a familiar unit basename.

This split can reduce startup-order coupling, but it does not remove dependencies. The socket unit must be active and listening before clients can use it, and required service, mount, device, and ordering dependencies still apply. Connections can wait in a finite kernel backlog while a compatible service starts, but clients can still fail or time out before the application is ready. The backlog size still deserves sizing: a flood of early clients against a slow-starting service overflows the same as any listen queue.

Service wakes on demand

On-demand activation defers the initial application launch while the socket is active. It does not put a running daemon to sleep or automatically stop one that becomes idle. Administrative services and infrequently used protocol handlers are plausible candidates when their activation interface, request deadlines and resource profile fit this lifecycle. Other dependencies or manual starts can still activate the service.

Idle exit completes a recurring savings cycle only when the application implements it safely. It must finish or transfer in-flight work before exiting and leave the manager-owned listener usable. With the socket unit still active, later demand can request another start. Without idle exit, the daemon may remain resident after its first activation. Neither case is a zero-cost state or a promise that the socket can never close.

Trade speed for memory

Budget cold-start time from connection through the first valid application response, and compare resident memory under representative load. A hypothetical 300 ms start saving 800 MB may fit one workload and violate another workload’s deadline. A 20-second warm-up might be acceptable for an asynchronous task but unsuitable for an interactive request. These are illustrative tradeoffs, not measurements or universal thresholds.

Some workloads cannot tolerate the gap. Health checks can activate a service themselves, or classify it as unavailable while it initializes. Client deadlines and load-balancer policy must fit the observed startup behavior. Keeping the daemon running may be simpler when predictable response time matters more than idle memory; warming it at boot does not itself implement later idle exit.

When it breaks

A monitoring sweep can wake many services together and create an aggregate CPU or memory spike. Staggering checks may reduce simultaneous demand. Per-service start-rate limits constrain repeated starts of each unit, but do not cap host-wide concurrency or memory use. Very short idle timeouts can increase churn. Test combined startup load and use suitable resource controls instead of treating these knobs as a global capacity guarantee.

A daemon that binds the occupied endpoint itself may fail with address-in-use errors. It must consume inherited descriptors through its native activation interface or a supported inetd-style interface. Merely inheriting a descriptor and executing an unaware daemon does not rewrite its bind logic. A protocol-aware adapter needs its own verified integration; there is no universal inherit-and-exec fix.

Check it quickly

The following inspection sketch assumes a separately configured, authorized foo.socket/foo.service pair serving the loopback health endpoint on port 8080. No such configuration is supplied here; this review did not execute the commands:

systemctl status foo.socket foo.service
ss -ltnp | grep ':8080'
curl -m 10 localhost:8080/health; systemctl show foo.service -p ActiveState,ExecMainStartTimestamp

Confirm the service is initially inactive, the intended socket is listening, and the request receives the expected application response. Compare invocation or start information before and after. Inspect each command’s result: the semicolon allows show to run even if curl fails, and status output alone does not prove health or activation causality.

Sources

systemd.socket manual, Official socket unit reference, checked 2026-06-01.
systemd.service manual, service activation, termination, and restart behavior, checked 2026-09-05.

NodePort exposes a Service through eligible node addresses

Tue, 11 May 2027 00:00:00 +0000

A NodePort Service allocates a cluster-wide port and arranges forwarding through eligible node addresses. It is useful for external load balancers, development environments and deliberately exposed services. It does not make every address on every machine reachable, nor does it promise an application-level listening socket on each node. Network routing, firewalls, proxy implementation and traffic policy remain part of the path.

NodePort forwards through eligible node addresses to the backend port. Cluster and Local policies change endpoint eligibility; allocation does not prove application reachability. Diagnostics were not rerun.

One allocation, several forwarding paths

The default allocation range is 30000–32767, but administrators can configure another range. Requesting a specific nodePort succeeds only if allocation and validation allow it; a conflict rejects the API transaction. It does not normally create a new Service with a silently unusable port. Inspect the stored object and the command’s exit status separately.

The Service port, targetPort and nodePort have different roles. The Service port is the virtual-service entry, targetPort identifies the backend port, and nodePort is the node-facing entry. A multi-port Service can have several allocations. LoadBalancer Services can also consume NodePorts, unless configured and supported otherwise, so counting only objects whose type is NodePort misses part of the allocation pool.

Forwarding may use iptables, IPVS, nftables or another implementation. Rules can steer packets without an application accepting them on a normal userspace socket. Socket listings therefore cannot establish whether the NodePort datapath works. The configured node-address selection also matters: a reachable primary address does not establish equivalent behavior on secondary or loopback addresses.

Traffic policy changes eligibility

With externalTrafficPolicy Cluster, eligible endpoints can reside on another node. The usual kube-proxy path can perform source NAT, so do not assume that the backend receives the original client address. With Local, external traffic uses local eligible endpoints and preserves the source address along that path. Upstream proxies or load balancers may already have changed it.

A node without suitable local endpoints cannot serve ordinary Local-policy traffic. External load balancing must account for this placement, rather than sending equal traffic to every node indefinitely. Terminating-endpoint handling and proxy versions add further details. Test the installed datapath instead of treating a one-replica example as a complete availability model.

Exposure follows the actual backend

NodePort itself provides transport forwarding, not HTTP authorization or TLS termination. A port selecting application Pods directly may bypass an ingress controller’s authentication, routing or WAF. A port selecting the ingress controller itself does not inherently bypass that controller. Draw the actual chain before deciding which controls a request traverses.

Restrict node-address reachability to the intended callers and record the exposure’s purpose. NodePort can be a supported long-term component; an arbitrary expiry date is not a substitute for ownership, firewall policy and monitoring. A public node IP alone neither proves the port is open nor proves that the application is protected.

When it breaks

Allocation failure, missing endpoints and unreachable nodes are different failures. An empty endpoint set can produce rejection or dropped traffic depending on implementation and policy. A timeout or refusal alone cannot identify a port collision, and HTTP 200 is only an appropriate expectation for an application that intentionally serves that response.

Check selector scope, readiness, targetPort, address selection, firewall rules and traffic policy before changing exposure. Review EndpointSlices for current endpoint conditions; the older Endpoints API is deprecated and can truncate large backend sets. Do not infer that an otherwise healthy Service object certifies the network path.

Check it quickly

These retained diagnostic commands require an explicitly selected disposable cluster, kubectl, jq and Linux networking tools. They were not rerun in this review. The final request assumes the shell is on the intended node and its first reported address is eligible; verify that assumption before running it. They inspect state and attempt one HTTP request, not an exhaustive NodePort test.

kubectl get svc web -o jsonpath='{.spec.type} {.spec.ports[0].nodePort}{"\n"}'
kubectl get endpoints web
kubectl get svc -A -o json | jq -r '.items[] | select(.spec.type=="NodePort") | "\(.spec.ports[0].nodePort) \(.metadata.namespace)/\(.metadata.name)"' | sort -n | uniq -D -w5
ss -lnt | grep -E ':3[0-9]{4}\b'
curl -sS -o /dev/null -w '%{http_code}\n' "http://$(hostname -I | awk '{print $1}'):$(kubectl get svc web -o jsonpath='{.spec.ports[0].nodePort}')/"

The first command shows only the first port. The duplicate search covers first ports on NodePort-type Services only; empty output is expected after normal API allocation and is not a complete capacity audit. The socket filter covers 30000–39999 and cannot identify datapath health or process ownership. Compare EndpointSlices separately. Test Cluster and Local policies on known node addresses in an authorized test network; retain response, endpoint and source-address evidence together.

Sources

Kubernetes Service, NodePort type, the 30000-32767 default range and cluster-wide port allocation, checked 2026-10-15.
Virtual IPs and Service Proxies, kube-proxy iptables/IPVS rule programming and externalTrafficPolicy, checked 2026-10-15.
Using Source IP, SNAT on NodePort and the client-IP consequence, checked 2026-10-15.

A Service gives a pod a stable virtual address

Tue, 09 Feb 2027 00:00:00 +0000

The selector turns the Service from a fixed Endpoint into a live view: refresh happens in the background, and a client never edits its config when pods restart.

The Service object lifetime bounds address stability. Membership, routing convergence and successful traffic remain separate. The exact allocator model passed again; no Kubernetes execution was added.

Declared, then given an address

An ordinary ClusterIP Service provides a virtual address and, with cluster DNS, a stable discovery name for its lifetime. A selector-based Service also declares which Pods in its namespace are intended backends. Other forms differ: headless Services allocate no virtual IP, ExternalName uses DNS indirection, and selectorless Services can use separately managed EndpointSlices.

The address is stable while the Service object exists, rather than forever after a name first appears. Recreating it may allocate a different address. Clients should use the intended discovery contract and refresh resolution appropriately, instead of treating a copied ClusterIP as permanent infrastructure identity.

Selector is the live membership

EndpointSlice controllers reconcile matching Pods and their endpoint conditions asynchronously. An unready Pod can remain represented with ready=false, rather than disappearing from every endpoint object. Consumers use those conditions for routing. publishNotReadyAddresses changes the normal readiness interpretation, and termination handling adds further detail.

Label agreement alone does not prove healthy traffic. Wrong ports, unsuitable probes or stale datapath programming can still fail requests. Inspect both the desired selector and the observed EndpointSlices, then verify actual application behavior. Automatic reconciliation avoids hand-maintaining ordinary backend lists, but does not promise instantaneous updates or eliminate every form of drift.

Stable even as pods churn

A stable destination allows backend replacement without publishing every Pod IP to callers. It does not guarantee uninterrupted requests. Existing connections can remain attached to old backends, new connections depend on converged routing state, and a Service with no usable endpoints cannot create missing capacity.

Readiness, graceful termination, rollout capacity and client retry policy therefore remain necessary parts of availability design. A DNS name surviving a deployment is one useful property; successful traffic and bounded recovery are separate properties that need observation. The allocation exercise below deliberately tests neither of them.

When it breaks

The pool is finite, and exhaustion is the most literal failure. A virtual-IP Service consumes an address for each configured IP family from the service address pools, so a small CIDR or leftover Services eventually reject creation with an allocation error while the pods look fine. Treat failed to allocate IP as capacity accounting: verify ownership before removing unused Services, and plan supported pool expansion for the installed cluster version.

Mixing headless and ordinary Services is the next confusion. Setting clusterIP: None allocates no virtual address, so DNS returns pod IPs instead of one stable VIP, and a client written for a single endpoint must suddenly balance across shifting addresses. The reverse also hurts: discovery code expecting per-pod DNS keeps hitting one VIP after someone recreates the Service with a ClusterIP. Because clusterIP is effectively immutable, switching modes means recreating the Service.

The last two failures live below the API. Iptables and IPVS modes program the same VIP differently, and a stale or restarting kube-proxy can leave an allocated ClusterIP resolving but blackholing traffic until rules resync. Finally, copying a ClusterIP into configs freezes a value the control plane may reassign: the address is stable only while the Service object lives. Callers must use the DNS name, and remember the VIP is intended for internal service routing; external reachability depends on explicit network configuration.

Check it quickly

Save the following as clusterip_pool.py in a scratch directory and run python3 clusterip_pool.py. It models a miniature allocator over 10.96.0.0/29 with the first address reserved, using only the standard library and changing nothing outside the process: one class hands out addresses, refuses duplicates and out-of-range requests, lets headless Services bypass the pool, and reuses released addresses, with five scenarios asserting each rule.

"""Simulate ClusterIP allocation from a small service CIDR."""

import ipaddress


class ClusterIPPool:
 """Hand out virtual addresses; headless Services bypass the pool."""

 def __init__(self, cidr, reserved=1):
 network = ipaddress.ip_network(cidr)
 hosts = list(network.hosts())
 self._all = set(hosts)
 self._reserved = set(hosts[:reserved])
 self.free = sorted(set(hosts) - self._reserved)
 self.allocated = {}

 def allocate(self, service, requested=None, headless=False):
 if headless:
 return None
 if service in self.allocated:
 raise ValueError(f"{service} already holds {self.allocated[service]}")
 if requested is not None:
 ip = ipaddress.ip_address(requested)
 if ip in self._reserved or ip not in self._all:
 raise ValueError(f"{requested} is outside the allocatable range")
 if ip not in self.free:
 raise ValueError(f"{requested} is already taken")
 self.free.remove(ip)
 self.allocated[service] = ip
 return str(ip)
 if not self.free:
 raise RuntimeError("service CIDR exhausted: no ClusterIP left")
 ip = self.free.pop(0)
 self.allocated[service] = ip
 return str(ip)

 def release(self, service):
 ip = self.allocated.pop(service, None)
 if ip is None:
 raise ValueError(f"{service} holds nothing to release")
 self.free.append(ip)
 self.free.sort()
 return str(ip)


def check(condition, message):
 if not condition:
 raise AssertionError(message)


def main():
 pool = ClusterIPPool("10.96.0.0/29", reserved=1)
 web = pool.allocate("web")
 api = pool.allocate("api", requested="10.96.0.5")
 check(web != api, "sequential Services must get distinct addresses")
 print("allocated:", web, api)

 try:
 pool.allocate("worker", requested="10.96.0.5")
 raise SystemExit("conflict was silently shared")
 except ValueError as exc:
 print("conflict rejected:", exc)

 try:
 pool.allocate("evil", requested="10.99.0.1")
 raise SystemExit("out-of-range request was accepted")
 except ValueError as exc:
 print("out-of-range rejected:", exc)

 headless = pool.allocate("db-headless", headless=True)
 check(headless is None, "headless Service must bypass the pool")
 check(len(pool.allocated) == 2, "headless must not consume the pool")
 print("headless bypass: None")

 freed = pool.release("web")
 worker = pool.allocate("worker")
 check(worker == freed, "released address must be reusable")
 print("reused:", freed, "-> worker")

 pool.allocate("extra-1")
 pool.allocate("extra-2")
 pool.allocate("extra-3")
 try:
 pool.allocate("overflow")
 raise SystemExit("exhaustion was not reported")
 except RuntimeError as exc:
 print("exhaustion reported:", exc)

 print("PASS: allocation, conflict rejection, release reuse, and exhaustion hold")


if __name__ == "__main__":
 main()

The run behaves like a healthy allocator: sequential Services get distinct addresses, a pinned request holds its address, and the deliberate double allocation is rejected instead of shared. Releasing one Service makes its address reusable, and filling the last slot proves exhaustion errors instead of recycling a live address. Both language models passed with Python 3.12.14 on Linux in this review. The first-host reservation and allocation order are teaching choices, not Kubernetes allocator guarantees.

The simulation checks bookkeeping only. It does not run the API server, etcd, or kube-proxy, and its tiny /29 stands in for a real service CIDR. Before deploying, inspect the live service CIDR, allocated ClusterIPs, and proxy rules on a node.

Sources

Kubernetes Service, documents service semantics, checked 2026-10-15.
Virtual IPs and Service Proxies, documents ClusterIP allocation and kube-proxy modes, checked 2026-10-15.
Primary reference, endpoint conditions and asynchronous discovery, checked 2026-10-15.

Service · XB Field Notes

Consul discovery reflects health policy and freshness

Registration and health are separate

DNS is not always passing-only

Freshness has several stages

When it breaks

Check it quickly

Sources

Socket activation starts a service only when needed

The listener waits first

Service wakes on demand

Trade speed for memory

When it breaks

Check it quickly

Sources

NodePort exposes a Service through eligible node addresses

One allocation, several forwarding paths

Traffic policy changes eligibility

Exposure follows the actual backend

When it breaks

Check it quickly

Sources

A Service gives a pod a stable virtual address

Declared, then given an address

Selector is the live membership

Stable even as pods churn

When it breaks

Check it quickly

Further reading

Sources