Consul discovery reports registered endpoints and recorded health observations. It cannot promise that every returned endpoint will answer the caller’s next request. A useful operating model separates four things: what is registered, which checks apply, which states the query accepts, and how old those observations may be. A successful lookup establishes none of those guarantees by itself.

Registration and health are separate

The catalog stores registrations; health endpoints add check information. An unfiltered /v1/health/service/web response can include instances with critical checks. A failed check does not automatically erase its service registration. Explicit deregistration and configured critical-service removal are separate lifecycle mechanisms, and ACL filtering can also change which records a reader sees.

Registered checks normally start critical unless an initial status is supplied. That does not mean every registered service necessarily has a meaningful readiness check. For one service instance, consider its own service checks together with node-level checks such as serfHealth. A failed database-service check on the same node does not automatically fail an unrelated web-service check. A critical node check, however, affects services on that node.

Check outcomes are not simply Boolean. Script exit zero means passing, one means warning, and other exit codes mean critical. HTTP checks classify 2xx as passing and 429 as warning after redirect handling; redirects are followed by default unless disable_redirects is set. Other final responses indicate failure. The choice of probe determines what those states actually establish. A TCP handshake or process-existence test does not prove an application can finish a request.

DNS is not always passing-only

The HTTP health endpoint’s passing=true option requires relevant checks to be passing. Consul DNS service lookups use a different default policy: warning instances can remain eligible, while critical instances are excluded. Setting dns_config.only_passing=true excludes warning states too. Therefore, a warning instance appearing in DNS but missing from the passing-only API response need not indicate a broken check.

When comparing outputs, use the same service, datacenter, namespace where applicable, and observation window. Consul normally listens for DNS on port 8600; querying the system resolver without appropriate forwarding is a different path. Querying Consul directly removes that intermediary, but does not turn the answer into a probe of the application from the caller’s network.

Freshness has several stages

Probe intervals, probe timeouts, consecutive-failure thresholds, agent-to-server propagation, read consistency, and client caching can all contribute delay. A 30-second interval is not a guaranteed 30-second end-to-end removal bound. A TTL check expires relative to its most recent update; a failure halfway through its TTL does not necessarily leave a whole additional TTL of eligibility.

Anti-entropy repairs differences between agent state and the catalog. Its documented periodic interval is approximate and depends on cluster size. Registration changes also trigger updates. Failed synchronization can continue across multiple intervals, so the periodic schedule is neither the only update mechanism nor a maximum convergence deadline.

HTTP reads normally use the default consistency mode; DNS normally permits stale reads. A consistent HTTP read verifies leadership with a quorum, at additional cost. Even that stronger catalog read cannot discover a service failure that no check has observed yet. Catalog consistency and real-world readiness remain distinct.

When it breaks

A passing probe may test the wrong dependency, run from a different network, or miss a deadlock. Conversely, coupling every instance’s readiness to one shared dependency can remove the whole pool during a common outage. Choose checks according to useful request capability and the intended degraded behavior, then retain client timeouts and bounded recovery paths.

Do not diagnose an API/DNS mismatch from one screenshot. Check warning policy, relevant check IDs, ACL visibility, deregistration, and cached or stale observations first. A critical instance can remain in the unfiltered health response; its continued presence there is expected, not proof that filtering was ignored.

Synthetic node and service health determine two selection policies; a stale snapshot stays old after failure, while registrations remain.
Consul health selection: mechanism and evidence boundaries.

Check it quickly

Save this as consul_filter.py and run python3 consul_filter.py. This standard-library exercise uses synthetic records, starts no Consul agent, and makes no network requests. Keep assertions enabled, without -O or PYTHONOPTIMIZE. It tests selection and stale snapshots, not Consul itself; this review did not rerun the retained exact fixture.

from copy import deepcopy

nodes = {"n1": "passing", "n2": "critical"}
services = [
    {"ID": "web-1", "Name": "web", "Node": "n1", "Status": "passing"},
    {"ID": "web-2", "Name": "web", "Node": "n1", "Status": "warning"},
    {"ID": "db-1", "Name": "db", "Node": "n1", "Status": "critical"},
    {"ID": "web-3", "Name": "web", "Node": "n2", "Status": "passing"},
]


def health_rows(name):
    # Synthetic snapshot: only the node and THIS service contribute checks.
    return [
        {"ID": item["ID"], "Checks": [nodes[item["Node"]], item["Status"]]}
        for item in services if item["Name"] == name
    ]


def selected(rows, allowed):
    return [row["ID"] for row in rows if all(s in allowed for s in row["Checks"])]


snapshot = health_rows("web")
assert [row["ID"] for row in snapshot] == ["web-1", "web-2", "web-3"]
assert selected(snapshot, {"passing"}) == ["web-1"]
assert selected(snapshot, {"passing", "warning"}) == ["web-1", "web-2"]
print("unfiltered: web-1 web-2 web-3; passing-only: web-1; default DNS policy: web-1 web-2")

previous = deepcopy(snapshot)
services[0]["Status"] = "critical"
current = health_rows("web")
assert selected(previous, {"passing"}) == ["web-1"]  # An old snapshot stays old.
assert selected(current, {"passing"}) == []
assert len(current) == 3  # Failing health is not deregistration.
assert selected(current, {"passing", "warning"}) == ["web-2"]
print("service failure changes selection, not registration; an old snapshot still selects web-1")

nodes["n1"] = "critical"
assert selected(health_rows("web"), {"passing", "warning"}) == []
print("node failure excludes both web instances on n1; db-1 never controlled their earlier health")

The assertions keep unrelated database health out of web selection, distinguish warning from critical, and preserve failed registrations in the unfiltered view. Updating current health does not rewrite a previously copied snapshot. Node failure then removes both local web instances from the modeled DNS-eligible set.

For deployment validation, compare actual unfiltered and passing-only health responses with a direct DNS query under the configured warning policy. Record check transitions and client-visible removal separately. That integration was not executed here; the offline model provides no measured detection time, DNS packet evidence, or availability guarantee.

Sources