Discovery · XB Field Notes

Consul discovery reflects health policy and freshness

Thu, 06 Apr 2028 00:00:00 +0000

Consul discovery reports registered endpoints and recorded health observations. It cannot promise that every returned endpoint will answer the caller’s next request. A useful operating model separates four things: what is registered, which checks apply, which states the query accepts, and how old those observations may be. A successful lookup establishes none of those guarantees by itself.

Registration and health are separate

The catalog stores registrations; health endpoints add check information. An unfiltered /v1/health/service/web response can include instances with critical checks. A failed check does not automatically erase its service registration. Explicit deregistration and configured critical-service removal are separate lifecycle mechanisms, and ACL filtering can also change which records a reader sees.

Registered checks normally start critical unless an initial status is supplied. That does not mean every registered service necessarily has a meaningful readiness check. For one service instance, consider its own service checks together with node-level checks such as serfHealth. A failed database-service check on the same node does not automatically fail an unrelated web-service check. A critical node check, however, affects services on that node.

Check outcomes are not simply Boolean. Script exit zero means passing, one means warning, and other exit codes mean critical. HTTP checks classify 2xx as passing and 429 as warning after redirect handling; redirects are followed by default unless disable_redirects is set. Other final responses indicate failure. The choice of probe determines what those states actually establish. A TCP handshake or process-existence test does not prove an application can finish a request.

DNS is not always passing-only

The HTTP health endpoint’s passing=true option requires relevant checks to be passing. Consul DNS service lookups use a different default policy: warning instances can remain eligible, while critical instances are excluded. Setting dns_config.only_passing=true excludes warning states too. Therefore, a warning instance appearing in DNS but missing from the passing-only API response need not indicate a broken check.

When comparing outputs, use the same service, datacenter, namespace where applicable, and observation window. Consul normally listens for DNS on port 8600; querying the system resolver without appropriate forwarding is a different path. Querying Consul directly removes that intermediary, but does not turn the answer into a probe of the application from the caller’s network.

Freshness has several stages

Probe intervals, probe timeouts, consecutive-failure thresholds, agent-to-server propagation, read consistency, and client caching can all contribute delay. A 30-second interval is not a guaranteed 30-second end-to-end removal bound. A TTL check expires relative to its most recent update; a failure halfway through its TTL does not necessarily leave a whole additional TTL of eligibility.

Anti-entropy repairs differences between agent state and the catalog. Its documented periodic interval is approximate and depends on cluster size. Registration changes also trigger updates. Failed synchronization can continue across multiple intervals, so the periodic schedule is neither the only update mechanism nor a maximum convergence deadline.

HTTP reads normally use the default consistency mode; DNS normally permits stale reads. A consistent HTTP read verifies leadership with a quorum, at additional cost. Even that stronger catalog read cannot discover a service failure that no check has observed yet. Catalog consistency and real-world readiness remain distinct.

When it breaks

A passing probe may test the wrong dependency, run from a different network, or miss a deadlock. Conversely, coupling every instance’s readiness to one shared dependency can remove the whole pool during a common outage. Choose checks according to useful request capability and the intended degraded behavior, then retain client timeouts and bounded recovery paths.

Do not diagnose an API/DNS mismatch from one screenshot. Check warning policy, relevant check IDs, ACL visibility, deregistration, and cached or stale observations first. A critical instance can remain in the unfiltered health response; its continued presence there is expected, not proof that filtering was ignored.

Consul health selection: mechanism and evidence boundaries.

Check it quickly

Save this as consul_filter.py and run python3 consul_filter.py. This standard-library exercise uses synthetic records, starts no Consul agent, and makes no network requests. Keep assertions enabled, without -O or PYTHONOPTIMIZE. It tests selection and stale snapshots, not Consul itself; this review did not rerun the retained exact fixture.

from copy import deepcopy

nodes = {"n1": "passing", "n2": "critical"}
services = [
 {"ID": "web-1", "Name": "web", "Node": "n1", "Status": "passing"},
 {"ID": "web-2", "Name": "web", "Node": "n1", "Status": "warning"},
 {"ID": "db-1", "Name": "db", "Node": "n1", "Status": "critical"},
 {"ID": "web-3", "Name": "web", "Node": "n2", "Status": "passing"},
]


def health_rows(name):
 # Synthetic snapshot: only the node and THIS service contribute checks.
 return [
 {"ID": item["ID"], "Checks": [nodes[item["Node"]], item["Status"]]}
 for item in services if item["Name"] == name
 ]


def selected(rows, allowed):
 return [row["ID"] for row in rows if all(s in allowed for s in row["Checks"])]


snapshot = health_rows("web")
assert [row["ID"] for row in snapshot] == ["web-1", "web-2", "web-3"]
assert selected(snapshot, {"passing"}) == ["web-1"]
assert selected(snapshot, {"passing", "warning"}) == ["web-1", "web-2"]
print("unfiltered: web-1 web-2 web-3; passing-only: web-1; default DNS policy: web-1 web-2")

previous = deepcopy(snapshot)
services[0]["Status"] = "critical"
current = health_rows("web")
assert selected(previous, {"passing"}) == ["web-1"] # An old snapshot stays old.
assert selected(current, {"passing"}) == []
assert len(current) == 3 # Failing health is not deregistration.
assert selected(current, {"passing", "warning"}) == ["web-2"]
print("service failure changes selection, not registration; an old snapshot still selects web-1")

nodes["n1"] = "critical"
assert selected(health_rows("web"), {"passing", "warning"}) == []
print("node failure excludes both web instances on n1; db-1 never controlled their earlier health")

The assertions keep unrelated database health out of web selection, distinguish warning from critical, and preserve failed registrations in the unfiltered view. Updating current health does not rewrite a previously copied snapshot. Node failure then removes both local web instances from the modeled DNS-eligible set.

For deployment validation, compare actual unfiltered and passing-only health responses with a direct DNS query under the configured warning policy. Record check transitions and client-visible removal separately. That integration was not executed here; the offline model provides no measured detection time, DNS packet evidence, or availability guarantee.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
Consul Health API, service instances, checks, passing filter, and ACL visibility, checked 2026-10-15.
Consul health checks, initial states, script and HTTP outcomes, and TTL expiry, checked 2026-10-15.
Consul DNS configuration, only_passing and relevant checks, checked 2026-10-15.
Consul DNS behavior, port 8600 and caching controls, checked 2026-10-15.
Consul consistency, agent updates and approximate anti-entropy intervals, checked 2026-10-15.
Consul consistency modes, HTTP and DNS defaults and quorum checks, checked 2026-10-15.
Consul health-check reference, timeouts, thresholds, and critical-service deregistration, checked 2026-10-15.

Sitemaps guide crawler discovery

Wed, 30 Jun 2027 00:00:00 +0000

A sitemap gives a crawler an explicit list of URLs to consider. It is useful when new or deeply linked documents are difficult to discover through ordinary navigation. The list does not bypass authentication, grant crawl permission, guarantee indexing, or promise a revisit date. Treat it as a published inventory of intended public documents, and keep that inventory aligned with the site’s canonical URLs and actual release state.

Generate from the publication inventory

Use the same authoritative content selection that produces the public pages. Exclude drafts, private documents, and redundant tracking variants. A filename present in a source checkout is not necessarily a published URL. Conversely, pagination, translated documents, or generated routes may exist without a one-to-one source filename. Check the generator’s actual output rather than guessing the inventory from a directory listing.

The XML sitemap format uses a declared namespace and a urlset containing URL entries. Each loc supplies a fully qualified URL. XML escaping and URL encoding solve different problems: an ampersand separating query parameters must be escaped in the XML text, while characters inside the URL must also have the appropriate URI representation. Parse the generated file back to verify that the intended URL survives both layers.

Respect the format’s scope and size requirements. A sitemap normally lists URLs for the relevant host and permitted location scope; supported cross-site submission arrangements have additional rules. Each sitemap is limited to fifty thousand URLs and 52,428,800 uncompressed bytes. Gzip reduces transfer size without removing the uncompressed limit. Split larger inventories and publish an appropriate sitemap index rather than silently truncating the list.

Treat the inventory itself as public data. Omitting a private document from navigation is not protection if its URL appears in a public sitemap. Review the generated XML for unintended disclosures before publication, and enforce access control at the resource independently. Crawler discovery preferences are not an authorization mechanism.

When it breaks

A valid XML file can still list redirects, missing pages, or content that should not be indexed. Validate destinations as well as syntax, and align their canonical declarations with the intended inventory. An HTTP 200 response containing a login screen or generic error page is not proof that the listed document is available. Sitemap delivery and document usefulness are separate checks.

Modification dates can also become misleading. If lastmod is supplied, derive it from a significant change to the page rather than replacing every timestamp on each build. Google documents that it uses consistently accurate lastmod values and ignores priority and changefreq. Those fields are not a scheduling interface, and setting maximum priority everywhere does not command more crawling.

A sitemap cannot make a private page publicly retrievable or override robots exclusions. Nor does submission guarantee that a search engine will select the same canonical, fetch every entry, or include every page in results. Keep ordinary navigation usable and use observed crawling or indexing reports for those later outcomes. Do not describe a successful upload as proof that discovery and indexing have completed.

Publish an inventory, verify its targets. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

Save this as sitemap.py and run python3 sitemap.py. The standard-library fixture serves a generated XML sitemap and a tiny local publication inventory. Three entries are public, including one with two query parameters; a draft is excluded. The generated URLs point only to the temporary loopback origin, and the validator checks that scope before requesting destinations.

from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from urllib.parse import urlsplit
import json
import xml.etree.ElementTree as ET

namespace = 'http://www.sitemaps.org/schemas/sitemap/0.9'
ET.register_namespace('', namespace)
pages = {'/article': True, '/zh/article': True, '/report?year=2026&region=west': True, '/draft': False}
state = {'bad_entry': False}


class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 if self.path == '/sitemap.xml':
 root = ET.Element(f'{{{namespace}}}urlset')
 paths = [path for path, public in pages.items() if public]
 if state['bad_entry']:
 paths.append('/missing')
 for path in paths:
 item = ET.SubElement(root, f'{{{namespace}}}url')
 ET.SubElement(item, f'{{{namespace}}}loc').text = f'http://127.0.0.1:{self.server.server_port}{path}'
 status, data, kind = 200, ET.tostring(root, encoding='utf-8', xml_declaration=True), 'application/xml'
 else:
 status = 200 if pages.get(self.path) else 404
 data, kind = (b'published page' if status == 200 else b'missing'), 'text/plain'
 self.send_response(status)
 self.send_header('Content-Type', kind)
 self.send_header('Content-Length', str(len(data)))
 self.end_headers()
 self.wfile.write(data)

 def log_message(self, *args):
 pass


server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever)
thread.start()
try:
 def fetch(path):
 connection = HTTPConnection('127.0.0.1', server.server_port, timeout=5)
 try:
 connection.request('GET', path)
 response = connection.getresponse()
 return response.status, response.read()
 finally:
 connection.close()

 def validate(data):
 root = ET.fromstring(data)
 assert root.tag == f'{{{namespace}}}urlset'
 locations = [item.text for item in root.findall(f'{{{namespace}}}url/{{{namespace}}}loc')]
 assert len(locations) == len(set(locations))
 for location in locations:
 parsed = urlsplit(location)
 assert (parsed.scheme, parsed.hostname, parsed.port) == ('http', '127.0.0.1', server.server_port)
 path = parsed.path + ('?' + parsed.query if parsed.query else '')
 status, _ = fetch(path)
 if status != 200:
 raise ValueError(f'listed URL returned {status}: {path}')
 return locations

 status, data = fetch('/sitemap.xml')
 assert status == 200 and b'&amp;' in data
 locations = validate(data)
 assert len(locations) == 3 and all('/draft' not in value for value in locations)
 assert any('?year=2026&region=west' in value for value in locations)
 state['bad_entry'] = True
 _, invalid = fetch('/sitemap.xml')
 try:
 validate(invalid)
 except ValueError as error:
 rejection = str(error)
 else:
 raise AssertionError('missing destination accepted')
 print(json.dumps({'public_urls': len(locations), 'query_round_trip': True,
 'draft_excluded': True, 'invalid_destination_rejected': rejection,
 'indexing_tested': False}))
finally:
 server.shutdown()
 thread.join()
 server.server_close()

The first sitemap contains three unique loc values in the correct namespace. Its raw XML escapes the query separator as &, while parsing recovers the intended ampersand in the URL. All three destinations return 200, and the draft does not appear. The example omits optional modification dates because it has no publication-history source to justify them.

The fixture then adds a missing destination while continuing to serve well-formed XML with status 200. The second validation fails when that listed URL returns 404. This demonstrates why a sitemap syntax check alone cannot establish that its inventory is useful. The failure is reported instead of quietly removing the entry during validation.

No search engine is contacted, and this small validator does not implement every protocol limit or indexing rule. It verifies generation, escaping, namespace, uniqueness, local scope, and destination availability for the stated examples. Before publishing the real sitemap, compare it with the complete intended public inventory and check the released file at its advertised URL. That establishes what was published without claiming control over a crawler’s later choices.

Sources

Sitemaps.org protocol, XML structure, URL encoding, scope, and size limits, checked 2026-09-13.
Google: build and submit a sitemap, canonical URLs, meaningful lastmod, and unsupported hints, checked 2026-09-13.

robots.txt guides crawlers but does not authorize access

Fri, 02 Oct 2026 00:00:00 +0000

A robots.txt file communicates crawling rules to automated clients that implement the Robots Exclusion Protocol. It can reduce unwanted traversal of expensive or unhelpful paths, but the server does not automatically enforce its contents against incoming requests. RFC 9309 explicitly separates these rules from access authorization. Treat robots.txt as a public interface for cooperating crawlers, while authentication and authorization decide who may actually retrieve protected resources.

Match the intended crawler and path

The standard file lives at /robots.txt for the relevant scheme and authority. A file under an application’s subdirectory does not automatically set the origin’s policy. User-agent groups identify the crawler products to which rules apply, with a wildcard group covering other agents according to the protocol. Path rules are evaluated against request targets; capitalization can change the match. Review concrete example URLs whenever routing changes.

Keep the file readable as UTF-8 plain text and make its intended response explicit. A broad static-site fallback might return a home page with status 200 at the robots path. That proves HTTP delivery, not that the crawler received the intended policy. Complicated Allow and Disallow combinations, escaping, and wildcard patterns need checks against the protocol and the actual crawler implementation. A simple parser demonstration does not certify every edge case.

When it breaks

Listing a private path in robots.txt does not conceal it. The file itself can advertise that path, and a client can request it without consulting the policy. Even a cooperating crawler’s decision does not configure an access rule in the origin server. If content must be restricted, enforce that restriction where the request is authorized; a Disallow line is not a replacement for the permission check.

Crawling and indexing are also different. Google documents that a disallowed URL can still appear in search results when other sources link to it. If the crawler cannot fetch the page, it may not see an intended noindex directive there. Choose the appropriate indexing control for public content, and use real access restrictions for private content. Do not assume that blocking retrieval automatically removes every existing index entry.

Availability failures have distinct semantics. RFC 9309 allows a crawler to access resources when the robots file is unavailable, for example under HTTP 4xx responses. When it is unreachable because of server or network errors, such as 5xx, the crawler must initially assume complete disallow. The specification also addresses prolonged failures and cached copies. Treat a broken deployment of the policy as a behavioral change, not merely a missing decorative file.

Crawler caching introduces publication delay. RFC 9309 normally advises against using a cached policy longer than twenty-four hours, with an exception when the file is unreachable. A successful update at the origin does not prove that every crawler has fetched the new rules. Keep the served response, intended rules, and observed crawler behavior as separate evidence. A sitemap pointer can aid discovery, but it does not grant access or guarantee crawling and indexing of every listed URL.

A crawl decision is not an access barrier. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

Save this as robots.py and run python3 robots.py. The fixture starts a loopback HTTP server serving a short robots file and harmless demonstration content. It fetches the actual policy and passes those bytes to Python’s RobotFileParser. The selected paths use a simple Disallow prefix without conflicting rules or advanced wildcard behavior.

from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from urllib.robotparser import RobotFileParser
import json


class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 if self.path == '/robots.txt':
 status, data = 200, b'User-agent: *\nDisallow: /private/\n'
 elif self.path in ('/private/report', '/public'):
 status, data = 200, b'public demonstration data\n'
 else:
 status, data = 404, b'missing\n'
 self.send_response(status)
 self.send_header('Content-Type', 'text/plain; charset=utf-8')
 self.send_header('Content-Length', str(len(data)))
 self.end_headers()
 self.wfile.write(data)

 def log_message(self, *args):
 pass


server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever)
thread.start()
try:
 origin = f'http://127.0.0.1:{server.server_port}'

 def fetch(path):
 connection = HTTPConnection('127.0.0.1', server.server_port, timeout=5)
 try:
 connection.request('GET', path)
 response = connection.getresponse()
 return response.status, response.read().decode()
 finally:
 connection.close()

 status, rules = fetch('/robots.txt')
 assert status == 200
 parser = RobotFileParser(origin + '/robots.txt')
 parser.parse(rules.splitlines())
 decisions = {path: parser.can_fetch('ExampleBot', origin + path)
 for path in ('/private/report', '/public', '/Private/report')}
 assert decisions == {'/private/report': False, '/public': True, '/Private/report': True}
 direct_status, direct_body = fetch('/private/report')
 assert direct_status == 200 and direct_body == 'public demonstration data\n'
 print(json.dumps({'parser_decisions': decisions, 'direct_http_status': direct_status,
 'direct_http_body': direct_body, 'search_index_tested': False}))
finally:
 server.shutdown()
 thread.join()
 server.server_close()

The parser rejects /private/report, permits /public, and permits the differently capitalized /Private/report. Those are decisions returned to the caller. The script then deliberately performs an ordinary HTTP GET for /private/report without applying the parser’s decision. The server returns 200 and the public demonstration text, proving that the policy did not install an access barrier.

Validation (2026-10-17): The unchanged parser and access exercise passed once on Python 3.14.4 (macOS arm64), with assertions enabled and loopback-only guards. No search engine or indexing service participates in the experiment. It verifies one parser’s selected matching behavior and the distinction between that decision and server enforcement. It does not establish full RFC conformance of the library or predict every crawler’s behavior. Before publishing a policy, check representative allowed and disallowed URLs, verify the actual response, and keep the resource’s authorization requirements independently satisfied.

Sources

RFC 9309: Robots Exclusion Protocol, advisory rules, path matching, and availability handling, checked 2026-10-15.
Google: introduction to robots.txt, crawl control and the limits of blocking indexing, checked 2026-10-15.
Python 3.14: urllib.robotparser, the parser exercised by the local example, checked 2026-10-15.

robots.txt and sitemaps answer different questions

Tue, 21 Jul 2026 00:00:00 +0000

Two small files are often treated as if they were a security boundary. They are not. A robots.txt file describes crawler preferences, while a sitemap gives search systems a list of URLs that belong to the site.

Discovery aids do not grant or deny access. Follow every listed sitemap, validate more than XML syntax, and handle robots error responses according to their actual class.

What robots.txt can and cannot do

The rules are published at the origin’s /robots.txt path and are read by cooperating crawlers. They do not protect a private directory, remove a URL from the network, or replace authentication. A route that must not be public needs server authorization or a network boundary.

Keep the file boring and test its actual location:

User-agent: *
Disallow:
Sitemap: https://www.example.test/sitemap.xml

The file should be UTF-8. If a deployment redirects it, verify the crawler behavior you intend rather than assuming every client follows the same chain.

What a sitemap adds

A sitemap is an inventory, not a permission list. Use absolute URLs from one host, encode XML values correctly, and include only canonical pages that you are prepared to publish. Do not list a URL merely because it exists in a build directory.

Each sitemap may contain at most 50,000 URLs and 50 MB (52,428,800 bytes) uncompressed; gzip does not raise that limit. An index is optional for a small site, but multilingual generators may legitimately produce one pointing to separate language sitemaps. Inspect the root element and follow each listed sitemap rather than assuming /sitemap.xml is a URL set. Validate it as XML and request it through the same HTTPS edge as a normal page.

curl -fsS https://www.example.test/robots.txt
curl -fsS -o sitemap.xml https://www.example.test/sitemap.xml && \
 xmllint --noout sitemap.xml

Separate download and parsing so a failed fetch cannot be hidden by a successful parser. Well-formed XML is only a syntax check: also validate the namespace, URL count, canonical hosts, and whether listed pages exist. If xmllint is not installed, parse the XML in the same CI environment that publishes it. The important part is that the check runs before the file is replaced at the edge.

A robots Disallow is not a reliable de-indexing instruction. Also test error responses: RFC 9309 permits crawling when robots is unavailable with a 4xx response, while a network/server error initially requires complete disallow. Accidentally serving the HTML fallback with status 200 can therefore change crawler behavior even though the URL appears reachable.

Sources

RFC 9309 — Robots Exclusion Protocol, robots rules, file location, and user-agent matching, checked 2026-07-21.
Sitemaps Protocol, sitemap XML structure and discovery, checked 2026-07-21.