Consul discovery reports registered endpoints and recorded health observations. It cannot promise that every returned endpoint will answer the caller’s next request. A useful operating model separates four things: what is registered, which checks apply, which states the query accepts, and how old those observations may be. A successful lookup establishes none of those guarantees by itself. ...
A sitemap gives a crawler an explicit list of URLs to consider. It is useful when new or deeply linked documents are difficult to discover through ordinary navigation. The list does not bypass authentication, grant crawl permission, guarantee indexing, or promise a revisit date. Treat it as a published inventory of intended public documents, and keep that inventory aligned with the site’s canonical URLs and actual release state. ...
A robots.txt file communicates crawling rules to automated clients that implement the Robots Exclusion Protocol. It can reduce unwanted traversal of expensive or unhelpful paths, but the server does not automatically enforce its contents against incoming requests. RFC 9309 explicitly separates these rules from access authorization. Treat robots.txt as a public interface for cooperating crawlers, while authentication and authorization decide who may actually retrieve protected resources. ...
robots.txt is a crawler-facing preference; a sitemap is a URL inventory; neither is an authorization layer.