Read a live edge the way a client does

The HTTP-foundations notes each take one contract and show single pieces of evidence. This tutorial walks the whole path a single client request touches, in order, using only read-only commands. The aim is to distinguish protocol requirements from choices made by a particular server, rather than demand the same status and headers from every endpoint.

We use www.xbcatm.com throughout. Replace it with the host you are inspecting. The requests use safe HTTP methods; they can still create access logs or consume rate limits. Inspect only endpoints you are permitted to query. Output blocks below are illustrative, not current production measurements. Run shell snippets in a disposable working directory, keep failures visible, and record time and client version.

1. Start with what a method may ask

Before you read a response body, pin down what the server will allow you to ask. A static handler often supports GET and HEAD, while a proxy may also implement OPTIONS. Ask the edge about the target resource:

$ curl -sS --max-time 20 -D - -o /dev/null -w 'OPTIONS -> HTTP %{http_code}\n' -X OPTIONS https://www.xbcatm.com/
HTTP/2 405
allow: GET, HEAD

OPTIONS -> HTTP 405

405 Method Not Allowed is valid when OPTIONS is recognized but not supported for this resource, and a 405 must include Allow. A successful 200 or 204 OPTIONS response can also be correct. 501 means the method is not implemented by the server. An OPTIONS result is not proof that every other method behaves correctly. Do not send POST merely to test rejection on an unfamiliar live service: it is not a safe method, and a 200 alone does not describe its side effects.

2. Read the representation line

The first line tells you the negotiated exchange; the headers then tell you what kind of bytes are coming. Capture them exactly as a client sees them, discarding the body:

$ curl -sS --max-time 20 -D - -o /dev/null https://www.xbcatm.com/
HTTP/2 200
accept-ranges: bytes
content-length: 15890
content-security-policy: default-src 'self'; base-uri 'self'; form-action 'none'; frame-ancestors 'none'; object-src 'none'; script-src 'none'; style-src 'self'; img-src 'self' data:
content-type: text/html; charset=utf-8
etag: "dkx84jwth7nkc9e"
last-modified: Mon, 24 Aug 2026 13:55:23 GMT
strict-transport-security: max-age=31536000; includeSubDomains
x-content-type-options: nosniff

Three headers deserve a pause. content-type: text/html; charset=utf-8 names the media type, and content-type-and-nosniff shows why it must describe the actual bytes you send. etag: "dkx84jwth7nkc9e" hands you a strong validator. x-content-type-options: nosniff constrains browser MIME handling, notably for script and stylesheet fetches; it does not validate the body as HTML. Each is evidence, not decoration, and together they say: this response advertises metadata to compare with its body and intended policy. A syntactically strong ETag is a server assertion, not independent proof of a content hash.

3. Compare HEAD and GET for the same representation

HEAD has GET semantics without a response body. Servers should return the same header fields but may omit fields whose values are determined only while generating content. Content-Length is optional; if sent on HEAD, it must equal the size of the content a corresponding GET would send. Compare stable requests with the same encoding and other selection headers:

$ curl -sSI --max-time 20 -H 'Accept-Encoding: identity' https://www.xbcatm.com/ | grep -i content-length
content-length: 15890
$ curl -sS --max-time 20 -H 'Accept-Encoding: identity' https://www.xbcatm.com/ | wc -c
15890

They match: 15890 on both. That is the head-request-parity contract. An absent HEAD length alone is not a protocol violation. A mismatched value needs investigation, including a deployment between the requests or different selected representations. A pipeline to wc can hide curl failure in shells without pipefail; for an automated gate, save the body only after curl succeeds and count that file.

4. Ask for precise bytes with a range

Because the origin knows its representation, it can serve part of it. A range request asks for a byte span and gets only that span:

$ curl -sS --max-time 20 -D - -o /dev/null -H 'Accept-Encoding: identity' -H 'Range: bytes=0-99' -w 'HTTP %{http_code} bytes=%{size_download}\n' https://www.xbcatm.com/
HTTP/2 206
content-range: bytes 0-99/15890
content-length: 100

HTTP 206 bytes=100

For a representation at least 100 bytes long, a successful single-range response should have 206, a matching Content-Range: bytes 0-99/…, and 100 content bytes. A server may legally ignore Range and return a full 200; that does not imply it lacks validators. An unsatisfiable range can return 416. A plain Range request does not require an ETag. Resuming a particular representation with If-Range, or safely combining partial responses, needs the relevant strong-validator rules. Ranges refer to the selected encoded bytes, hence the explicit identity encoding here.

5. Let the cache revalidate with a validator

Now combine method, representation, and byte selection through caching. A fresh stored response may be reusable without checking a validator. To test revalidation, capture the current response and replay its actual ETag using the same representation selection. Do not copy an old tag from an article:

set -eu
url=https://www.xbcatm.com/
work=$(mktemp -d)
status=$(curl -sS --max-time 20 -H 'Accept-Encoding: identity' \
  -D "$work/headers" -o "$work/body" -w '%{http_code}' "$url")
test "$status" = 200
etag=$(awk 'tolower($1)=="etag:" {sub(/\r$/, ""); sub(/^[^:]*:[ \t]*/, ""); print; exit}' "$work/headers")
test -n "$etag" || { echo 'No ETag: inspect Last-Modified instead' >&2; exit 1; }
curl -sS --max-time 20 -H 'Accept-Encoding: identity' \
  -H "If-None-Match: $etag" -D - -o /dev/null \
  -w 'HTTP %{http_code}\n' "$url"

Run this fragment as a standalone shell script. It must finish the initial transfer and require status 200 before storing a usable response; an error representation can also have an ETag. If the selected representation still matches and normal processing succeeds, expect 304 and reuse that complete stored body with updated metadata. A changed representation can correctly return 200. Keep the temporary directory while inspecting the result, then remove only that generated directory when finished. If-None-Match uses weak comparison, so a 304 need not prove byte identity. Freshness lifetime and current age decide reuse without validation; a validator answers a different question and does not itself set that lifetime.

6. Negotiate a smaller representation when you can

The same content can travel in fewer bytes when the client says it can decompress. Offer gzip and watch the body shrink while the meaning stays identical:

$ curl -sS --max-time 20 -H 'Accept-Encoding: identity' -o /dev/null -w 'no-enc: %{size_download} bytes\n' https://www.xbcatm.com/
$ curl -sS --max-time 20 -o /dev/null -H 'Accept-Encoding: gzip' -w 'gzip:   %{size_download} bytes\n' https://www.xbcatm.com/
no-enc: 15890 bytes
gzip:   4368 bytes

These example sizes illustrate a possible reduction; the server may choose identity instead. That is the http-compression-negotiation note’s core idea: changing the selected representation changes the bytes on the wire, and the metadata (content-length, encoding) changes with it. A client that requests compression and a cache that stores the encoded form must agree on which representation they selected, otherwise the cache can serve a mismatch. Inspect Content-Encoding and Vary: Accept-Encoding, not just the byte count. A manually supplied Accept-Encoding does not make curl automatically decompress; --compressed requests supported encodings and decodes the response. Compare decoded bodies to establish equal content, and do not assume strong ETags remain identical across encodings.

7. Follow the story when a URL changes

When identity changes, the response should say where the story continues rather than guess the body:

$ curl -sS --max-time 20 -o /dev/null -w '%{http_code} -> %{redirect_url}\n' -I http://www.xbcatm.com/ | head -1
308 -> https://www.xbcatm.com/

308 Permanent Redirect keeps the method and preserves the story, which is the concern of redirects-and-status-codes. Whether it is a 301/308 for a permanent move, or a 404 handled by the site, the status line is the first sentence of the story and should be read before the body.

8. Let the crawler read the map last

The edge is also read by machines that do not request HTML at all. Discovery answers two different questions, as robots-and-sitemaps explains. Confirm discovery points at the real sitemap:

$ curl -sS --max-time 20 https://www.xbcatm.com/robots.txt
User-agent: *
Disallow:
Sitemap: https://www.xbcatm.com/sitemap.xml

Fetch the sitemap URL too and check canonical host and language URLs. robots.txt is crawler guidance, not access control; listing a URL in a sitemap does not guarantee crawling or indexing.

The whole path in one glance

Each row was a note; now it is a command you can run against any edge. Read the linked note when you need the details, and keep this table as your re-check before you trust an endpoint.

Sources

• WHATWG Fetch Standard, nosniff processing and request destinations, checked 2026-10-15.

• RFC 9110 — HTTP Semantics, methods, validators, and status codes, checked 2026-10-15.

• RFC 9111 — HTTP Caching, freshness and revalidation, checked 2026-10-15.

• RFC 7538 — 308 Permanent Redirect, method-preserving redirect, checked 2026-10-15.

• curl manual, read-only inspection flags, checked 2026-10-15.