HTTP foundations · XB Field Notes

Content-Type should describe the bytes you actually send

Thu, 11 Jun 2026 00:00:00 +0000

A static server can return the correct bytes with the wrong Content-Type. The page may still look acceptable in one browser, while a download, stylesheet, feed reader, or security policy treats the same response differently. The public response header is the contract; the filename is only an input to server configuration.

A suffix cannot prove what arrived. Read the public response status, bytes, and media type together; an HTML error page at a CSS path is not a mapping test. Nosniff makes script and style type mismatches visible.

Inspect representative formats

Check one HTML document, stylesheet, SVG, feed, and downloadable script. Include headers but discard the body:

for path in / /assets/site.css /favicon.svg /index.xml /downloads/check-http-headers.sh; do
 curl -sS -D - -o /dev/null "https://www.example.test$path" \
 | awk -v path="$path" '
 BEGIN { print "== " path }
 /^HTTP\// || tolower($1) == "content-type:" || tolower($1) == "x-content-type-options:" { print }
 '
done

Replace the example asset paths with files that actually exist in the release; an HTML 404 response at an invented CSS path is not a MIME-mapping test. Expected media types depend on the resource, but they should be specific and stable. HTML should normally include a character set. CSS and SVG should not be served as generic binary data. A shell helper offered for download should not be advertised as HTML.

This loop displays headers; it does not assert success, inspect the discarded bytes, or exercise browser blocking. Confirm the resource against the release artifact and inspect an actual browser load as a stylesheet or script. Opening the same URL as a document tests a different request destination.

Keep nosniff consistent

X-Content-Type-Options: nosniff asks supporting browsers to respect the declared type for script and style destinations instead of guessing. For that Fetch check, a script-like destination needs a JavaScript MIME type, and a style destination needs text/css; a missing or unusable type also fails. This turns a mismatch into an observable failure rather than browser-dependent interpretation.

The header does not repair a wrong type or make arbitrary content safe. Check the inventory while deploying it, and fix any rejected script/style response rather than removing the protection:

X-Content-Type-Options: nosniff

Check the server mapping

Static servers usually derive types from extensions. If a new extension is served as application/octet-stream, decide whether that is intentional before adding a global override. A narrow mapping is easier to review than a rule that changes every response.

Repeat the checks at the public hostname. A CDN or reverse proxy can replace a correct origin header, and a friendly error page can return HTML for a path whose suffix looks like CSS or JavaScript.

Sources

RFC 9110 — HTTP Semantics, representation metadata and Content-Type, checked 2026-10-15.
MDN: Content-Type, media types and charset examples, checked 2026-10-15.
MDN: X-Content-Type-Options, nosniff behavior, checked 2026-10-15.
WHATWG Fetch Standard, script-like and style destination blocking, checked 2026-10-15.

HEAD should describe the same representation as GET

Mon, 22 Jun 2026 00:00:00 +0000

The HEAD method asks for the response a server would send to GET, without transferring the response content. It is useful for checks that need status, validators, type, or length, but only when the metadata remains aligned with the real GET path.

Match the URL, request headers, release, and content coding before comparing HEAD with GET. Permitted omissions are not automatic failures. Keep a small GET body or checksum check, because plausible metadata cannot prove that the payload is intact.

Compare a normalized header set

Dynamic headers such as Date will differ between requests. Compare the fields that describe the selected representation:

url=https://www.example.test/downloads/manual.pdf

curl -sS -I "$url" > head.headers
curl -sS -D get.headers -o /dev/null "$url"

for file in head.headers get.headers; do
 printf '== %s\n' "$file"
 awk '
 /^HTTP\// ||
 tolower($1) ~ /^(content-type:|content-length:|content-encoding:|cache-control:|etag:|last-modified:|vary:)$/ { print }
 ' "$file"
done

Run this in a disposable working directory; the two header files are overwritten. Match the URL, request headers, and release version. Compression negotiation belongs in the comparison: send the same Accept-Encoding value to both requests.

HEAD should describe the corresponding GET, but RFC 9110 permits omission of fields determined only while generating content. Missing Content-Length alone is not a failure. If present on HEAD, it must be the length the corresponding GET content would have, not zero just because HEAD has no body. Compare encoded lengths consistently and investigate changed validators or types rather than requiring byte-identical header lists.

Do not use HEAD as the only release check

A server can synthesize plausible headers while the file is missing, unreadable, or truncated. Keep one small GET in the smoke test and verify its body or checksum. HEAD reduces transfer cost; it does not prove the payload.

Check redirects explicitly

Without --location, curl -I reports the first response. That is useful when testing redirect policy. With --location, it prints header blocks for the followed responses, including intermediate redirects and the final destination. Record which question the command answers:

curl -sS -I --max-redirs 0 https://old.example.test/manual
curl -sS -I --location https://old.example.test/manual

Treat method-specific failures as configuration defects

A static page that returns 200 to GET and 405 or 404 to HEAD usually exposes a proxy or routing inconsistency. Fix the method handling instead of teaching monitors to ignore it, unless the endpoint contract deliberately disallows HEAD.

Sources

RFC 9110 — HTTP Semantics, HEAD method semantics, checked 2026-10-15.
curl manual, --head, --include, and redirect options, checked 2026-10-15.

Compression changes the selected HTTP representation

Mon, 29 Jun 2026 00:00:00 +0000

Compression saves transfer bytes, but it also creates another representation of a resource. A client announces acceptable content codings with Accept-Encoding; the server describes the applied coding with Content-Encoding.

Compression selects representation bytes, so negotiation, cache matching, and validators must agree. Distinct coded bytes need distinct strong tags. Compare the decoded download with the artifact; an encoded Content-Length need not equal the decoded file size.

Make negotiation visible

When a response can change according to Accept-Encoding, include that request field in Vary. This tells shared caches that a compressed response and an identity response are not interchangeable cache entries.

Compare the two paths instead of checking only a browser request:

curl -sS -D - -o /dev/null \
 -H 'Accept-Encoding: identity' https://www.example.test/
curl -sS --compressed -D - -o /dev/null \
 https://www.example.test/

Inspect Content-Encoding, Vary, Content-Length when present, and the validator. Strong entity tags must differ when content coding changes the representation bytes. A shared weak tag is possible for semantically equivalent variants, but it cannot support strong comparison. Check conditionals against the same negotiated representation.

Compress appropriate content

Text formats such as HTML, CSS, JSON, XML, and SVG usually benefit from compression. Formats that are already compressed might not. Caddy’s encode directive negotiates supported encodings and normally avoids encoding a response without an appropriate content type or minimum size.

Do not add compression solely to satisfy a score. Measure a representative response, confirm that the edge emits the intended headers, and verify that downloadable files retain the correct content type and checksum after transfer decoding.

With --compressed, curl decodes the body automatically but leaves saved response headers unchanged. A reported Content-Length, when present, describes encoded content, so it need not equal the size of the decoded download. Compare a checksum with the artifact only after content decoding; HTTP transfer framing is a separate layer. The option also does not guarantee compression: inspect the actual response, especially for small or already-compressed files.

Sources

RFC 9110 — HTTP Semantics, content codings and Vary, checked 2026-10-15.
Caddy encode directive, content encoding selection and negotiation, checked 2026-10-15.
curl manual, --compressed, checked 2026-10-15.

Range requests should prove which bytes were selected

Mon, 06 Jul 2026 00:00:00 +0000

Range requests let a client ask for part of a representation. Media players, resumable download tools, and diagnostic clients use them, but a server that advertises or returns ranges incorrectly can produce silent corruption instead of a clean failure.

Keep the representation and content coding stable. Match interval, total size and actual body bytes; a complete 200 can be valid when Range is ignored. Resume only with an appropriate strong validator.

Establish the complete size

Use an immutable test download of at least 16 bytes. Run the fragments in the same disposable shell session so url and size remain available. Their explicit set -eu stops after a failed transfer, size check, or comparison; inspect the HTTP status and range fields separately. Save the complete body, headers, and validator in that temporary working directory:

set -eu
url=https://www.example.test/downloads/archive.tar.gz
curl -fsS -H 'Accept-Encoding: identity' -D full.headers -o full.body "$url"
size=$(wc -c < full.body)
test "$size" -ge 16
awk 'tolower($1) ~ /^(content-length:|content-encoding:|etag:|last-modified:|accept-ranges:)$/ { print }' full.headers

Check that the full response is 200 and has no unexpected content coding. The commands request the identity representation and do not use automatic decompression; byte ranges apply to the selected encoded representation, so changing encoding invalidates the comparison. Content-Length may be absent, which is why the complete body supplies the size.

Accept-Ranges: bytes is an explicit signal, but absence of the header does not by itself prove that range requests are unsupported. Exercise the request.

Request a known interval

Ask for the first 16 bytes and keep both headers and body:

set -eu
curl -sS -D range.headers -o range.body \
 -H 'Accept-Encoding: identity' -H 'Range: bytes=0-15' "$url"

wc -c range.body
head -c 16 full.body > expected.body
cmp expected.body range.body
awk 'tolower($1) ~ /^(content-range:|content-length:|etag:)$/ || /^HTTP\// { print }' range.headers

A successful byte-range response uses 206 Partial Content. Content-Range should identify 0-15 and the complete representation length, while the body should contain exactly 16 bytes.

The byte comparison deliberately tests the partial-body case. A valid full 200 larger than 16 bytes will fail it and stop this shell session; inspect the saved headers before calling that corruption. Conversely, a complete 16-byte file can pass cmp even with 200, so status and Content-Range still need checking.

Test an impossible range

Start at the complete byte count: for a file of size N, the last valid offset is N−1. Unlike a fixed large constant, this is outside the known representation:

set -eu
curl -sS -D - -o /dev/null \
 -H 'Accept-Encoding: identity' -H "Range: bytes=${size}-" "$url"

When the server rejects an unsatisfiable byte range with 416 Range Not Satisfiable, it should include Content-Range, for example bytes */12345, to report the current size. A 200 response may be valid when the server ignores the range, but it must return the complete representation rather than a mislabeled fragment.

Keep validators in the story

Resumed downloads can use If-Range with a strong ETag. A weak ETag is forbidden; an HTTP date is allowed only when no ETag is available and the date qualifies as a strong validator. If the representation changed, the safe response is the full current representation, not bytes spliced from two versions. Test this behavior before calling a download resumable.

Sources

RFC 9110 — HTTP Semantics, range units, partial responses, and Content-Range, checked 2026-07-06.
curl manual, --range and header options, checked 2026-07-06.

Cache headers are a contract with two clocks

Thu, 25 Jun 2026 00:00:00 +0000

Caching is not a single switch. A response has a freshness decision made by a cache and a validation conversation with the origin. Mixing those two ideas produces either stale content or needless requests.

Storage, freshness, and validation answer separate questions. Match the policy to how the URL changes, preserve validators exactly, and check shared-cache variants.

Pick the change model first

For an HTML document that may change after publication, Cache-Control: no-cache is usually a more accurate starting point than no-store. no-cache permits storage but requires a cache to validate before reuse; no-store asks caches not to store the response at all. The distinction matters when you want conditional requests and a cheap 304 Not Modified response.

For a fingerprinted asset such as app.4f2c.js, the URL changes when the bytes change. A long freshness lifetime can then be reasonable:

Cache-Control: public, max-age=31536000, immutable

That policy is unsafe for a stable URL whose contents can be replaced in place. If the URL is stable, prefer a shorter lifetime and a validator such as an entity tag (ETag) or a meaningful Last-Modified value.

Choose who may store the response as well as how long it stays fresh. Use the unqualified private directive when shared caches must not store user-specific content; a browser’s private cache can still store it. no-cache alone does not provide that restriction. The public example above is for assets safe to share across users, not a default for personalized responses.

Inspect both paths

Make one ordinary request and one conditional request. The exact server flags vary, but the evidence you want is stable:

curl -sS -D - -o /dev/null https://www.example.test/
curl -sS -D - -o /dev/null \
 -H 'If-None-Match: "copy-a-real-etag-from-the-first-response"' \
 https://www.example.test/

Do not paste a made-up validator into a runbook. Copy the value from the first response or use a test fixture. For this conditional GET, a matching If-None-Match can yield 304 using weak comparison: it permits semantic equivalence, not necessarily byte identity. A changed validator normally produces a new 200 response. Keep the ETag’s quotes and any W/ prefix intact.

Keep intermediaries visible

Age, Vary, and the cache-control directives explain why two clients can receive different results. If a response varies by Accept-Encoding, language, or another request field, that dimension belongs in Vary; otherwise a shared cache can reuse the wrong representation.

The useful release check is not “the scanner found a cache header”. It is “the header matches the replacement policy, and a conditional request behaves as documented”.

immutable applies during the advertised freshness lifetime; it is not a permanent ban on revalidation. Keep versioned URLs immutable and older assets reachable while old HTML can still reference them. Changing today’s cache header does not retroactively shorten a previously cached response’s lifetime, so a cache-policy mistake may require a new URL or an explicitly supported cache purge.

Sources

RFC 9111 — HTTP Caching, sections 4–5, checked 2026-10-15.
RFC 9110 — HTTP Semantics, sections 8 and 13, checked 2026-10-15.
MDN: Cache-Control, directive semantics and defaults, checked 2026-10-15.
RFC 8246 — Immutable Responses, freshness-limited immutable semantics, checked 2026-10-15.

Redirects and error codes should preserve the story

Tue, 14 Jul 2026 00:00:00 +0000

An HTTP status is part of the interface. A browser may render a friendly page, but a monitor, crawler, cache, and API client also need the status code and headers to tell the same story.

Choose relocation or absence deliberately. Preserve the required method, resolve Location, account for caching, and keep error-page status truthful.

Permanent moves are not all interchangeable

308 Permanent Redirect communicates that the resource has a new permanent URI while preserving the request method. That is important for a client that sends POST, PUT, or another non-GET method. A redirect from an old page to a new page is easy to test with GET; a migration of an endpoint must also be tested with the method that real callers use.

curl -sS -D - -o /dev/null https://old.example.test/api/item

Without --location, curl leaves the redirect unfollowed. Resolve Location against the original URI: an absolute URI, root-relative path, or other relative reference can be valid. Check the resolved host, path, and scheme. 301/302 permit changing POST to GET; 307/308 preserve the method. Test state-changing methods only against an isolated endpoint. 308 is heuristically cacheable, subject to method and cache controls, so a cached move can outlive a server configuration change.

If the intent is to show a result after a completed action, 303 See Other points to a separate resource that the client can retrieve with GET or HEAD. It does not ask the client to replay the original write at a new address; choose that behavior deliberately rather than treating every redirect as a relocation.

Distinguish missing from removed

Use 404 Not Found when the server cannot find a current representation and the absence may be temporary or unknown. 410 Gone is stronger: it says access is no longer available and the condition is likely permanent; it does not establish why the resource disappeared. Both should have a body that helps a human, but neither should return 200 just because a custom error page rendered successfully.

404 can also conceal whether a forbidden resource exists. Both 404 and 410 are heuristically cacheable, subject to method and cache controls. Choose an explicit policy for error responses: a cached missing-route result can remain visible after the route is repaired.

The check belongs at the edge:

curl -sS -D missing.headers -o missing.html -w '%{http_code} %{url_effective}\n' \
 https://www.example.test/a-route-that-does-not-exist

Keep the response headers on the error path. Security headers, a useful content type, and a correct status are more valuable than a decorative error animation.

Sources

RFC 9110 — HTTP Semantics, sections 15 and 13, checked 2026-07-14.
RFC 7538 — 308 Permanent Redirect, section 3, checked 2026-07-14.
curl manual, status and redirect inspection flags, checked 2026-07-14.

robots.txt and sitemaps answer different questions

Tue, 21 Jul 2026 00:00:00 +0000

Two small files are often treated as if they were a security boundary. They are not. A robots.txt file describes crawler preferences, while a sitemap gives search systems a list of URLs that belong to the site.

Discovery aids do not grant or deny access. Follow every listed sitemap, validate more than XML syntax, and handle robots error responses according to their actual class.

What robots.txt can and cannot do

The rules are published at the origin’s /robots.txt path and are read by cooperating crawlers. They do not protect a private directory, remove a URL from the network, or replace authentication. A route that must not be public needs server authorization or a network boundary.

Keep the file boring and test its actual location:

User-agent: *
Disallow:
Sitemap: https://www.example.test/sitemap.xml

The file should be UTF-8. If a deployment redirects it, verify the crawler behavior you intend rather than assuming every client follows the same chain.

What a sitemap adds

A sitemap is an inventory, not a permission list. Use absolute URLs from one host, encode XML values correctly, and include only canonical pages that you are prepared to publish. Do not list a URL merely because it exists in a build directory.

Each sitemap may contain at most 50,000 URLs and 50 MB (52,428,800 bytes) uncompressed; gzip does not raise that limit. An index is optional for a small site, but multilingual generators may legitimately produce one pointing to separate language sitemaps. Inspect the root element and follow each listed sitemap rather than assuming /sitemap.xml is a URL set. Validate it as XML and request it through the same HTTPS edge as a normal page.

curl -fsS https://www.example.test/robots.txt
curl -fsS -o sitemap.xml https://www.example.test/sitemap.xml && \
 xmllint --noout sitemap.xml

Separate download and parsing so a failed fetch cannot be hidden by a successful parser. Well-formed XML is only a syntax check: also validate the namespace, URL count, canonical hosts, and whether listed pages exist. If xmllint is not installed, parse the XML in the same CI environment that publishes it. The important part is that the check runs before the file is replaced at the edge.

A robots Disallow is not a reliable de-indexing instruction. Also test error responses: RFC 9309 permits crawling when robots is unavailable with a 4xx response, while a network/server error initially requires complete disallow. Accidentally serving the HTML fallback with status 200 can therefore change crawler behavior even though the URL appears reachable.

Sources

RFC 9309 — Robots Exclusion Protocol, robots rules, file location, and user-agent matching, checked 2026-07-21.
Sitemaps Protocol, sitemap XML structure and discovery, checked 2026-07-21.

HTTP methods are a promise about side effects

Thu, 02 Jul 2026 00:00:00 +0000

A request method tells the server what the client is asking it to do. A read-only file origin has a narrow contract, but “static site” alone does not prove that proxies or imported handlers expose no write operations. Check the route actually serving the resource.

Separate requested state change from intended repeat effect, then inspect the actual route. The file-handler boundary is scoped; method support is separate from CORS permission.

Safe and idempotent answer different questions

GET and HEAD are safe: the client does not request a state-changing action. Incidental logging, metrics, or accounting can still change server state. PUT and DELETE are idempotent in their defined semantics, but are not safe: repeating the same request is intended to have the same effect as performing it once. That does not require identical response statuses, and it does not make every retry after a network failure appropriate.

For a read-only static origin:

GET retrieves a representation.
HEAD describes the corresponding GET without response content.
OPTIONS asks about the target’s communication options; a file handler may reject it.
POST / PUT / DELETE / PATCH should not fall through to successful file serving unless a separate, intended handler implements their semantics.

A Caddy file handler’s method boundary

Caddy v2.10.2’s file-serving path accepts GET and HEAD and rejects other methods with 405 and Allow: GET, HEAD. Redirects, missing paths, error handlers, and other configured routes can answer earlier or differently, so this is not a universal response for every static-site URL.

The earlier note recorded:

$ curl -s -o /dev/null -w 'OPTIONS -> HTTP %{http_code}\n' -X OPTIONS https://www.xbcatm.com/
OPTIONS -> HTTP 405

The original response log is not retained here, and this review did not reproduce that live result. The command reports only status; it does not establish that the required Allow field was present. Inspect that field separately. 405 means a known method is not supported for this target; 501 Not Implemented is the usual distinction when the server does not recognize or implement a method at all.

HEAD parity allows documented omissions

HEAD should describe the same selected representation as GET, while omitting content. Fields calculated only while generating the content may be omitted. If Content-Length is present, it must describe the corresponding GET content, using the same content coding.

The earlier note reported this comparison:

HEAD content-length: 15890
GET bytes read : 15890

Those historical numbers are not a fixed expected size or a current verification result. A missing HEAD length alone is legal; a wrong supplied length needs investigation. Follow the head-request-parity note for paired probes. The downloadable check-http-headers.sh helper inspects a GET response and does not perform this comparison automatically.

Reject unintended writes without hiding the contract

For a method recognized but disallowed on the target, return 405 with Allow listing the methods actually supported. The list is not necessarily restricted to safe methods on an application route. Do not return a plausible 200 page for an ignored write request.

If browser clients need CORS preflight, define that OPTIONS behavior explicitly; method support and CORS permission are separate decisions. Test write rejection only against an authorized disposable fixture rather than assuming a public endpoint is harmless because the site looks static.

Sources

RFC 9110 — HTTP Semantics, safe/idempotent methods and HEAD/405/501 semantics, checked 2026-10-15.
MDN: HTTP request methods, method overview, checked 2026-10-15.
curl manual, method and response-inspection options, checked 2026-10-15.
Caddy v2.10.2 file-server source, method rejection after file selection, checked 2026-10-15.
WHATWG Fetch Standard, CORS preflight processing, checked 2026-10-15.

Validators decide whether a cached copy is still the answer

Tue, 30 Jun 2026 00:00:00 +0000

A cache can ask whether its stored copy is still usable without always downloading the body again. It sends a validator; a matching conditional GET can receive 304 Not Modified, while a changed representation normally needs a new 200 response. Validation saves bytes on a hit, not on every request.

Illustrative tags; otherwise successful requests. GET/HEAD use weak comparison. HEAD has no body. Matching is neither byte identity nor freshness. Other methods normally return 412; If-Range cannot use weak ETags.

Validator type and strength are different questions

An ETag is an opaque, quoted value. It can be strong ("release-a") or explicitly weak (W/"release-a"). Strong comparison requires both tags to be strong and equal; weak comparison compares the opaque values regardless of the W/ marker. Weak tags can describe semantic equivalence even when bytes differ.

Last-Modified is a timestamp with one-second HTTP-date resolution. It is usually weak unless the conditions in RFC 9110 section 8.8.2.2 establish it as strong. Do not classify every ETag as strong or every timestamp as unconditionally weak.

The earlier site observation recorded:

etag: "dkx84jwth7nkc9e"
last-modified: Mon, 24 Aug 2026 13:55:23 GMT

These are historical values; the original response log is not retained here and the live result was not rechecked in this review. Caddy’s file server normally sets both fields, subject to file metadata and configuration.

Conditional GET uses weak comparison

Send the ETag exactly as received in If-None-Match, including its quotes and any W/. The original note reported HTTP 304 for the value above but printed an unquoted request value. That request text was not valid entity-tag syntax and should not be copied as a working probe. Obtain a current validator before testing a live endpoint.

This socket-free exercise illustrates the comparison for individual, already-parsed tags; it is not a complete HTTP header parser:

def weak_value(tag):
 return tag[2:] if tag.startswith('W/') else tag

def strong_equal(a, b):
 return not a.startswith('W/') and not b.startswith('W/') and a == b

stored = 'W/"release-a"'
current = '"release-a"'
assert weak_value(stored) == weak_value(current)
assert not strong_equal(stored, current)
assert weak_value(stored) != weak_value('"release-b"')
print("weak match: True; strong match: False; changed tag: no match")

For an otherwise successful GET or HEAD, a matching If-None-Match produces a body-free 304. The same false precondition on another method normally produces 412 Precondition Failed. If both If-None-Match and If-Modified-Since are present, the ETag condition takes precedence. Byte-range validation with If-Range cannot use a weak ETag.

Republish without preserving a misleading validator

An ETag is not necessarily a content hash. Caddy v2.10.2’s default file tag is derived from modification time and size; replacing bytes while preserving both can preserve the tag. Inspect the deployed version and any configured overrides, then make the release process update the validator’s inputs or use content-based versioning.

Keep byte checksums in the release manifest and test both unchanged and changed responses. A checksum does not automatically become the server’s ETag. A 304 also needs the applicable cache metadata so a cache can update its stored response; it is not merely an empty success status.

Sources

RFC 9110 — HTTP Semantics, sections 8.8 and 13 on validator strength and preconditions, checked 2026-10-15.
RFC 9111 — HTTP Caching, section 4.3 on validation and updating stored responses, checked 2026-10-15.
MDN: HTTP conditional requests, conditional-request overview, checked 2026-10-15.
Caddy v2.10.2 file-server source, calculateEtag inputs and default response fields, checked 2026-10-15.

Read a live edge the way a client does

Fri, 07 Aug 2026 00:00:00 +0000

The HTTP-foundations notes each take one contract and show single pieces of evidence. This tutorial walks the whole path a single client request touches, in order, using only read-only commands. The aim is to distinguish protocol requirements from choices made by a particular server, rather than demand the same status and headers from every endpoint.

We use www.xbcatm.com throughout. Replace it with the host you are inspecting. The requests use safe HTTP methods; they can still create access logs or consume rate limits. Inspect only endpoints you are permitted to query. Output blocks below are illustrative, not current production measurements. Run shell snippets in a disposable working directory, keep failures visible, and record time and client version.

Keep the selected representation consistent, and interpret each observed response within its own contract. No live edge was queried for this illustration.

1. Start with what a method may ask

Before you read a response body, pin down what the server will allow you to ask. A static handler often supports GET and HEAD, while a proxy may also implement OPTIONS. Ask the edge about the target resource:

$ curl -sS --max-time 20 -D - -o /dev/null -w 'OPTIONS -> HTTP %{http_code}\n' -X OPTIONS https://www.xbcatm.com/
HTTP/2 405
allow: GET, HEAD

OPTIONS -> HTTP 405

405 Method Not Allowed is valid when OPTIONS is recognized but not supported for this resource, and a 405 must include Allow. A successful 200 or 204 OPTIONS response can also be correct. 501 means the method is not implemented by the server. An OPTIONS result is not proof that every other method behaves correctly. Do not send POST merely to test rejection on an unfamiliar live service: it is not a safe method, and a 200 alone does not describe its side effects.

2. Read the representation line

The first line tells you the negotiated exchange; the headers then tell you what kind of bytes are coming. Capture them exactly as a client sees them, discarding the body:

$ curl -sS --max-time 20 -D - -o /dev/null https://www.xbcatm.com/
HTTP/2 200
accept-ranges: bytes
content-length: 15890
content-security-policy: default-src 'self'; base-uri 'self'; form-action 'none'; frame-ancestors 'none'; object-src 'none'; script-src 'none'; style-src 'self'; img-src 'self' data:
content-type: text/html; charset=utf-8
etag: "dkx84jwth7nkc9e"
last-modified: Mon, 24 Aug 2026 13:55:23 GMT
strict-transport-security: max-age=31536000; includeSubDomains
x-content-type-options: nosniff

Three headers deserve a pause. content-type: text/html; charset=utf-8 names the media type, and content-type-and-nosniff shows why it must describe the actual bytes you send. etag: "dkx84jwth7nkc9e" hands you a strong validator. x-content-type-options: nosniff constrains browser MIME handling, notably for script and stylesheet fetches; it does not validate the body as HTML. Each is evidence, not decoration, and together they say: this response advertises metadata to compare with its body and intended policy. A syntactically strong ETag is a server assertion, not independent proof of a content hash.

3. Compare HEAD and GET for the same representation

HEAD has GET semantics without a response body. Servers should return the same header fields but may omit fields whose values are determined only while generating content. Content-Length is optional; if sent on HEAD, it must equal the size of the content a corresponding GET would send. Compare stable requests with the same encoding and other selection headers:

$ curl -sSI --max-time 20 -H 'Accept-Encoding: identity' https://www.xbcatm.com/ | grep -i content-length
content-length: 15890
$ curl -sS --max-time 20 -H 'Accept-Encoding: identity' https://www.xbcatm.com/ | wc -c
15890

They match: 15890 on both. That is the head-request-parity contract. An absent HEAD length alone is not a protocol violation. A mismatched value needs investigation, including a deployment between the requests or different selected representations. A pipeline to wc can hide curl failure in shells without pipefail; for an automated gate, save the body only after curl succeeds and count that file.

4. Ask for precise bytes with a range

Because the origin knows its representation, it can serve part of it. A range request asks for a byte span and gets only that span:

$ curl -sS --max-time 20 -D - -o /dev/null -H 'Accept-Encoding: identity' -H 'Range: bytes=0-99' -w 'HTTP %{http_code} bytes=%{size_download}\n' https://www.xbcatm.com/
HTTP/2 206
content-range: bytes 0-99/15890
content-length: 100

HTTP 206 bytes=100

For a representation at least 100 bytes long, a successful single-range response should have 206, a matching Content-Range: bytes 0-99/…, and 100 content bytes. A server may legally ignore Range and return a full 200; that does not imply it lacks validators. An unsatisfiable range can return 416. A plain Range request does not require an ETag. Resuming a particular representation with If-Range, or safely combining partial responses, needs the relevant strong-validator rules. Ranges refer to the selected encoded bytes, hence the explicit identity encoding here.

5. Let the cache revalidate with a validator

Now combine method, representation, and byte selection through caching. A fresh stored response may be reusable without checking a validator. To test revalidation, capture the current response and replay its actual ETag using the same representation selection. Do not copy an old tag from an article:

set -eu
url=https://www.xbcatm.com/
work=$(mktemp -d)
status=$(curl -sS --max-time 20 -H 'Accept-Encoding: identity' \
 -D "$work/headers" -o "$work/body" -w '%{http_code}' "$url")
test "$status" = 200
etag=$(awk 'tolower($1)=="etag:" {sub(/\r$/, ""); sub(/^[^:]*:[ \t]*/, ""); print; exit}' "$work/headers")
test -n "$etag" || { echo 'No ETag: inspect Last-Modified instead' >&2; exit 1; }
curl -sS --max-time 20 -H 'Accept-Encoding: identity' \
 -H "If-None-Match: $etag" -D - -o /dev/null \
 -w 'HTTP %{http_code}\n' "$url"

Run this fragment as a standalone shell script. It must finish the initial transfer and require status 200 before storing a usable response; an error representation can also have an ETag. If the selected representation still matches and normal processing succeeds, expect 304 and reuse that complete stored body with updated metadata. A changed representation can correctly return 200. Keep the temporary directory while inspecting the result, then remove only that generated directory when finished. If-None-Match uses weak comparison, so a 304 need not prove byte identity. Freshness lifetime and current age decide reuse without validation; a validator answers a different question and does not itself set that lifetime.

6. Negotiate a smaller representation when you can

The same content can travel in fewer bytes when the client says it can decompress. Offer gzip and watch the body shrink while the meaning stays identical:

$ curl -sS --max-time 20 -H 'Accept-Encoding: identity' -o /dev/null -w 'no-enc: %{size_download} bytes\n' https://www.xbcatm.com/
$ curl -sS --max-time 20 -o /dev/null -H 'Accept-Encoding: gzip' -w 'gzip: %{size_download} bytes\n' https://www.xbcatm.com/
no-enc: 15890 bytes
gzip: 4368 bytes

These example sizes illustrate a possible reduction; the server may choose identity instead. That is the http-compression-negotiation note’s core idea: changing the selected representation changes the bytes on the wire, and the metadata (content-length, encoding) changes with it. A client that requests compression and a cache that stores the encoded form must agree on which representation they selected, otherwise the cache can serve a mismatch. Inspect Content-Encoding and Vary: Accept-Encoding, not just the byte count. A manually supplied Accept-Encoding does not make curl automatically decompress; --compressed requests supported encodings and decodes the response. Compare decoded bodies to establish equal content, and do not assume strong ETags remain identical across encodings.

7. Follow the story when a URL changes

When identity changes, the response should say where the story continues rather than guess the body:

$ curl -sS --max-time 20 -o /dev/null -w '%{http_code} -> %{redirect_url}\n' -I http://www.xbcatm.com/ | head -1
308 -> https://www.xbcatm.com/

308 Permanent Redirect keeps the method and preserves the story, which is the concern of redirects-and-status-codes. Whether it is a 301/308 for a permanent move, or a 404 handled by the site, the status line is the first sentence of the story and should be read before the body.

8. Let the crawler read the map last

The edge is also read by machines that do not request HTML at all. Discovery answers two different questions, as robots-and-sitemaps explains. Confirm discovery points at the real sitemap:

$ curl -sS --max-time 20 https://www.xbcatm.com/robots.txt
User-agent: *
Disallow:
Sitemap: https://www.xbcatm.com/sitemap.xml

Fetch the sitemap URL too and check canonical host and language URLs. robots.txt is crawler guidance, not access control; listing a URL in a sitemap does not guarantee crawling or indexing.

The whole path in one glance

Check	What to inspect	Note
`OPTIONS` status + Allow	observed method policy for this resource	http-method-semantics
`content-type` + body + `nosniff`	advertised type and MIME-handling policy	content-type-and-nosniff
HEAD/GET metadata	corresponding representation, with allowed omissions	head-parity
Range + Content-Range	observed partial-transfer support	http-range-requests
captured `ETag` -> 304	conditional representation match	conditional-requests
308 follows method	redirects preserve the story	redirects-and-status-codes
robots -> sitemap	discovery answers differ	robots-and-sitemaps

Each row was a note; now it is a command you can run against any edge. Read the linked note when you need the details, and keep this table as your re-check before you trust an endpoint.

Sources

WHATWG Fetch Standard, nosniff processing and request destinations, checked 2026-10-15.
RFC 9110 — HTTP Semantics, methods, validators, and status codes, checked 2026-10-15.
RFC 9111 — HTTP Caching, freshness and revalidation, checked 2026-10-15.
RFC 7538 — 308 Permanent Redirect, method-preserving redirect, checked 2026-10-15.
curl manual, read-only inspection flags, checked 2026-10-15.