Web · XB Field Notes

nginx chooses the first matching regex, not a specificity score

Mon, 27 Dec 2027 00:00:00 +0000

nginx does not assign a general specificity score to every location expression. Ordinary prefixes are selected by matching length, exact locations can finish the search, and competing regex locations are evaluated in declaration order. A regex that looks narrower to a human can lose to a broader expression placed earlier. Mixing these selection rules is a common reason apparently careful routing changes affect unexpected paths.

When it breaks

A general suffix regex for .css can win before a later regex restricted to /assets/. The later expression may look more specific because it names a directory, but nginx does not compare the apparent specificity of those two patterns. If they both match, their order decides which configuration handles the request in the flat structure used here.

Reordering ordinary prefix blocks is a different operation. Moving /assets/ above / does not make it win more strongly; its longer matching prefix already gives it priority when the prefix result applies. A review that treats all location declarations as one first-match list can therefore make ineffective changes while missing the regex order that actually controls the failing route.

A shorter ^~ prefix is also not a permanent shield around an entire subtree. If a longer ordinary prefix is selected, the shorter marked prefix is no longer the candidate that suppresses the regex phase. Adding one seemingly harmless more-detailed prefix can therefore allow a suffix regex to take requests that previously stayed in the marked location.

Flat nginx 1.28.0 locations use exact completion, longest-prefix selection and ordered regexes as distinct rules. Route markers do not validate downstream policy.

Check it quickly

These are historical article reports, not a rerun; raw integration logs were not recovered in the retained validation files. Use locally available pinned images with assertions enabled; cleanup is best-effort.

Run the script with host Python, Docker, and the pinned nginx image. The recorded environment used nginx 1.28.0 on Linux arm64 with Docker 29.3.1. It builds two temporary configurations, checks each with nginx -t, and issues real requests through host-loopback ports. Each configuration uses only flat locations, and cleanup attempts to remove its containers, image tags, and private network:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True);target.write_text(content)
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'

prefixes=[
 'location / { return 200 "fallback"; }',
 'location /assets/ { return 200 "asset-prefix"; }',
 'location = /assets/exact.css { return 200 "exact"; }',
 'location ^~ /static/ { return 200 "static-prefix"; }',
 'location /static/dynamic/ { return 200 "dynamic-prefix"; }']
regexes=[r'location ~ \.css$ { return 200 "generic-css"; }',r'location ~ ^/assets/ { return 200 "asset-regex"; }']
for reverse,expected in [(False,'generic-css'),(True,'asset-regex')]:
 ordered_prefixes=list(reversed(prefixes)) if reverse else prefixes
 ordered_regexes=list(reversed(regexes)) if reverse else regexes
 locations='\n'.join(ordered_prefixes+ordered_regexes)+'\n'+r'location ~* \.txt$ { return 200 "case-folded-text"; }'
 with lab(config(locations)) as (container,request):
 cases={'/assets/file.css':expected,'/assets/exact.css':'exact','/static/file.css':'static-prefix','/static/dynamic/file.css':'generic-css','/static/dynamic/plain':'dynamic-prefix','/README.TXT':'case-folded-text','/elsewhere.CSS':'fallback','/elsewhere':'fallback'}
 for path,wanted in cases.items():
 status,headers,body=request(path);assert status==200 and body.decode()==wanted,(path,status,body)
print('reversing the matching regex declarations changes the winner; the first matching regex is not necessarily the most specific')
print('exact locations and longest-prefix selection ignore declaration order; a longer ordinary prefix can make a shorter ^~ prefix irrelevant')

The first configuration places the .css regex before the regex beginning with /assets/. Requesting /assets/file.css returns generic-css. The second configuration reverses those regexes, and the identical request returns asset-regex. Nothing in the request became more or less specific; only the order of two simultaneously matching patterns changed.

The script also reverses the ordinary prefix declarations between configurations. Their relevant results remain stable. /static/dynamic/plain continues to use the longer ordinary prefix, and the broad fallback handles an unrelated path. This comparison separates textual rearrangement of prefixes from textual rearrangement of regexes instead of assuming both edits have the same routing effect.

An exact /assets/exact.css location wins in both configurations despite matching both regex patterns. It returns exact. The test therefore includes a route deliberately isolated from regex ordering, rather than presenting the first matching regex as an unconditional rule that overrides every other location type.

The path /static/file.css stays in the ^~ /static/ prefix and returns static-prefix. But /static/dynamic/file.css has a longer ordinary prefix, so the .css regex is considered and returns generic-css. This verifies the selected-prefix qualification on ^~; the modifier belongs to that location, not to an invisible inherited protection on every deeper prefix.

Case modifiers are tested separately. The ~* text-file pattern matches /README.TXT, while the case-sensitive .css pattern does not match /elsewhere.CSS, which reaches the fallback. These are observations for the explicit regexes and Linux environment in the fixture, not a claim that all filesystem paths or every platform use identical case rules.

Make overlaps visible in review

For each regex, identify paths that also match earlier expressions and record which handler is intended to win. Include boundary cases where two patterns overlap, as well as cases where neither matches. A test that exercises only one regex in isolation cannot reveal the priority conflict introduced when another expression is added before it.

Use exact or prefix locations when they naturally describe the route, and keep intentional regex overlaps understandable. A regex is not inherently wrong, but its ordering becomes part of the service contract. Changes to included configuration can alter the effective sequence, so inspect the complete loaded configuration rather than reviewing a file fragment without its surrounding context.

When adding a longer prefix below an existing marked prefix, retest suffix routes through that new path. Do not assume the earlier ^~ still governs the request. Finally, verify downstream behavior such as cache headers, proxying, and access control in the winning block; selecting the expected marker is a routing test, not proof that every policy inside that block is correct.

Sources

nginx core location directive, first matching regex, prefix priority, and case modifiers, checked 2026-10-15.
How nginx processes a request, configuration selection inside the matched server, checked 2026-10-15.
nginx 1.28.0 core implementation, version-specific prefix and regex location-search behavior, checked 2026-10-15.

Frame policy names who may embed your page

Thu, 23 Sep 2027 00:00:00 +0000

Clickjacking deceives a user about which interface receives an action. A hostile parent can place a framed application behind a misleading visual layer so that a click reaches a real control in that application. The parent need not read the child’s DOM for the user to interact with it. Same-origin read restrictions therefore do not replace a decision about whether the application may be embedded at all.

The embedded response can declare Content-Security-Policy: frame-ancestors 'none' to disallow embedding, or use an explicit set of permitted ancestors. The named origins belong to the pages allowed to contain this document. If a merchant embeds a payment page, the payment page’s policy needs to permit the merchant’s origin; listing the payment widget’s own origin does not express that relationship.

Put each restriction on the right response

frame-ancestors answers who may embed the current document. frame-src answers which frame sources the current document may load. A restrictive frame-src on the child does not stop a different parent from embedding that child. Similarly, default-src 'none' does not supply a fallback for frame-ancestors. These directives control different directions of the relationship and should not be substituted merely because their names contain “frame.”

Frame-ancestors is not supported through a CSP meta element; deliver it as a response header. For nested embedding, every ancestor must match the policy, not just the immediate parent. A widget legitimately allowed inside one application can still be blocked when that application itself appears inside an unapproved outer frame. Review the complete intended embedding chain before choosing an allowlist.

When it breaks

A deployment may add the policy to the container page while leaving the sensitive child response unrestricted. Another may put it in HTML metadata and see a reassuring configuration string that the browser does not enforce. Verify the response actually being framed and test browser rendering. An HTTP 200 alone proves neither successful embedding nor the absence of frame protections: browsers can receive a document and refuse to display it in that context.

Conflicting headers can also mislead reviewers. In modern supporting browsers, an enforced frame-ancestors directive takes precedence over X-Frame-Options for that response. Pairing a permissive ancestor policy with X-Frame-Options: DENY does not necessarily produce the stricter result someone expects. Keep any deliberately retained header policy consistent with the intended audience and browser support, rather than assuming every restriction combines the same way as multiple CSP policies.

Check it quickly

Save this as frame-policy.py, run python3 frame-policy.py --serve, and open the printed parent URL. Python 3.14.4 and Chrome 152 were used. The parent and child servers occupy different loopback ports. Six child responses contain only synthetic text and a message indicating successful rendering. The parent accepts messages from the expected child origin and displays them. Control-C closes both temporary servers.

import json
import sys
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Event, Thread
from urllib.request import urlopen

paths = ["open", "csp-block", "xfo-block", "csp-allow", "meta", "frame-src"]
requests = []


class Child(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def do_GET(self):
 name = self.path.lstrip("/")
 assert name in paths
 requests.append(name)
 meta = ('<meta http-equiv="Content-Security-Policy" content="frame-ancestors \'none\'">'
 if name == "meta" else "")
 body = ("<!doctype html><meta charset=utf-8>" + meta + "<p>Rendered: " + name + "</p>"
 "<script>parent.postMessage({frame:" + json.dumps(name) + "}," + json.dumps(app_origin) + ");</script>").encode()
 self.send_response(200)
 self.send_header("Content-Type", "text/html; charset=utf-8")
 self.send_header("Content-Length", str(len(body)))
 self.send_header("Cache-Control", "no-store")
 if name == "csp-block":
 self.send_header("Content-Security-Policy", "frame-ancestors 'none'")
 elif name == "xfo-block":
 self.send_header("X-Frame-Options", "DENY")
 elif name == "csp-allow":
 self.send_header("Content-Security-Policy", "frame-ancestors " + app_origin)
 self.send_header("X-Frame-Options", "DENY")
 elif name == "frame-src":
 self.send_header("Content-Security-Policy", "frame-src 'none'")
 self.end_headers()
 self.wfile.write(body)


class Parent(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def do_GET(self):
 if self.path == "/events":
 body = json.dumps(requests).encode()
 kind = "application/json"
 else:
 body = ("""<!doctype html><meta charset="utf-8"><title>Frame policy lab</title>
<h1>Which child responses can this parent embed?</h1><pre id="result">Waiting for frame messages...</pre>
<button id="check">Show server requests</button><div id="frames"></div><script>
const child=""" + json.dumps(child_origin) + ";const names=" + json.dumps(paths) + """;
window.received=[];
window.snapshot=async()=>{
 const result={rendered:[...window.received].sort(),serverRequests:await (await fetch('/events')).json()};
 document.getElementById('result').textContent=JSON.stringify(result,null,2);return result;
};
addEventListener('message',event=>{
 if(event.origin===child && names.includes(event.data?.frame)) {
 window.received.push(event.data.frame);
 document.getElementById('result').textContent=JSON.stringify(window.received,null,2);
 }
});
document.getElementById('check').onclick=()=>window.snapshot();
for(const name of names) {
 const frame=document.createElement('iframe');frame.src=child+'/'+name;frame.title=name;
 frame.width=240;frame.height=100;document.getElementById('frames').append(frame);
}
</script>""").encode()
 kind = "text/html; charset=utf-8"
 self.send_response(200)
 self.send_header("Content-Type", kind)
 self.send_header("Content-Length", str(len(body)))
 self.end_headers()
 self.wfile.write(body)


child = HTTPServer(("127.0.0.1", 0), Child)
app = HTTPServer(("127.0.0.1", 0), Parent)
child_origin = f"http://127.0.0.1:{child.server_port}"
app_origin = f"http://127.0.0.1:{app.server_port}"
threads = [Thread(target=server.serve_forever, daemon=True) for server in (child, app)]
for thread in threads:
 thread.start()
try:
 for name in paths:
 with urlopen(child_origin + "/" + name, timeout=5) as response:
 assert response.status == 200
 if name == "csp-allow":
 assert response.headers["Content-Security-Policy"] == "frame-ancestors " + app_origin
 assert response.headers["X-Frame-Options"] == "DENY"
 response.read()
 requests.clear()
 print(json.dumps({"app_url": app_origin, "child_origin": child_origin,
 "wire_responses": 6}), flush=True)
 if "--serve" in sys.argv:
 Event().wait()
except KeyboardInterrupt:
 pass
finally:
 for server in (child, app):
 server.shutdown()
 server.server_close()
 for thread in threads:
 thread.join(timeout=5)

The open child renders. csp-block is refused by frame-ancestors none, and xfo-block is refused by X-Frame-Options DENY. The csp-allow child renders because its enforcing ancestor policy explicitly permits the parent, even though it also sends DENY. The meta child renders because that delivery mechanism cannot enforce frame-ancestors, while frame-src renders because its restriction governs frames the child would load, not this parent-child relationship.

Expect four rendering messages and six server requests. Click “Show server requests” to inspect both; the browser check also records two response-blocking failures for the denied frames. The default Python probe separately receives all six HTTP 200 responses, illustrating why header inspection alone cannot replace rendering checks. This experiment tests a single embedding level in one browser, not an account-action workflow or every nested integration. Before shipping, exercise legitimate embedding paths and blocked parents through the actual proxy or CDN, and verify that the intended child responses retain their policy.

In the article’s single-level browser fixture, all six child responses return HTTP 200, but only open, csp-allow, meta, and frame-src render. The child’s enforcing ancestor policy permits the parent in csp-allow and takes precedence over X-Frame-Options DENY. Real nested integrations still need their own checks.

Sources

MDN CSP frame-ancestors, allowed embedding parents, ancestor checks, and unsupported meta delivery, checked 2026-09-13.
MDN CSP frame-src, control over frames loaded by the current document, checked 2026-09-13.
MDN X-Frame-Options, DENY and SAMEORIGIN framing restrictions, checked 2026-09-13.
W3C CSP Level 3, frame-ancestors enforcement and its relationship with X-Frame-Options, checked 2026-09-13.

SSRF makes the server open the door

Sun, 05 Sep 2027 00:00:00 +0000

Server-side request forgery makes a server issue a request selected or influenced by an attacker. A thumbnail service, importer, webhook tester, or document renderer may reach destinations that the requesting user cannot reach directly. The server contributes its network position and whatever credentials that particular fetch path actually attaches. A URL fetch does not automatically inherit every account credential available to the process.

Identify the authority being delegated

Internal services, loopback administration endpoints, and cloud metadata services are important targets when reachable. The consequence depends on authentication, response visibility, allowed methods, and network policy. Blind SSRF can still trigger actions or reveal timing without returning the target’s response. Do not rank severity from the feature’s name alone; identify what it can contact and what that contact can accomplish.

On EC2, instance-role credentials can be available through metadata when a role is attached and the relevant access is permitted. The role-name listing and the role-specific credential document are separate requests. Those temporary credentials represent that role’s permissions, not unlimited AWS access or necessarily the credentials used by every application on the instance.

IMDSv2 requires a token obtained through a PUT request and supplied in subsequent metadata requests. Requiring it blocks a simple unauthenticated GET-only path, but does not solve arbitrary SSRF that can perform the necessary methods and headers. Pair metadata configuration with least-privilege roles and effective network restrictions. Metadata can have IPv4 and, when enabled and supported, IPv6 reachability; review both.

Parse, resolve, and preserve the checked destination

An allowlist should describe the feature’s intended destinations. When possible, accept a resource identifier rather than an arbitrary URL. Otherwise constrain schemes, hosts, and ports with a parser whose interpretation matches the HTTP client. Reject unsupported userinfo and ambiguous syntax rather than approving a URL merely because an allowed name appears somewhere inside it.

For example, trusted.example before an at-sign is userinfo, not proof that the destination host is trusted. Legacy numeric IPv4 forms may be interpreted differently by different URL parsers and resolvers. IPv6 loopback is ::1, not 127.0.0.1. A string check for one spelling is therefore insufficient, and one runtime’s parsing result is not a portable guarantee about another client’s behavior.

Resolve allowed names and evaluate all returned addresses against the intended network policy. For a public-only fetcher, reject loopback, private, link-local, multicast, and other disallowed ranges. Address libraries and registries evolve, so test the versions actually deployed. A destination being globally routable does not itself make it an authorized or benign service.

When it breaks

Checking DNS and then allowing the client to resolve again leaves a time-of-check gap. Pin the validated connection address using a supported transport mechanism while preserving the intended TLS hostname, certificate verification, and HTTP authority. A proxy may resolve independently; validate the actual egress path, retries and fallback connections rather than assuming one local preflight controls them.

Redirects create new destinations. Disable them unless needed, or validate each hop before following it, including resolution and address selection. A previously approved hostname cannot authorize an arbitrary Location target. Egress rules provide a separate containment layer; they should match the workload’s required services and cover relevant address families.

A 200 response does not by itself prove SSRF, and a 4xx response does not prove that no forbidden request occurred. Applications can wrap errors, return cached data, or reject after making contact. Confirm the request against a controlled observation endpoint and distinguish response handling from actual network activity. Never probe real metadata credentials merely to make an example convincing.

Approved data is not proof of actual transport. Match policy to the real fetch path, including resolution, redirects, retries, proxies and TLS identity.

Check it quickly

Save as ssrf_policy.py and run python3 ssrf_policy.py. This deliberately narrow offline policy uses supplied address snapshots instead of DNS. The public address is only an input value; the script never contacts it. All decisions happen in memory, without credentials, sockets, metadata calls, or changes to network controls.

import ipaddress
from urllib.parse import urlsplit

ALLOWED = {'cdn.example.test'}


def approve(url, addresses):
 # Deliberately narrow input language for this offline model.
 if any(ord(c) <= 32 or ord(c) >= 127 for c in url) or '\\' in url:
 raise ValueError('unsupported URL characters')
 parts = urlsplit(url)
 if parts.scheme != 'https' or parts.hostname not in ALLOWED:
 raise ValueError('scheme or host denied')
 if parts.username is not None or parts.password is not None or parts.fragment:
 raise ValueError('userinfo or fragment denied')
 if parts.port not in (None, 443):
 raise ValueError('port denied')
 if not addresses:
 raise ValueError('empty resolution')
 checked = []
 for text in addresses:
 address = ipaddress.ip_address(text)
 if '%' in text or address.is_multicast or not address.is_global:
 raise ValueError('non-public address denied')
 checked.append(str(address))
 return parts.hostname, tuple(checked)


def rejected(label, url, addresses):
 try:
 approve(url, addresses)
 except ValueError:
 print('rejected:', label)
 return
 raise RuntimeError('unexpected acceptance: ' + label)


url = 'https://cdn.example.test/logo.png'
# Address values only. There is no DNS lookup, socket, or HTTP request.
public_snapshot = ['8.8.8.8']
name, pinned = approve(url, public_snapshot)
rejected('userinfo on allowed host', 'https://attacker@cdn.example.test/logo.png', public_snapshot)
rejected('userinfo host confusion', 'https://cdn.example.test@169.254.169.254/', public_snapshot)
rejected('decimal host', 'https://2130706433/', ['127.0.0.1'])
rejected('loopback IPv6', 'https://[::1]/', ['::1'])
rejected('link-local answer', url, ['169.254.169.254'])
rejected('mixed answers', url, ['8.8.8.8', '127.0.0.1'])
rejected('multicast answer', url, ['224.0.0.1'])
rejected('redirect destination', 'https://internal.example.test/', ['10.0.0.1'])
rejected('changed DNS answer', url, ['127.0.0.1'])
if pinned != ('8.8.8.8',) or name != 'cdn.example.test':
 raise RuntimeError('approved snapshot changed')
print('PASS: public fixture accepted; nine denied cases; approved address snapshot retained')

Expect nine rejected cases and PASS. The allowed synthetic hostname with one public address establishes a positive path. Userinfo on an allowed host, userinfo host confusion, numeric and IPv6 destinations outside the allowlist, link-local and mixed answers, multicast, a redirected hostname, and a changed DNS answer fail. The approved tuple remains unchanged. Numeric and IPv6 URLs fail the host allowlist first, so these cases do not independently test IPv6 address classification. No example ran in this source review.

This tests decisions and retained data, not a production fetch implementation. It does not exercise DNS, transport pinning, proxies, TLS, redirect execution, or response limits. Passing it cannot establish that an HTTP library connects to the approved address. Integration tests must verify that wiring, including timeouts and resource limits, using isolated destinations with synthetic responses.

Sources

OWASP: SSRF Prevention Cheat Sheet, intended destinations, address validation, and redirects, checked 2026-10-15.
AWS: Instance Metadata Service, IMDSv2 token requests and endpoint behavior, checked 2026-10-15.
RFC 3986: URI Generic Syntax, authority, userinfo, and host syntax, checked 2026-10-15.
Python: ipaddress, address classification and version changes, checked 2026-10-15.

CSRF asks the browser where the request came from

Sat, 04 Sep 2027 00:00:00 +0000

Browsers can attach session cookies to attacker-triggered requests. CSRF defenses distinguish authorized request contexts; they do not prove that a person saw or intended an action.

This shows the article’s server fixture, not a browser test. SameSite reduces cookie delivery; trusted request context does not prove human intent. XSS and authentication require separate defenses.

Cookies travel without asking

Browsers attach cookies to a request according to domain, path, Secure, SameSite, and other browser rules, not on whether the page causing the request was the user’s earlier intent. That means a page on an attacker’s origin can make the victim’s browser send a state-changing request — a transfer, a password change — that carries the authenticated session cookie, and the server sees a request that looks like it came from the legitimate user. This is why a request’s mere presence of credentials is not evidence of the user’s intent; the browser is following the cookie rules, not the user’s choice.

A token ties the action to a page

The defense is to bind each state-changing request to a value that the attacking page cannot know. The server embeds an unpredictable per-session anti-CSRF token in its own forms and reads it back on submit; the attacker’s cross-site page, lacking the token, fails the check. Pair the token property with strict methods (don’t let GET perform a state change) so tokenized requests are the only way to mutate state. It still must be compared server-side — refusing when missing or mismatched — and never accepted from an attacker-supplied origin.

SameSite adds a browser-side guard

SameSite=Strict withholds the cookie on cross-site requests; explicit Lax also permits it on top-level navigations using safe methods. These restrictions reduce cookie delivery but do not replace the token check in this design. Same-site is broader than same-origin, and browser support and request context matter. The example uses a server-stored synchronizer token, not a double-submit cookie. Cross-site scripting can defeat these defenses by acting inside the trusted origin.

Read Origin as a source signal

Check Origin when present; otherwise a server can inspect the origin portion of Referer. These fields may be absent, and Origin can be null. This fixture deliberately rejects requests without a trusted source signal as well as a valid token. Compare the complete allowed origin, including scheme, host, and port, rather than a hostname prefix. Sec-Fetch-Site adds browser context but reports sites, not an exact origin. Non-browser clients can forge these headers; origin checks address browser request forgery, not user authentication or proof of human intent.

When it breaks

Four failures show up repeatedly. First, a state change behind GET: an <img> or link can trigger it, and SameSite=Lax still sends cookies on top-level GET navigation, so the forged navigation arrives authenticated. Second, intentional cross-site cookies: SameSite=None; Secure permits that delivery, so another CSRF defense must protect mutations. Modern browsers reject None without Secure; legacy behavior is not a safe fallback. Third, a token check that accepts absence: the handler verifies the value only when present, skips verification for API paths, or reads the token from the query string where it leaks through Referer and logs. Fourth, subdomain trust: Lax and Strict compare sites not origins, so evil-sub.example.com shares the site, and a subdomain able to plant parent-domain cookies can attack naive double-submit schemes or session handling. Prefer host-scoped session cookies and reject ambiguous duplicates. Credentials alone never establish request legitimacy.

Check it quickly

Save the following as csrf_probe.py and run python3 csrf_probe.py. It models the server gate with only the standard library: the method must be POST, the token must match the session value, and the origin signal must agree. No network or browser is involved, so its cases run without a browser. The fixed token is test data; a real application must generate an unpredictable token bound to the session.

from hmac import compare_digest
from urllib.parse import urlsplit
import sys

EXPECTED_ORIGIN = "https://bank.example"
SESSION_TOKEN = "token-issued-with-own-form"

def decide(method, origin, referer, token):
 """Mimic a CSRF gate: POST-only mutation plus token plus origin must agree."""
 if method != "POST":
 return 405, "state change requires POST"
 if not isinstance(token, str) or not token.isascii() or not compare_digest(token, SESSION_TOKEN):
 return 403, "missing or wrong token"
 if origin:
 if origin != EXPECTED_ORIGIN:
 return 403, "origin mismatch"
 elif referer:
 base = urlsplit(EXPECTED_ORIGIN)
 try:
 seen = urlsplit(referer)
 except ValueError:
 return 403, "referer mismatch"
 if seen.scheme != base.scheme or seen.netloc != base.netloc:
 return 403, "referer mismatch"
 else:
 return 403, "no origin signal"
 return 200, "transfer accepted"

def check(actual, expected, label):
 assert actual == expected, f"{label}: got {actual}, want {expected}"
 print(f"{label}: {actual[0]} {actual[1]}")

check(decide("POST", EXPECTED_ORIGIN, "", SESSION_TOKEN),
 (200, "transfer accepted"), "legitimate form post")
check(decide("POST", "https://evil.example", "", SESSION_TOKEN),
 (403, "origin mismatch"), "forged cross-site post")
check(decide("POST", EXPECTED_ORIGIN, "", ""),
 (403, "missing or wrong token"), "missing token")
check(decide("POST", "", "https://bank.example/transfer", SESSION_TOKEN),
 (200, "transfer accepted"), "referer fallback navigation")
check(decide("GET", EXPECTED_ORIGIN, "", SESSION_TOKEN),
 (405, "state change requires POST"), "state change over GET")

check(decide("POST", "", "", SESSION_TOKEN),
 (403, "no origin signal"), "missing origin signal")
check(decide("POST", "https://evil.bank.example", "", SESSION_TOKEN),
 (403, "origin mismatch"), "same-site subdomain")
check(decide("POST", EXPECTED_ORIGIN, "", "令牌"),
 (403, "missing or wrong token"), "non-ASCII token")
check(decide("POST", "", "https://[", SESSION_TOKEN),
 (403, "referer mismatch"), "malformed referer")

print("PASS: legitimate, forged, and missing-token cases behave differently")
print(sys.version)

The legitimate POST with the correct origin and token returns 200, the cross-site origin returns 403 even with a valid token, and the missing token returns 403 even with a correct origin. The Referer fallback accepts a same-origin source and the GET case returns 405, which shows why safe methods matter. Expected output ends with PASS: legitimate, forged, and missing-token cases behave differently plus the Python version.

Additional cases reject missing source headers, a same-site subdomain, a non-ASCII token, and a malformed fallback URL. This checks decision logic, not browser cookie handling, TLS, or session generation. Its literal comparisons intentionally omit URL normalization; equivalent explicit default ports can be rejected. Re-test actual routes with synthetic accounts in an isolated browser setup.

Sources

MDN: Cross-site request forgery (CSRF), attack shape, token and SameSite defenses, checked 2026-10-19.
OWASP: Cross-Site Request Forgery Prevention Cheat Sheet, token and SameSite guidance, checked 2026-10-19.
RFC 6265: HTTP State Management Mechanism, baseline cookie scoping; this RFC does not define SameSite, checked 2026-10-19.

Enable gzip explicitly, then verify negotiation and decoded bytes

Sun, 29 Aug 2027 00:00:00 +0000

gzip is a response content coding: nginx can send a compressed representation that an accepting client decodes. Enabling it requires gzip on; gzip_types extends the eligible MIME types but does not activate compression by itself. The result depends on the response, request capabilities, and effective configuration. A smaller body is useful only if it still decodes to the intended content.

When it breaks

A MIME list beneath gzip off changes no response into gzip. Another common omission is gzip_vary on. nginx does not unconditionally add Vary: Accept-Encoding merely because compression occurred; the directive controls that behavior. A shared cache selecting representations needs the relevant request-header dimensions, otherwise clients with different coding capabilities can receive an inappropriate stored response.

The minimum-length setting has a narrower contract than its name suggests. gzip_min_length determines length from the response’s Content-Length field. It does not necessarily know the final size of a response whose length is unspecified. A tiny dynamically streamed response can therefore be compressed even when an equally small static file falls below the configured threshold.

Measurements can also be misleading. The size of a decompressed output file is not the compressed transfer-body size, and compressed body bytes do not include all HTTP framing, headers, or TLS traffic. File extensions such as .gz prove nothing by themselves. Check the actual content coding and decode the bytes before interpreting any apparent bandwidth saving.

Decode back to the same bytes. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

The integration results below are historical observations; this revision did not rerun the fixture.

Save this as check-gzip.py and run it with Python, Docker, and the pinned images. The recorded setup used nginx 1.28.0 on Linux arm64 and a small Python backend. nginx enables gzip at level three, allows JSON in addition to its built-in HTML eligibility, sets a 1,024-byte floor, and explicitly enables Vary:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
 target.write_bytes(content if isinstance(content,bytes) else content.encode())
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 request.port=port
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
import gzip,hashlib
backend=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
class Handler(BaseHTTPRequestHandler):
 protocol_version='HTTP/1.1'
 def log_message(self,*args):pass
 def do_GET(self):
 self.send_response(200);self.send_header('Content-Type','application/json')
 self.send_header('Connection','close');self.end_headers();self.wfile.write(b'{"ok":true}\n');self.close_connection=True
ThreadingHTTPServer(('0.0.0.0',8081),Handler).serve_forever()
'''
payload=json.dumps({'message':'compressible nginx response '*1000}).encode()
locations='''
location = /disabled.json { gzip off; gzip_types application/json; }
location = /novary.json { gzip_vary off; }
location = /stream { proxy_pass http://backend:8081; }
'''
extra='''
types { application/json json; text/html html; application/octet-stream bin; }
gzip on; gzip_types application/json; gzip_vary on; gzip_min_length 1024; gzip_comp_level 3;
'''
files={name:payload for name in ['large.json','other.bin','page.html','disabled.json','novary.json']};files['tiny.json']=b'{"ok":true}\n'
with lab(config(locations,extra),files,backend) as (_,request):
 results={}
 for encoding in ['gzip','identity','gzip;q=0','br']:
 status,h,b=request('/large.json',headers={'Accept-Encoding':encoding});assert status==200
 if encoding=='gzip':assert h['content-encoding']=='gzip' and gzip.decompress(b)==payload and len(b)<len(payload)
 else:assert 'content-encoding' not in h and b==payload
 assert h['vary']=='Accept-Encoding'
 results[encoding]={'encoding':h.get('content-encoding','identity'),'body_bytes':len(b)}
 for path,encoded in [('/tiny.json',False),('/other.bin',False),('/page.html',True),('/disabled.json',False),('/novary.json',True),('/stream',True)]:
 status,h,b=request(path,headers={'Accept-Encoding':'gzip'});assert status==200
 assert (h.get('content-encoding')=='gzip')==encoded,(path,h)
 decoded=gzip.decompress(b) if encoded else b
 assert decoded==(files['tiny.json'] if path=='/stream' else files[path[1:]])
 if path=='/novary.json':assert 'vary' not in h
 results[path]={'encoding':h.get('content-encoding','identity'),'body_bytes':len(b),'vary':h.get('vary')}
 results['decoded_sha256']=hashlib.sha256(payload).hexdigest()
 print(json.dumps(results,sort_keys=True))

The large synthetic JSON representation is 28,015 bytes. With Accept-Encoding: gzip, its transferred response body is 246 bytes in the recorded run, and Python’s gzip decoder reproduces the original bytes exactly. This intentionally repetitive payload demonstrates operation and integrity; its compression ratio is not a prediction for production JSON, JavaScript, or arbitrary text.

Requests accepting identity, setting gzip;q=0, or advertising only br receive the original bytes without a Content-Encoding field. Zero quality disallows gzip. Advertising Brotli does not enable a Brotli compressor in this nginx configuration, and identity remains acceptable in that request. The eligible large responses carry Vary: Accept-Encoding, including the uncompressed variants.

The same synthetic payload served as an ineligible binary type remains uncompressed, while the HTML type is eligible without appearing in gzip_types. A location explicitly setting gzip off also remains uncompressed despite listing JSON. These controls separate type eligibility from the enable switch, rather than treating every absent coding as evidence that negotiation failed.

The 12-byte static JSON response stays uncompressed because its known length is below the floor. The backend returns the same 12 bytes without Content-Length, ending the response by closing its connection. nginx compresses that response into 32 bytes. The floor therefore does not guarantee that all tiny responses avoid compression, and this particular tiny payload actually grows.

A final location keeps gzip active but turns gzip_vary off. Its body still arrives gzip encoded, with no Vary field. This makes the missing-cache-dimension failure observable without requiring a real shared cache to misdeliver content. The fixture checks the header policy; a deployed cache’s representation selection should also be tested through that cache.

Tune the policy using representative responses

Select content types whose actual bytes compress well. Already compressed media or archives usually offer little benefit from another coding layer, while repetitive text may shrink substantially. Do not infer CPU cost from this byte-count experiment. Compare representative sizes, compression settings, server utilization, and client latency before choosing a level for a busy service.

Keep the response metadata honest through proxies. Content-Encoding tells the recipient how to decode the representation, while Vary tells caches which request fields affect selection. If an application already sends an encoded response, review that path separately rather than adding a misleading header to unencoded bytes. The decisive check remains negotiated headers plus a successful byte-for-byte decode.

Sources

nginx gzip module, enabling gzip, MIME eligibility, Content-Length thresholds, and Vary, checked 2026-10-15.
RFC 9110 HTTP Semantics, Accept-Encoding, Content-Encoding, and representation selection, checked 2026-10-15.

A request limit protects the budget named by its key

Sat, 28 Aug 2027 00:00:00 +0000

A request-rate limit applies to the identity expressed by its key, not automatically to a person, account, or endpoint. limit_req_zone stores shared state for those key values, and limit_req applies the zone while handling a request. Two locations using the same zone and key can consume the same budget. Choosing a variable therefore defines a real fairness and enforcement boundary.

The source describes nginx 1.28.0, two workers and rate=1r/m. Outcomes are historical observations, not new measurements. The shown limit_req configuration has no burst or delay experiment; rejection and dry-run allowance are separate outcomes. Current audit: revised, execution blocked. No rerun, page verification or publication credit.

When it breaks

A client-controlled identifier can turn one apparent limit into many independent allowances. If changing a query value changes the key, an untrusted client can select another budget. An empty key is more direct: nginx does not account for requests whose key evaluates to an empty value. A missing identity variable can therefore silently leave requests outside the intended limit.

An IP key measures an address, which may represent many users behind NAT or an intermediary. Forwarded address headers need an explicit trust policy before they become identity input. Trusting arbitrary X-Forwarded-For lets callers manufacture identities; ignoring a real trusted proxy’s address chain may instead put every user under the proxy’s own budget. Neither outcome is fixed by increasing the rate.

Shared memory scope matters too. A zone is shared among workers of the nginx instance; that does not create a fleet-wide counter across unrelated processes or machines. Compatible graceful reloads can retain zone state. Assuming that every configuration reload resets all allowances can make a test or recovery procedure misread continued rejections as a broken deployment.

Check it quickly

The reported outcomes lack a retained runtime artifact. This revision did not rerun the fixture.

Save the script as check-limit-keys.py. It requires Python, a disposable local Docker daemon, build/run permission and platform-compatible pinned images. Docker context/environment overrides must not select a remote daemon: requests target host loopback. Build --network none restricts build-step networking, not registry resolution. Cleanup ignores return codes; verify removal separately. The original report uses nginx 1.28.0 with two workers and a disposable HTTP backend. At one request per minute, dependent observations must finish before the allowance recovers; this is elapsed-time accounting, not a fixed minute window. Distinct fixture keys isolate cases:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None,extra_ports=()):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
 target.write_bytes(content if isinstance(content,bytes) else content.encode())
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 published=[arg for value in (8080,*extra_ports) for arg in ('-p','127.0.0.1::'+str(value))]
 docker('run','-d','--name',containers[0],'--network',name,*published,tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 request.port=port;request.network=name
 request.ports={int(key.split('/')[0]):int(value[0]['HostPort']) for key,value in details['NetworkSettings']['Ports'].items() if value}
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
backend=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json
class Handler(BaseHTTPRequestHandler):
 protocol_version='HTTP/1.1'
 def log_message(self,*args):pass
 def do_GET(self):
 body=json.dumps({'path':self.path,'method':self.command}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
ThreadingHTTPServer(('0.0.0.0',8081),Handler).serve_forever()
'''
from concurrent.futures import ThreadPoolExecutor
extra='''
limit_req_zone $arg_client zone=identity:1m rate=1r/m;
limit_req_zone $binary_remote_addr zone=address:1m rate=1r/m;
limit_req_status 429;
add_header X-Limit $limit_req_status always; add_header X-Worker $pid always;
'''
locations='''
location /a { limit_req zone=identity; proxy_pass http://backend:8081; }
location /b { limit_req zone=identity; proxy_pass http://backend:8081; }
location /dry { limit_req zone=identity; limit_req_dry_run on; proxy_pass http://backend:8081; }
location /ip { limit_req zone=address; proxy_pass http://backend:8081; }
location /open { proxy_pass http://backend:8081; }
'''
configuration=config(locations,extra).replace('worker_processes 1;','worker_processes 2;')
with lab(configuration,backend=backend) as (proxy,request):
 results={}
 assert request('/a?client=one')[0]==200 and request('/b?client=one')[0]==429
 assert request('/a?client=two')[0]==200
 results['same_zone_cross_location']=[200,429];results['changed_untrusted_key']=200
 empty=[request('/a') for _ in range(2)];assert all(r[0]==200 for r in empty)
 assert all(r[1].get('x-limit') is None for r in empty)
 results['empty_keys']=[{'status':r[0],'limit':r[1].get('x-limit')} for r in empty]
 first=request('/ip',headers={'X-Forwarded-For':'192.0.2.1'})
 second=request('/ip',headers={'X-Forwarded-For':'192.0.2.2'})
 assert first[0]==200 and second[0]==429;results['forged_forwarding_header']=[200,429]
 first=request('/dry?client=dry');second=request('/dry?client=dry')
 assert first[0]==second[0]==200 and second[1]['x-limit']=='REJECTED_DRY_RUN'
 results['dry_run']={'status':second[0],'limit':second[1]['x-limit']}
 first=request('/a?client=shared');assert first[0]==200;workers={first[1]['x-worker']}
 deadline=time.monotonic()+8
 while time.monotonic()<deadline and len(workers)<2:
 with ThreadPoolExecutor(max_workers=8) as pool:
 replies=list(pool.map(lambda _:request('/b?client=shared'),range(8)))
 assert all(r[0]==429 for r in replies);workers.update(r[1]['x-worker'] for r in replies)
 assert len(workers)==2,workers
 results['shared_budget_worker_ids']=sorted(workers)
 docker('exec',proxy,'nginx','-s','reload')
 wait(lambda:request('/open')[1]['x-worker'] not in workers)
 reply=request('/a?client=shared');assert reply[0]==429 and reply[1]['x-worker'] not in workers
 results['graceful_reload']={'status':reply[0],'worker':reply[1]['x-worker']}
 print(json.dumps(results,sort_keys=True))

/a?client=one succeeds, while /b?client=one returns 429 because both locations use the same zone. Changing the query identity to two gives a new successful request. This is useful as a controlled demonstration, but it is precisely why an unverified query parameter is unsuitable as the sole production identity for an abuse-resistant limit.

Two requests without the client argument both succeed and have no X-Limit decision value. The empty key was not accounted for. This control is important when a deployment derives identity from an optional header or an authentication result: normal traffic with a populated variable can look correctly limited while missing-variable traffic bypasses the intended accounting.

The IP-based location receives two requests with different fabricated X-Forwarded-For values. The first succeeds and the second returns 429. No real-IP trust configuration is installed, so the socket peer still supplies the address key. This verifies that merely sending a different forwarding header does not alter the configured identity in this setup; it does not authenticate any original internet user.

The dry-run location returns 200 on a request that would have been rejected, with REJECTED_DRY_RUN in the decision header. Accounting still occurs, but enforcement is disabled there. A successful response alone would hide this distinction. Use the dry-run decision variable when assessing a proposed policy instead of counting only client-visible rejections.

For a fresh shared key, the first request succeeds and subsequent requests return 429 from both observed worker process IDs. nginx then performs a graceful reload with the same zone configuration. A new worker serves the still-rejected key, demonstrating retained budget state rather than a reset. The test waits for a new worker identity before attributing the result to the reload.

Define identity, scope, and observation together

Decide whether the intended budget belongs to an address, authenticated account, route, or combination. Verify that its inputs exist and are trustworthy at the limiting phase. Sharing a zone across locations can intentionally create one service-wide budget, while separate zones create distinct accounting. Avoid assuming a location name itself becomes part of a key that never includes it.

Size zone memory for key cardinality as well as request volume. The binary-address documentation gives 128-byte state on 64-bit platforms and 64-byte state on 32-bit platforms, not an exact capacity guarantee for arbitrary variable-length keys. Storage exhaustion can evict old state and, if allocation still fails, reject requests. This experiment checks enforcement semantics, not a measured maximum key capacity.

Finally, inspect effective configuration and decision telemetry after changes. Distinguish empty-key bypass, dry-run allowance, actual rejection, and preserved reload state. A working per-instance limiter does not prove a global tenant quota, and a request returning 429 does not by itself identify which intended user or route budget was consumed.

Sources

nginx request limit module, shared key state, empty keys, dry run, memory sizing, and decision variables, checked 2026-10-15.
nginx real IP module, trusted proxy boundaries when deriving a client address, checked 2026-10-15.
RFC 6585 additional HTTP status codes, 429 and identification of the rate-limited subject, checked 2026-10-15.
nginx 1.28.0 request-limit source, shared-zone state reuse across compatible reloads, checked 2026-10-15.

A reverse proxy needs an explicit header trust boundary

Fri, 27 Aug 2027 00:00:00 +0000

A reverse proxy accepts a client connection and makes a separate request to an upstream application. The application’s socket peer is therefore the proxy, while information about an earlier client can arrive through request headers. Those headers are useful only when the deployment defines who may set them and how they are validated. Proxying a request does not automatically make every forwarded identity or scheme claim trustworthy.

When it breaks

A client can supply X-Forwarded-For before reaching the proxy. Ordinary forwarding may preserve that value, and $proxy_add_x_forwarded_for appends the current peer address rather than removing an untrusted prefix. An application that blindly trusts the first address can therefore attribute the request to a client-selected value. The correct interpretation depends on the actual trusted proxy chain.

Host handling is another independent decision. nginx’s default upstream Host is based on the proxy destination, not automatically the public Host the client used. An application building absolute redirects or selecting virtual hosts can consequently behave differently behind the proxy. Forward only the intended host context and apply the application’s accepted-host policy; copying a header does not validate the name it contains.

The proxy also does not guarantee that backend details remain undisclosed. Response headers, redirects, bodies, diagnostics, or other public configuration can reveal information about the origin. Restricting direct reachability and controlling selected response fields are useful measures, but they are different from claiming that one proxy_pass directive hides all upstream identity or provides every required transport protection.

The direct-client fixture distinguishes socket observations from client text. Replacing selected fields does not validate all proxy chains, Host values or transport security.

Check it quickly

Run the script with host Python, Docker, and the pinned nginx and Python images. The recorded environment used nginx 1.28.0 and Python 3.11.16 on Linux arm64 with Docker 29.3.1. It creates a private Docker network and a real echo backend, publishes only the proxy on host loopback, validates configuration, and attempts to clean up its own resources. No production endpoint or credential is involved:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True);target.write_text(content)
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'

backend=r'''from http.server import BaseHTTPRequestHandler,HTTPServer
import json
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=self.rfile.read(int(self.headers.get('Content-Length','0')))
 document={'path':self.path,'method':self.command,'body':body.decode(),'headers':{key:self.headers.get(key) for key in ['Host','X-Forwarded-For','X-Real-IP','X-Forwarded-Proto','X-Forwarded-Host']}}
 data=json.dumps(document).encode()
 self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.send_header('X-Backend-Info','fixture-node');self.end_headers();self.wfile.write(data)
 do_POST=do_GET
 def log_message(self,*args):pass
HTTPServer(('0.0.0.0',8081),Handler).serve_forever()
'''

locations=r'''
location /default { proxy_pass http://backend:8081; }
location /append { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://backend:8081; }
location /edge {
 proxy_set_header Host $host;
 proxy_set_header X-Forwarded-For $remote_addr;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_set_header X-Forwarded-Host $host;
 proxy_hide_header X-Backend-Info;
 proxy_pass http://backend:8081;
}
'''
with lab(config(locations,'server_tokens off;'),backend=backend) as (container,request):
 forged={'Host':'public.example','X-Forwarded-For':'198.51.100.77','X-Forwarded-Proto':'forged-scheme','X-Real-IP':'198.51.100.88'}
 status,headers,body=request('/default',headers=forged);plain=json.loads(body)
 assert status==200 and plain['headers']['Host']=='backend:8081'
 assert plain['headers']['X-Forwarded-For']=='198.51.100.77' and plain['headers']['X-Forwarded-Proto']=='forged-scheme'
 assert headers['x-backend-info']=='fixture-node'
 status,headers,body=request('/append',headers=forged);appended=json.loads(body)['headers']['X-Forwarded-For'].split(', ')
 assert status==200 and len(appended)==2 and appended[0]=='198.51.100.77'
 status,headers,body=request('/edge',headers=forged);edge=json.loads(body)['headers']
 assert status==200 and edge['Host']==edge['X-Forwarded-Host']=='public.example'
 assert edge['X-Forwarded-For']==edge['X-Real-IP']==appended[-1]
 assert edge['X-Forwarded-For'] not in ['198.51.100.77','198.51.100.88'] and edge['X-Forwarded-Proto']=='http'
 assert 'x-backend-info' not in headers and headers['server']=='nginx'
 print('ordinary forwarding preserves forged forwarding headers and defaults Host to the upstream authority')
 print('append retains the untrusted prefix; the direct-client edge configuration replaces it with the actual socket peer and scheme')
 print('explicit response-header suppression removes one backend field, while server_tokens off still exposes the nginx product name')

Every test request supplies public.example as Host and deliberately forged forwarding values using documentation-only IP addresses. The default route sends Host as backend:8081 and preserves the forged X-Forwarded-For and X-Forwarded-Proto. This directly demonstrates that normal proxy forwarding is not an identity-sanitization step and does not preserve the public host automatically.

The append route uses $proxy_add_x_forwarded_for. Its backend receives two comma-separated addresses: the client-supplied address first, followed by nginx’s observed socket peer. The original untrusted prefix remains. Appending useful connection context does not make the entire resulting list equally authoritative, and trusting its leftmost entry would defeat the intended boundary in this direct-client setup.

The edge route replaces X-Forwarded-For and X-Real-IP with $remote_addr, sets the forwarded scheme from $scheme, and forwards the chosen host context explicitly. The backend receives the actual observed peer value rather than either forged address, and the scheme is http in this deliberately plain-HTTP fixture. The test compares the values across requests without assuming a particular Docker bridge address.

That socket peer can reflect Docker’s networking path; it is not a claim to reconstruct an arbitrary internet user’s address through every NAT or earlier proxy. If the edge itself sits behind a trusted load balancer, the deployment needs a deliberate policy for accepting that hop’s information. The example tests a direct-client trust boundary, not every possible proxy chain.

The backend also emits X-Backend-Info: fixture-node. The default route exposes it, while the edge route’s explicit proxy_hide_header removes it. With server_tokens disabled, the response still says Server: nginx. This separates suppression of one backend field from hiding the proxy product name or proving that no other response content discloses implementation details.

Connect network access and header interpretation

Limit which callers can reach the backend directly when it relies on proxy-established context. Configure trusted proxy addresses narrowly if using the real IP module, and ensure the resulting address interpretation matches the application’s own trust settings. A broad trust range can allow an unintended caller to supply apparently authoritative forwarding information.

Decide where TLS terminates and whether the upstream connection also needs encryption and certificate verification. The scheme nginx observes on one hop does not automatically describe an earlier hop. Test redirects, secure-cookie behavior, and generated URLs through the actual deployment path when they depend on forwarded scheme or host.

Finally, inspect the request received by the application and the response received by the client. Header configuration, network isolation, and disclosure controls each answer a different question. Evidence from both sides makes their combined behavior reviewable without assuming that the proxy silently supplies every missing guarantee.

Sources

nginx HTTP proxy module, default Host handling, forwarded headers, and response-header suppression, checked 2026-10-15.
nginx real IP module, explicitly trusted address sources and proxy chains, checked 2026-10-15.
nginx core server_tokens and variables, socket-peer address, request scheme, and server-version disclosure, checked 2026-10-15.

nginx matches normalized paths and can select a location again

Thu, 26 Aug 2027 00:00:00 +0000

nginx location matching uses a normalized URI path rather than the untouched request-target text. Percent-encoded characters are decoded, relative path components are resolved, and adjacent slashes can be merged according to configuration. Later processing can change the current URI again and trigger another location search. Understanding both transformations is necessary when a request appears to enter a different handler than its original spelling suggests.

When it breaks

A path filter that reasons only about the raw request text can disagree with nginx’s routing view. An encoded slash can become a path separator, and a dot segment can remove an earlier component. A backend might apply its own decoding rules as well. Security and routing decisions need an agreed interpretation at each boundary, rather than assuming that every component sees an identical path string.

Logging only $uri can also obscure the incoming request. That variable represents the current normalized URI and may change during request processing. $request_uri retains the full original URI, including its original query. Neither is a universal replacement for the other: one explains current routing, while the other helps reconstruct what the client actually requested.

Index handling and try_files introduce another transition. A request for a directory can internally become a request for its index file. A failed file lookup can redirect internally to a fallback URI, causing a different location and different query arguments to apply. The first prefix match therefore does not necessarily identify the configuration that produces the final response.

The current path and arguments may change during internal reselection while request_uri preserves the incoming URI. Assertions remain unrun in this graphics review.

Check it quickly

Run the script with host Python, Docker, and the pinned nginx image. The article reports nginx 1.28.0 on Linux arm64 and Docker 29.3.1. It builds and syntax-checks a temporary configuration, publishes only a host-loopback port, sends request targets without client-side path normalization, and attempts to remove its own image, containers, and network:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True);target.write_text(content)
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'

locations=r'''
location / { return 200 "$uri|$request_uri|$args"; }
location /protected/ { return 403 "blocked"; }
location /docs/ { try_files $uri $uri/ /fallback?reason=missing; }
location = /docs/index.html { return 200 "$uri|$request_uri|$args"; }
location = /fallback { return 200 "$uri|$request_uri|$args"; }
'''
with lab(config(locations),{'docs/index.html':'stored-index'}) as (container,request):
 cases={
 '/a//b/../c?x=1':'/a/c|/a//b/../c?x=1|x=1',
 '/a%2Fb/%2e%2e/c?x=1':'/a/c|/a%2Fb/%2e%2e/c?x=1|x=1',
 '/Protected/item':'/Protected/item|/Protected/item|',
 '/docs/':'/docs/index.html|/docs/|',
 '/docs/missing?x=1':'/fallback|/docs/missing?x=1|reason=missing'}
 for path,expected in cases.items():
 status,headers,body=request(path);assert status==200 and body.decode()==expected,(path,status,body)
 for path in ['/protected/item','/protected%2Fitem','/a/../protected/item']:assert request(path)[0]==403,path
 assert request('/bad%ZZ')[0]==400
 print('matching uses normalized paths: percent-decoding, dot segments, and merged slashes differ from the original request URI')
 print('index and try_files internal redirects choose another location; request_uri remains original while uri and args can change')

The request /a//b/../c?x=1 produces current URI /a/c, while the original request URI remains /a//b/../c?x=1 and the arguments remain x=1. This demonstrates slash merging and dot-segment resolution in the tested default configuration. The response includes all three fields so their meanings can be compared directly.

The encoded request /a%2Fb/%2e%2e/c?x=1 reaches the same current path /a/c. Its original encoded spelling is still preserved in request_uri. This case shows that percent-decoding participates in normalization before routing. The test does not infer behavior from a browser address bar, which may already have rewritten a path before sending it.

The protected prefix returns 403 for /protected/item, /protected%2Fitem, and /a/../protected/item. All normalize into the protected path family. In this Linux environment, /Protected/item instead reaches the ordinary fallback because the case differs. An invalid percent escape such as %ZZ is rejected with HTTP 400 rather than treated as a successful route to arbitrary content.

The fixture includes a real /srv/docs/index.html. Requesting /docs/ reaches the exact /docs/index.html location after index processing, which returns a diagnostic marker instead of the stored file’s body. The original request_uri remains /docs/. This makes the internal location change visible even though the client receives no redirect response.

For /docs/missing?x=1, try_files cannot find the requested file or directory and internally selects /fallback?reason=missing. The response reports current URI /fallback, original request URI /docs/missing?x=1, and current arguments reason=missing. The fallback’s explicit query replaces the earlier arguments in this example; preserving arguments would require an intentional configuration choice.

Verify the path contract across the full request chain

Test representative encoded forms, repeated slashes, dot segments, missing files, and directory requests when changing routing. Include the actual client, proxy, and application in integration tests if they normalize differently. The small fixture establishes nginx behavior for its chosen configuration; it does not prove every upstream framework or security filter interprets the same bytes identically.

Use original and current URI fields deliberately in diagnostics, while recognizing that query strings can contain sensitive data. Log only the request information the operational task needs. An internal redirect can change the final handler and logging context, so verify where the event is ultimately recorded as well as which initial location matched.

Keep fallback behavior explicit. A missing asset should not accidentally become a successful application page if that masks deployment errors, and protected routes should not escape into an unrelated fallback after rewriting. Syntax validation catches malformed configuration; requests with asserted statuses, fields, and bodies establish the behavior users actually receive.

Sources

nginx core location, try_files, and variables, URI normalization, internal redirects, and original versus current URI, checked 2026-10-15.
How nginx processes a request, index processing and subsequent location selection, checked 2026-10-15.
nginx 1.28.0 core implementation, the tested version’s location search and internal-redirection behavior, checked 2026-10-15.

A 404 page answers honestly

Sat, 03 Jul 2027 00:00:00 +0000

A 404 response has two audiences. Software needs an accurate status; a person needs a useful next step. A beautifully designed error page cannot compensate for a false 200, and a correct status does not excuse an inaccessible page. Keep both parts of the response truthful without turning a missing address into a business outage.

Choose the status before rendering

RFC 9110 says 404 means the origin found no current representation, or is unwilling to disclose that one exists. It does not establish whether the absence is permanent. Use 410 when removal is known to be likely permanent. For a documented move, redirect to the corresponding replacement; do not redirect every unknown URL to the homepage. That hides broken links and sends visitors somewhere unrelated.

A dependency outage is different from a missing resource. If a database failure prevents determining whether an object exists, preserve the appropriate server-error response instead of converting every exception into 404. Conversely, an authorization policy may deliberately conceal existence with 404. Document that decision without revealing identifiers, account membership, or permission details in the error body.

Google can classify a successful HTTP response containing an error message as a soft 404. A 200 does not guarantee indexing, and a 404 does not guarantee immediate disappearance from search results. The operational requirement is an honest status and useful content, not a promised crawler timetable.

Make the body independent and safe

Render a short explanation, a real home or index link, and enough context to recover. A static site need not add a search form or external widget just for this page. Keep its assets local and its operation independent of the database that may be failing. An API should return its documented machine-readable error format, not a successful HTML document.

Set the status before writing the body. In Go, the first body write implicitly selects 200 unless WriteHeader has already selected another status. HEAD should return the same status and applicable representation metadata as GET without transferring its content. Testing only a directly opened error-template file does not exercise the missing-resource route that invokes it.

When it breaks

A catch-all rewrite can serve the homepage for missing articles, JavaScript, or CSS files. A reverse proxy can replace the origin’s 404 with its own 200 error page. A custom handler can commit a successful header before discovering a missing file. Probe an invented path through the actual entry point, inspect the first response without following redirects, and separately check a known-good path.

Negative responses can also be cached: 404 is heuristically cacheable unless other rules or explicit controls prevent it. A long negative lifetime may hide a newly created or restored page. Choose the policy deliberately; no-store prevents new storage but is not a purge of an already stored response. Avoid echoing untrusted query strings into HTML or copying sensitive URLs into logs. Count normalized missing routes without creating unbounded metric labels.

One response, two audiences. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

Save the following as not_found_probe.go in a scratch directory and run go run not_found_probe.go. It starts one loopback HTTP/1.1 server, follows no redirects, and modifies only an in-memory restoration flag. The deliberate /soft-miss bug is a negative control, not a recommended route.

package main

import (
 "fmt"
 "io"
 "net/http"
 "net/http/httptest"
 "runtime"
 "strconv"
 "strings"
 "sync/atomic"
 "time"
)

func must(err error) {
 if err != nil {
 panic(err)
 }
}

func check(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func main() {
 const missing = `<!doctype html><title>Not found</title><h1>Page not found</h1><a href="/">Home</a>`
 var restored atomic.Bool
 server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 status, body, media := 404, missing, "text/html; charset=utf-8"
 switch r.URL.Path {
 case "/":
 status, body = 200, "<!doctype html><title>Home</title><h1>Home</h1>"
 case "/gone":
 status = 410
 case "/moved":
 status = 308
 w.Header().Set("Location", "/")
 case "/offline":
 status, body = 503, "Service temporarily unavailable"
 case "/soft-miss":
 status = 200 // Deliberate bug: the body alone cannot identify success.
 case "/restored":
 if restored.Load() {
 status, body = 200, "Restored resource"
 }
 default:
 if strings.HasPrefix(r.URL.Path, "/api/") {
 media, body = "application/json", `{"error":"not_found"}`
 }
 }
 if r.Method != "GET" && r.Method != "HEAD" {
 status, body = 405, "Method not allowed"
 w.Header().Set("Allow", "GET, HEAD")
 if media == "application/json" {
 body = `{"error":"method_not_allowed"}`
 }
 }
 w.Header().Set("Content-Type", media)
 w.Header().Set("Cache-Control", "no-store")
 w.Header().Set("Content-Length", strconv.Itoa(len(body)))
 w.WriteHeader(status) // Status must be committed before writing the body.
 if r.Method != "HEAD" {
 _, err := io.WriteString(w, body)
 must(err)
 }
 }))
 defer server.Close()
 transport := &http.Transport{}
 defer transport.CloseIdleConnections()
 client := &http.Client{Transport: transport, Timeout: 2 * time.Second,
 CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
 }
 probe := func(method, route string, status int, media string) (*http.Response, string) {
 request, err := http.NewRequest(method, server.URL+route, nil)
 must(err)
 response, err := client.Do(request)
 must(err)
 body, readError := io.ReadAll(response.Body)
 closeError := response.Body.Close()
 must(readError)
 must(closeError)
 check(response.StatusCode == status && response.Proto == "HTTP/1.1", "wrong status or protocol")
 check(response.Header.Get("Content-Type") == media, "wrong media type")
 check(response.Header.Get("Cache-Control") == "no-store", "fixture must not reuse negative responses")
 check(!strings.Contains(string(body), "do-not-echo"), "request query leaked into error page")
 fmt.Printf("%s %s: %d, body=%d bytes\n", method, request.URL.Path, status, len(body))
 return response, string(body)
 }
 const html = "text/html; charset=utf-8"
 probe("GET", "/", 200, html)
 get, body := probe("GET", "/missing?hint=do-not-echo", 404, html)
 check(body == missing && strings.Contains(body, `href="/"`), "missing page must offer a real home link")
 head, empty := probe("HEAD", "/missing", 404, html)
 check(empty == "" && head.ContentLength == get.ContentLength, "HEAD must preserve status and length without a body")
 probe("GET", "/assets/missing.css", 404, html)
 _, api := probe("GET", "/api/missing", 404, "application/json")
 check(api == `{"error":"not_found"}`, "API must not receive an HTML success page")
 probe("GET", "/gone", 410, html)
 move, _ := probe("GET", "/moved", 308, html)
 check(move.Header.Get("Location") == "/", "known move must name its target")
 probe("GET", "/offline", 503, html)
 soft, softBody := probe("GET", "/soft-miss", 200, html)
 check(softBody == missing && soft.StatusCode != get.StatusCode, "negative control must expose a soft 404")
 probe("GET", "/restored", 404, html)
 restored.Store(true)
 _, recovered := probe("GET", "/restored", 200, html)
 check(recovered == "Restored resource", "restored resource must no longer return 404")
 rejected, _ := probe("POST", "/missing", 405, html)
 check(rejected.Header.Get("Allow") == "GET, HEAD", "method rejection must describe allowed methods")
 apiRejected, apiError := probe("POST", "/api/missing", 405, "application/json")
 check(apiRejected.Header.Get("Allow") == "GET, HEAD" && apiError == `{"error":"method_not_allowed"}`, "API method errors must retain JSON")
 fmt.Println("PASS: thirteen requests; missing, moved, unavailable, and restored remain distinct")
 fmt.Println(runtime.Version())
}

The thirteen requests distinguish a healthy homepage, HTML and API misses, a missing asset, known removal, a known move, and temporary unavailability. GET and HEAD agree on the missing resource’s status and declared length, but HEAD has no body. The query marker never appears in the response. The soft-miss body is identical to the genuine error page while its 200 status exposes the defect. Restoring a resource changes its answer from 404 to 200; POST is rejected separately with 405 and Allow, and an API method error retains its JSON format.

The assertions passed with Go 1.27.0 on macOS. The fixture checks response construction, not crawler indexing, CDN negative-cache invalidation, or a production proxy configuration. Before deployment, repeat the good-path and invented-path checks against the built site’s real serving stack in both languages.

Sources

RFC 9110: HTTP Semantics, 404, 410, HEAD, and representation semantics, checked 2026-09-09.
RFC 9111: HTTP Caching, negative response storage and cache controls, checked 2026-09-09.
MDN: 404 Not Found, missing resources and custom error pages, checked 2026-09-09.
Google Search: HTTP and network errors, soft 404s and indexing behavior, checked 2026-09-09.
Go net/http, response status commitment and redirect control, checked 2026-09-09.

Sitemaps guide crawler discovery

Wed, 30 Jun 2027 00:00:00 +0000

A sitemap gives a crawler an explicit list of URLs to consider. It is useful when new or deeply linked documents are difficult to discover through ordinary navigation. The list does not bypass authentication, grant crawl permission, guarantee indexing, or promise a revisit date. Treat it as a published inventory of intended public documents, and keep that inventory aligned with the site’s canonical URLs and actual release state.

Generate from the publication inventory

Use the same authoritative content selection that produces the public pages. Exclude drafts, private documents, and redundant tracking variants. A filename present in a source checkout is not necessarily a published URL. Conversely, pagination, translated documents, or generated routes may exist without a one-to-one source filename. Check the generator’s actual output rather than guessing the inventory from a directory listing.

The XML sitemap format uses a declared namespace and a urlset containing URL entries. Each loc supplies a fully qualified URL. XML escaping and URL encoding solve different problems: an ampersand separating query parameters must be escaped in the XML text, while characters inside the URL must also have the appropriate URI representation. Parse the generated file back to verify that the intended URL survives both layers.

Respect the format’s scope and size requirements. A sitemap normally lists URLs for the relevant host and permitted location scope; supported cross-site submission arrangements have additional rules. Each sitemap is limited to fifty thousand URLs and 52,428,800 uncompressed bytes. Gzip reduces transfer size without removing the uncompressed limit. Split larger inventories and publish an appropriate sitemap index rather than silently truncating the list.

Treat the inventory itself as public data. Omitting a private document from navigation is not protection if its URL appears in a public sitemap. Review the generated XML for unintended disclosures before publication, and enforce access control at the resource independently. Crawler discovery preferences are not an authorization mechanism.

When it breaks

A valid XML file can still list redirects, missing pages, or content that should not be indexed. Validate destinations as well as syntax, and align their canonical declarations with the intended inventory. An HTTP 200 response containing a login screen or generic error page is not proof that the listed document is available. Sitemap delivery and document usefulness are separate checks.

Modification dates can also become misleading. If lastmod is supplied, derive it from a significant change to the page rather than replacing every timestamp on each build. Google documents that it uses consistently accurate lastmod values and ignores priority and changefreq. Those fields are not a scheduling interface, and setting maximum priority everywhere does not command more crawling.

A sitemap cannot make a private page publicly retrievable or override robots exclusions. Nor does submission guarantee that a search engine will select the same canonical, fetch every entry, or include every page in results. Keep ordinary navigation usable and use observed crawling or indexing reports for those later outcomes. Do not describe a successful upload as proof that discovery and indexing have completed.

Publish an inventory, verify its targets. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

Save this as sitemap.py and run python3 sitemap.py. The standard-library fixture serves a generated XML sitemap and a tiny local publication inventory. Three entries are public, including one with two query parameters; a draft is excluded. The generated URLs point only to the temporary loopback origin, and the validator checks that scope before requesting destinations.

from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from urllib.parse import urlsplit
import json
import xml.etree.ElementTree as ET

namespace = 'http://www.sitemaps.org/schemas/sitemap/0.9'
ET.register_namespace('', namespace)
pages = {'/article': True, '/zh/article': True, '/report?year=2026&region=west': True, '/draft': False}
state = {'bad_entry': False}


class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 if self.path == '/sitemap.xml':
 root = ET.Element(f'{{{namespace}}}urlset')
 paths = [path for path, public in pages.items() if public]
 if state['bad_entry']:
 paths.append('/missing')
 for path in paths:
 item = ET.SubElement(root, f'{{{namespace}}}url')
 ET.SubElement(item, f'{{{namespace}}}loc').text = f'http://127.0.0.1:{self.server.server_port}{path}'
 status, data, kind = 200, ET.tostring(root, encoding='utf-8', xml_declaration=True), 'application/xml'
 else:
 status = 200 if pages.get(self.path) else 404
 data, kind = (b'published page' if status == 200 else b'missing'), 'text/plain'
 self.send_response(status)
 self.send_header('Content-Type', kind)
 self.send_header('Content-Length', str(len(data)))
 self.end_headers()
 self.wfile.write(data)

 def log_message(self, *args):
 pass


server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever)
thread.start()
try:
 def fetch(path):
 connection = HTTPConnection('127.0.0.1', server.server_port, timeout=5)
 try:
 connection.request('GET', path)
 response = connection.getresponse()
 return response.status, response.read()
 finally:
 connection.close()

 def validate(data):
 root = ET.fromstring(data)
 assert root.tag == f'{{{namespace}}}urlset'
 locations = [item.text for item in root.findall(f'{{{namespace}}}url/{{{namespace}}}loc')]
 assert len(locations) == len(set(locations))
 for location in locations:
 parsed = urlsplit(location)
 assert (parsed.scheme, parsed.hostname, parsed.port) == ('http', '127.0.0.1', server.server_port)
 path = parsed.path + ('?' + parsed.query if parsed.query else '')
 status, _ = fetch(path)
 if status != 200:
 raise ValueError(f'listed URL returned {status}: {path}')
 return locations

 status, data = fetch('/sitemap.xml')
 assert status == 200 and b'&amp;' in data
 locations = validate(data)
 assert len(locations) == 3 and all('/draft' not in value for value in locations)
 assert any('?year=2026&region=west' in value for value in locations)
 state['bad_entry'] = True
 _, invalid = fetch('/sitemap.xml')
 try:
 validate(invalid)
 except ValueError as error:
 rejection = str(error)
 else:
 raise AssertionError('missing destination accepted')
 print(json.dumps({'public_urls': len(locations), 'query_round_trip': True,
 'draft_excluded': True, 'invalid_destination_rejected': rejection,
 'indexing_tested': False}))
finally:
 server.shutdown()
 thread.join()
 server.server_close()

The first sitemap contains three unique loc values in the correct namespace. Its raw XML escapes the query separator as &, while parsing recovers the intended ampersand in the URL. All three destinations return 200, and the draft does not appear. The example omits optional modification dates because it has no publication-history source to justify them.

The fixture then adds a missing destination while continuing to serve well-formed XML with status 200. The second validation fails when that listed URL returns 404. This demonstrates why a sitemap syntax check alone cannot establish that its inventory is useful. The failure is reported instead of quietly removing the entry during validation.

No search engine is contacted, and this small validator does not implement every protocol limit or indexing rule. It verifies generation, escaping, namespace, uniqueness, local scope, and destination availability for the stated examples. Before publishing the real sitemap, compare it with the complete intended public inventory and check the released file at its advertised URL. That establishes what was published without claiming control over a crawler’s later choices.

Sources

Sitemaps.org protocol, XML structure, URL encoding, scope, and size limits, checked 2026-09-13.
Google: build and submit a sitemap, canonical URLs, meaningful lastmod, and unsupported hints, checked 2026-09-13.

Canonical URLs name a preferred representation

Sat, 26 Jun 2027 00:00:00 +0000

Several URLs can deliver the same document: a tracking parameter, alternate host, or trailing-slash variant may leave the meaningful content unchanged. A canonical link identifies the preferred representation among suitable duplicates. It helps consumers interpret that relationship, but it does not redirect an HTTP request or force a search engine to select the declared URL. The declaration is useful only when its target and surrounding signals describe the actual content relationship.

Choose identity before writing the tag

Decide which public URL should represent the document, then make internal links and relevant sitemap entries consistent with that choice. Do not strip every query parameter automatically. A tracking parameter may be redundant, while a language, page number, or product selection can materially change the representation. Collapsing those URLs without comparing their meaning can point distinct documents at the wrong preferred page.

In HTML, put a link element with rel="canonical" in a valid head and use a clear target URL. An HTTP Link header can express the relation for other representations, such as a PDF. Prefer an absolute public URL to avoid ambiguous resolution through a base URL or deployment prefix. Generate it from the site’s intended public configuration, rather than accidentally publishing an internal proxy host.

Canonicalization is not limited to a single origin by the link relation itself. Cross-domain declarations need careful content and ownership review; a canonical value does not prove control of the target. For translated pages, normally select an appropriate canonical in the same language when one exists. Language alternatives and canonical duplicates describe different relationships, so sending every translation to the English homepage erases meaningful distinctions.

When it breaks

A target can return an error, redirect somewhere else, or declare another canonical. Multiple conflicting declarations can also appear when a template, plugin, and response header each choose independently. Review the complete response rather than looking for the mere presence of one tag. Keep the preferred target usable and the declarations consistent with the actual representation.

An ordinary link relation leaves the original URL accessible unless routing separately changes it. If an old route should move for all clients, configure an appropriate HTTP redirect and verify its method behavior. If variants remain useful, a canonical declaration may coexist with successful responses at them. Do not equate the metadata signal with a redirect, an authorization rule, or an instruction that a crawler must obey.

Search engines combine signals and may choose another canonical. A correct tag therefore does not prove that indexing has converged. Distinguish what the origin serves from what a search service reports. Check for inconsistent internal links, sitemap entries, redirects, or language metadata before assuming that adding more copies of the same tag will solve disagreement.

Duplicates converge; languages stay distinct. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

Save this as canonical.py and run python3 canonical.py. The script creates a temporary loopback HTTP origin and reads actual HTML responses with Python’s HTML parser. It serves an English document, a tracking variant, a Chinese document, and two intentionally inconsistent metadata cases. The local URLs are demonstration identities, not search-engine submissions.

from html.parser import HTMLParser
from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
import json


class Canonical(HTMLParser):
 def __init__(self):
 super().__init__()
 self.urls = []

 def handle_starttag(self, tag, attrs):
 values = dict(attrs)
 if tag == 'link' and 'canonical' in values.get('rel', '').split():
 self.urls.append(values.get('href'))


class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 origin = f'http://127.0.0.1:{self.server.server_port}'
 chinese = self.path.startswith('/zh/')
 target = '/zh/article' if chinese else '/article'
 if self.path == '/zh/wrong':
 target = '/article'
 link = f'<link rel="canonical" href="{origin}{target}">'
 if self.path == '/conflict':
 link += f'<link rel="canonical" href="{origin}/another">'
 title = '中文文章' if chinese else 'English article'
 data = f'<!doctype html><html lang="{"zh" if chinese else "en"}"><head><meta charset="utf-8"><title>{title}</title>{link}</head><body><h1>{title}</h1></body></html>'.encode()
 self.send_response(200)
 self.send_header('Content-Type', 'text/html; charset=utf-8')
 self.send_header('Content-Length', str(len(data)))
 self.end_headers()
 self.wfile.write(data)

 def log_message(self, *args):
 pass


server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever)
thread.start()
try:
 origin = f'http://127.0.0.1:{server.server_port}'

 def fetch(path):
 connection = HTTPConnection('127.0.0.1', server.server_port, timeout=5)
 try:
 connection.request('GET', path)
 response = connection.getresponse()
 text = response.read().decode()
 parsed = Canonical()
 parsed.feed(text)
 return response.status, response.getheader('Location'), text, parsed.urls
 finally:
 connection.close()

 variant = fetch('/article?utm_source=example')
 primary = fetch('/article')
 translated = fetch('/zh/article')
 assert variant[:2] == (200, None) and variant[2] == primary[2]
 assert variant[3] == [origin + '/article']
 assert translated[3] == [origin + '/zh/article']
 wrong = fetch('/zh/wrong')
 conflict = fetch('/conflict')
 assert wrong[3] != [origin + '/zh/article']
 assert len(conflict[3]) == 2
 print(json.dumps({'variant_status': variant[0], 'variant_redirect': variant[1],
 'same_representation': variant[2] == primary[2],
 'english_target': variant[3], 'chinese_target': translated[3],
 'wrong_language_detected': True, 'conflicting_targets': len(conflict[3]),
 'search_engine_selection_tested': False}))
finally:
 server.shutdown()
 thread.join()
 server.server_close()

The tracking variant returns 200 without a Location header. Its bytes match the English primary page and its canonical points to /article. The Chinese page instead points to /zh/article. This verifies both the duplicate relationship in the fixture and the separation between the language representations.

The wrong-language case points a Chinese response at the English target, while the conflicting case emits two different canonical targets. Both still return 200, so a status-only check would miss the inconsistency. The assertions identify those specific errors by inspecting the metadata and the intended language mapping.

No crawler participates, and the small parser is not a complete canonicalization validator. The experiment establishes served declarations and response behavior, not Google’s selected canonical or ranking. Before changing production identity, review equivalent content, public targets, incoming routes, and language relationships together. Preserve existing URLs or their deliberate redirects so a metadata cleanup does not quietly break users’ saved links.

Sources

RFC 6596: canonical link relation, preferred representations and the scope of the link relation, checked 2026-09-13.
Google: consolidate duplicate URLs, canonical signals, consistent targets, and language handling, checked 2026-09-13.

Well-known endpoints provide a discovery convention

Tue, 22 Jun 2027 00:00:00 +0000

A well-known URI gives software a conventional place to discover information associated with an origin. The reserved /.well-known/ prefix reduces collisions between independently designed protocols, and registered suffixes identify their defining specifications. The prefix is a naming mechanism, not an implementation or a guarantee of trustworthy content. A server must still route the selected endpoint correctly, and a client must interpret the response according to that endpoint’s own contract.

Follow the suffix’s specification

RFC 8615 reserves the namespace for supported URI schemes. It does not require every site to publish every registered resource, and it does not define a common response schema for all suffixes. Consult the IANA registration and its linked specification before deciding the media type, authentication, redirect handling, caching, or transport requirements. A familiar path cannot substitute for these protocol-specific rules.

For web-based vulnerability contact information, RFC 9116 places security.txt at /.well-known/security.txt and requires HTTPS retrieval. A legacy /security.txt location can redirect to the well-known location. The document is UTF-8 plain text with defined fields, including at least one Contact and a single Expires field. Contact gives a reporting channel; it does not prove that a mailbox is staffed or that a submitted report has been accepted.

The document’s Expires field uses a timestamp to limit the age of the information. It is distinct from an HTTP Expires response header. A transport-level 200 does not make stale contact metadata current. Keep the address and policy maintained, and renew the document before it becomes stale. If optional fields such as Canonical or Policy are used, apply their specification rather than treating arbitrary URLs as trusted declarations.

When it breaks

A broad application fallback can return an HTML login page or a site’s home page with status 200 for every unknown path. That looks reachable to a shallow health check but supplies no usable protocol document. Preserve the exact route and inspect both media type and content. An accidental redirect to an unrelated origin can also change the trust context instead of merely moving a file.

The namespace itself authenticates nothing. HTTPS validation establishes the connection to the intended host, while document validation and any additional verification requirements establish whether its contents are usable. Do not infer permission to scan or attack a service from the presence of a security contact. Nor should a deployment blindly expose an entire directory because one standardized resource is intended to be public.

Returning 404 for an unimplemented suffix is different from silently inventing a generic successful document. Each supported endpoint should have an owner and a check suited to its protocol. A registry entry describes a convention; it is not evidence that a particular origin has implemented it or that its current response passed validation.

Well-known discovery: mechanism and evidence boundaries.

Check it quickly

Save this as well-known.py and run it with Python 3.14 using python3 well-known.py. The fixture serves only loopback HTTP, with an explicitly fictitious contact at example.invalid. It tests routing and selected document fields. Because it does not use HTTPS, it is not a complete RFC 9116 deployment or a transport-security test.

Validation (2026-10-17): Retained macOS arm64 Python 3.14.4 records match both complete fence payloads and report success with assertions enabled and loopback guards. This source review did not rerun them. Do not use -O or PYTHONOPTIMIZE: the unimplemented-route fetch is inside an assertion. Guards recorded connection attempts, not packet-capture request counts.

from datetime import datetime, timedelta, timezone
from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
import json

now = datetime.now(timezone.utc)
future = (now + timedelta(days=30)).isoformat().replace('+00:00', 'Z')
past = (now - timedelta(days=1)).isoformat().replace('+00:00', 'Z')
content = {'text': f'Contact: mailto:security@example.invalid\nExpires: {future}\n'}


class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 status = 200 if self.path == '/.well-known/security.txt' else 301 if self.path == '/security.txt' else 404
 data = content['text'].encode() if status == 200 else b''
 self.send_response(status)
 if status == 301:
 self.send_header('Location', '/.well-known/security.txt')
 self.send_header('Content-Type', 'text/plain; charset=utf-8')
 self.send_header('Cache-Control', 'no-store')
 self.send_header('Content-Length', str(len(data)))
 self.end_headers()
 self.wfile.write(data)

 def log_message(self, *args):
 pass


def check_contact_and_expiry(text):
 fields = dict(line.split(': ', 1) for line in text.splitlines())
 if not fields.get('Contact', '').startswith('mailto:'):
 raise ValueError('missing demonstration contact')
 expires = datetime.fromisoformat(fields['Expires'])
 if expires <= now:
 raise ValueError('expired metadata')


server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever)
thread.start()
try:
 def fetch(path):
 connection = HTTPConnection('127.0.0.1', server.server_port, timeout=5)
 try:
 connection.request('GET', path)
 response = connection.getresponse()
 return {'status': response.status, 'body': response.read().decode(),
 'location': response.getheader('Location'),
 'content_type': response.getheader('Content-Type')}
 finally:
 connection.close()

 valid = fetch('/.well-known/security.txt')
 assert valid['status'] == 200 and valid['content_type'] == 'text/plain; charset=utf-8'
 check_contact_and_expiry(valid['body'])
 legacy = fetch('/security.txt')
 assert legacy['status'] == 301 and legacy['location'] == '/.well-known/security.txt'
 assert fetch('/.well-known/unimplemented')['status'] == 404
 rejected = []
 for text in (content['text'].replace(future, past), f'Expires: {future}\n'):
 content['text'] = text
 response = fetch('/.well-known/security.txt')
 assert response['status'] == 200
 try:
 check_contact_and_expiry(response['body'])
 except ValueError as error:
 rejected.append(str(error))
 assert rejected == ['expired metadata', 'missing demonstration contact']
 print(json.dumps({'valid_status': valid['status'], 'legacy_status': legacy['status'],
 'unimplemented_status': 404, 'rejected_200_documents': rejected,
 'https_transport_tested': False}))
finally:
 server.shutdown()
 thread.join()
 server.server_close()

The well-known route first returns 200 with the expected plain-text media type, a contact, and expiry thirty days ahead. The legacy path returns 301 to the standard location, and an unimplemented suffix returns 404. The script then serves two invalid documents at the same successful route: one expired and one missing its contact. Both still return HTTP 200 but fail the field checks.

The parser handles only the demonstration format: its dictionary collapses duplicate fields, and it accepts only mailto contacts. It compares against startup time, not a refreshed observation time. It is not a complete parser for repeated fields, signatures, canonical URLs, redirects or URI syntax. These are deliberate limits, not full RFC compliance. Validate actual transport, content and freshness before publishing a real endpoint.

Sources

RFC 8615: Well-Known URIs, the reserved namespace and registration model, checked 2026-09-13.
RFC 9116: security.txt, location, HTTPS transport, contact fields, and expiry, checked 2026-09-13.
IANA: Well-Known URIs registry, registered suffixes and their defining specifications, checked 2026-09-13.
Python assert statement, optimization removes assertions and their expressions, checked 2026-10-18.

A charset tells the reader how the bytes become characters

Fri, 05 Mar 2027 00:00:00 +0000

Character encoding connects bytes to characters. In text/plain; charset=utf-8, the charset parameter names that connection; changing the parameter does not change the payload. Reliable text delivery requires agreement between the producer’s actual encoding and the consumer’s decoding rules, not simply the appearance of the word UTF-8 in a configuration file.

The format determines the rules

An application generally encodes its character strings into bytes when producing a response and decodes received bytes before processing text. For a compressed response, content decoding such as gzip decompression precedes character decoding. These are separate transformations. A database connection, template engine, and HTTP response generator can each introduce a mismatch, even when the final response header looks correct.

HTML has a specified encoding-detection algorithm, not arbitrary guessing whenever declarations conflict. In ordinary network parsing without a user override, a recognized byte order mark is considered before a supported transport declaration; an in-document declaration is considered later. Thus an HTML meta declaration does not generally override an explicit, supported HTTP charset. Keep both accurate, with the HTML encoding declaration entirely within the first 1024 bytes, instead of depending on conflict resolution to repair broken output.

Different formats do not share one universal missing-charset default. JSON exchanged outside a closed ecosystem must use UTF-8 under RFC 8259; its media-type registration defines no charset parameter. Networked JSON producers must not add a byte order mark. HTML’s encoding detection cannot be applied to JSON just because both are readable text. Choose the format first, then implement its encoding contract.

When it breaks

ASCII-only tests conceal many mistakes because their bytes are identical in UTF-8 and common legacy encodings. Add accented text and a currency symbol, or representative text from the languages actually supported. A Windows-1252 payload labeled UTF-8 can fail strict decoding. Conversely, these UTF-8 sample bytes decode successfully as Windows-1252 but produce different characters. No exception does not imply correct text.

Replacement decoding can make corrupt data look survivable while discarding evidence. Re-encoding replacement characters cannot generally reconstruct the original invalid bytes. Similarly, a string that was decoded incorrectly and later encoded as valid UTF-8 remains wrong: the final label and byte syntax can agree while the original meaning has already been lost. Preserve a protected original sample before attempting a bulk repair.

Compare decoded characters with the known original. A successful decode can still be wrong; changing a label alone is not transcoding.

Check it quickly

Run the following with Python 3.11 or newer; it needs no network or dependencies. A standard header parser extracts the charset, then Python explicitly decodes the supplied bytes. This consumer requires a charset and rejects unknown codecs. Those are its chosen rules, not a simulation of browser HTML detection or a claim that HTTP always requires this parameter.

from email.message import Message

text = "caf\u00e9 \u20ac"
utf8 = text.encode("utf-8")
legacy = text.encode("windows-1252")


def decode(header, payload):
 message = Message()
 message["Content-Type"] = header
 charset = message.get_content_charset()
 if charset is None:
 raise ValueError("missing charset")
 return payload.decode(charset, errors="strict")


cases = [
 ("utf8", "text/plain; charset=utf-8", utf8),
 ("legacy", "text/plain; charset=windows-1252", legacy),
 ("wrong-label", "text/plain; charset=utf-8", legacy),
 ("mojibake", "text/plain; charset=windows-1252", utf8),
 ("missing", "text/plain", utf8),
 ("unknown", "text/plain; charset=x-not-a-codec", utf8),
]
print("UTF-8 bytes:", utf8.hex(" "))
print("Windows-1252 bytes:", legacy.hex(" "))
outcomes = {}
for name, header, payload in cases:
 try:
 decoded = decode(header, payload)
 except (UnicodeDecodeError, LookupError, ValueError) as error:
 outcome = type(error).__name__
 print(name, outcome)
 else:
 outcome = "same" if decoded == text else "different"
 print(name, outcome, ascii(decoded))
 outcomes[name] = outcome

assert outcomes == {
 "utf8": "same", "legacy": "same", "wrong-label": "UnicodeDecodeError",
 "mojibake": "different", "missing": "ValueError", "unknown": "LookupError",
}
converted = legacy.decode("windows-1252").encode("utf-8")
assert converted == utf8
print("transcoded equals UTF-8:", converted == utf8)
replaced = legacy.decode("utf-8", errors="replace")
assert replaced == "caf\ufffd \ufffd"
print("replacement:", ascii(replaced))
assert b"\x80".decode("latin-1") == "\u0080"
assert b"\x80".decode("windows-1252") == "\u20ac"
print("Python Latin-1:", ascii(b"\x80".decode("latin-1")))

The first two cases report same despite different byte sequences. The mismatched legacy payload raises UnicodeDecodeError; the reverse mismatch reports different. Missing and unknown declarations produce ValueError and LookupError. Escaped output from ascii() makes differences visible without depending on terminal encoding or fonts. The successful transcode actually changes bytes by decoding with the source encoding and encoding with the destination encoding.

The replacement example contains two U+FFFD characters, confirming information loss. Finally, Python’s Latin-1 maps byte 0x80 to U+0080, whereas Windows-1252 maps it to the euro sign. WHATWG-compatible browsers treat labels such as iso-8859-1 and latin1 as Windows-1252. A Python codec result therefore cannot establish browser label behavior; the standard, not this fixture, supplies that mapping rule.

For production diagnosis, compare original payload bytes, the served header, any byte order mark, and the HTML declaration before changing storage. Locate the first incorrect decode or encode operation and test a known string through that boundary. If code points are correct but glyphs are missing, investigate font coverage separately. After fixing the producer, retest the full response path with non-ASCII samples; relabeling or accepting replacement output alone is not a verified repair.

Sources

WHATWG Encoding, encoding labels, Windows-1252 aliases, and decoder error modes, checked 2026-09-07.
HTML encoding detection, BOM, transport metadata, and document declaration order, checked 2026-09-07.
RFC 8259: JSON, UTF-8, BOM restrictions, and media-type registration, checked 2026-09-07.

CORS preflight depends on the request shape

Fri, 22 Jan 2027 00:00:00 +0000

CORS preflight asks whether a browser may send a particular cross-origin request shape. It is an OPTIONS request containing the proposed method and relevant request-header names. The browser checks the server’s response before sending the actual request. This mechanism is distinct from checking whether application JavaScript may read an actual response that has already arrived. Both stages can fail, but their effects on server-side execution differ.

The trigger is not simply “credentials are present” or “the request has side effects.” Safelisted GET, HEAD, or POST requests can avoid preflight when the other conditions are satisfied. A PUT or an application-supplied header such as Authorization generally requires it; a JSON content type also falls outside the safelisted content types. Changing request shape can therefore introduce an OPTIONS exchange even when the destination and business operation stay the same.

Separate the server’s request evidence from the script’s ability to read a response. “Allowed origin” means Access-Control-Allow-Origin; credential permission means Access-Control-Allow-Credentials: true. The five cases use synthetic local responses without cookies or authentication.

Match each permission to its stage

The preflight carries Access-Control-Request-Method and, when relevant, Access-Control-Request-Headers. For the PUT and X-Lab request below, the preflight response must authorize both the method and the custom header. Success at that stage does not remove the actual response’s CORS requirements. Error responses can also need the appropriate exposure headers when the permitted frontend must inspect their status and body. A functioning backend handler alone does not establish browser accessibility.

Credentials mode adds another independent constraint. With credentials: include, response exposure requires a matching explicit origin and Access-Control-Allow-Credentials: true; an allow-origin wildcard fails even if this particular test has no cookie to send. Browser cookie restrictions still apply independently. Do not use the credentials header as a substitute for cookie configuration, authentication, or an origin allowlist, and do not require session cookies merely to answer the browser’s preflight metadata query.

When it breaks

An API may permit PUT in its application router but omit PUT or a custom header from the preflight response. The browser then sends OPTIONS and never dispatches the actual PUT. Retrying the business handler or changing its database code will not fix that boundary. Conversely, a simple credential-mode GET can reach the handler without OPTIONS and then fail response exposure because the server returned a wildcard.

Preflight caching can obscure configuration changes during diagnosis. Access-Control-Max-Age controls a dedicated browser permission cache, subject to browser limits; it is not a cache of the API’s response body. Inspect the actual exchanges, including cache effects, before declaring a policy change ineffective. Where a response varies by requesting origin, configure cache variation appropriately so a shared cache does not reuse one origin’s selected permission for another request.

Check it quickly

Save the script as cors-preflight.py and run python3 cors-preflight.py --serve. Open the printed page and close both local servers with Control-C afterward. The fixture uses separate loopback ports and synthetic responses. It intentionally tests credentials mode without setting cookies or implementing authentication, so it demonstrates protocol checks rather than a complete credential-delivery setup.

Validation note, 17 October 2026: The revised loopback fixture passed its monitored startup checks with Python 3.14.4 and five sequential browser cases in headless Chrome 156.0.8078.36 using a fresh private profile. Wildcard plus include was unreadable; wildcard plus omit and exact-origin plus include were readable. The denied PUT stopped at preflight; the permitted PUT reached the server. Observed 204 responses omitted Content-Length. The private Python opener handles only startup OPTIONS; additional monitoring supplied connected-peer checks that the article helper does not contain. Normal browser CORS remained active. These synthetic results do not demonstrate real authentication or production behavior. Separate raw-response checks of the byte-identical API handler found empty 204s without Content-Length or Transfer-Encoding. Six offline malformed-response controls were rejected.

import json
import sys
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Event, Thread
from urllib.request import HTTPRedirectHandler, ProxyHandler, Request, build_opener

events = []


class NoRedirect(HTTPRedirectHandler):
 def redirect_request(self, *args, **kwargs):
 raise ValueError("This fixture does not follow redirects")


def open_owned_preflight(request):
 endpoint = api.socket.getsockname()
 if (endpoint != ("127.0.0.1", api.server_port)
 or type(endpoint[1]) is not int or not 1024 <= endpoint[1] <= 65535
 or request.full_url not in {f"http://127.0.0.1:{endpoint[1]}/{p}" for p in ("deny", "permit")}
 or request.get_method() != "OPTIONS" or request.data is not None
 or request.has_proxy()):
 raise ValueError("Only this fixture's owned loopback preflights are allowed")
 return direct_opener.open(request, timeout=5)


class API(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def reply(self, preflight=False):
 events.append({"method": self.command, "path": self.path,
 "requested_method": self.headers.get("Access-Control-Request-Method"),
 "requested_headers": self.headers.get("Access-Control-Request-Headers")})
 body = b"" if preflight else b"readable result"
 self.send_response(204 if preflight else 200)
 if not preflight:
 self.send_header("Content-Length", str(len(body)))
 self.send_header("Access-Control-Allow-Origin", "*" if self.path == "/wildcard" else app_origin)
 self.send_header("Access-Control-Allow-Credentials", "true")
 self.send_header("Vary", "Origin")
 if preflight and self.path == "/permit":
 self.send_header("Access-Control-Allow-Methods", "PUT")
 self.send_header("Access-Control-Allow-Headers", "X-Lab")
 self.send_header("Access-Control-Max-Age", "0")
 self.end_headers()
 self.wfile.write(body)

 def do_OPTIONS(self):
 self.reply(preflight=True)

 def do_GET(self):
 self.reply()

 def do_PUT(self):
 self.rfile.read(int(self.headers.get("Content-Length", "0")))
 self.reply()


class App(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def do_GET(self):
 if self.path == "/events":
 body = json.dumps(events).encode()
 kind = "application/json"
 else:
 body = ("""<!doctype html><meta charset="utf-8"><title>CORS preflight lab</title>
<h1>Credentials mode and preflight</h1><pre id="result">Running...</pre><script>
const api=""" + json.dumps(api_origin) + """;
async function run() {
 const cases=[
 ['wildcardInclude','/wildcard',{credentials:'include'}],
 ['wildcardOmit','/wildcard',{credentials:'omit'}],
 ['exactInclude','/exact',{credentials:'include'}],
 ['deniedPut','/deny',{method:'PUT',headers:{'X-Lab':'demo'},body:'synthetic'}],
 ['allowedPut','/permit',{method:'PUT',headers:{'X-Lab':'demo'},body:'synthetic'}]
 ];
 const results={};
 for(const [name,path,options] of cases) {
 try {const response=await fetch(api+path,options);results[name]={readable:true,status:response.status,body:await response.text()};}
 catch(error) {results[name]={readable:false,error:error.name};}
 }
 results.serverEvents=await (await fetch('/events')).json();
 window.labResult=results;
 document.getElementById('result').textContent=JSON.stringify(results,null,2);
}
run().catch(error=>{window.labError=String(error);});
</script>""").encode()
 kind = "text/html; charset=utf-8"
 self.send_response(200)
 self.send_header("Content-Type", kind)
 self.send_header("Content-Length", str(len(body)))
 self.send_header("Cache-Control", "no-store")
 self.end_headers()
 self.wfile.write(body)


api = HTTPServer(("127.0.0.1", 0), API)
app = HTTPServer(("127.0.0.1", 0), App)
api_origin = f"http://127.0.0.1:{api.server_port}"
app_origin = f"http://127.0.0.1:{app.server_port}"
# A process-local client for these two owned loopback requests only.
# It changes no system settings or global urllib opener.
direct_opener = build_opener(ProxyHandler({}), NoRedirect())
threads = [Thread(target=server.serve_forever, daemon=True) for server in (api, app)]
for thread in threads:
 thread.start()
try:
 headers = {"Origin": app_origin, "Access-Control-Request-Method": "PUT",
 "Access-Control-Request-Headers": "x-lab"}
 for path in ("deny", "permit"):
 with open_owned_preflight(Request(api_origin + "/" + path, method="OPTIONS", headers=headers)) as response:
 assert response.status == 204
 assert response.headers.get("Content-Length") is None
 assert response.headers.get("Access-Control-Allow-Methods") == ("PUT" if path == "permit" else None)
 assert response.headers.get("Access-Control-Allow-Headers") == ("X-Lab" if path == "permit" else None)
 events.clear()
 print(json.dumps({"app_url": app_origin, "wire_preflight_checks": "PASS",
 "real_authentication_or_cookies": False}), flush=True)
 if "--serve" in sys.argv:
 Event().wait()
except KeyboardInterrupt:
 pass
finally:
 for server in (api, app):
 server.shutdown()
 server.server_close()
 for thread in threads:
 thread.join(timeout=5)

Five fetch cases run. Wildcard plus include is unreadable; the same wildcard with omit is readable. An exact origin plus include is readable. The denied PUT fails preflight because its permissions are omitted, while the permitted PUT succeeds with an explicitly allowed method and X-Lab header. The page displays server events, not just JavaScript success flags, so the execution boundary can be checked directly.

Expect three GET events, two OPTIONS events, and exactly one PUT to /permit. Both preflights request PUT and the lowercase header name x-lab; no PUT reaches /deny. The permit response sets max-age to zero to avoid relying on cached permission. The default Python-only run checks wire headers, while the browser run verifies dispatch and exposure. Preserve those distinct checks when adapting the example, and test the actual method, content type, custom headers, credentials mode, and response path used by your frontend.

Sources

MDN CORS, preflight criteria and credentialed request exchanges, checked 2026-09-13.
MDN Access-Control-Allow-Credentials, credential mode requirements for exposing a response, checked 2026-09-13.
WHATWG Fetch Standard, method and header permission checks and the CORS-preflight cache, checked 2026-09-13.

Caddy automates HTTPS issuance and renewal

Thu, 21 Jan 2027 00:00:00 +0000

Caddy can obtain and renew managed certificates without an operator manually implementing ACME. That removes repetitive work, not the prerequisites for issuance or the need to notice failures. A valid certificate today proves neither that the next renewal can complete nor that every TLS hop uses the same certificate. Identify the names, validation route, storage, and termination boundary first.

Give automatic HTTPS a name

Automatic HTTPS activates when Caddy knows a qualifying hostname or address through its configuration, subject to documented exceptions and overrides. A hostname in a Caddyfile site address is one common trigger. The file_server directive serves files; it is not, by itself, a request for a public certificate for an unspecified domain.

The command-line distinction is concrete: bare caddy file-server defaults to port 80. Supplying --domain makes it attempt HTTPS for that hostname and changes the default listener to port 443. Neither command spelling proves that DNS, port forwarding, or certificate authorization is correct. Explicit HTTP configuration and manually loaded certificates also change automatic behavior.

Local development has a different trust boundary. Caddy can use its local certificate authority for internal names. Clients must trust that authority through an appropriate installation process; a certificate working on one machine does not make it publicly trusted. Do not distribute a private CA’s signing key to solve a client trust problem.

Match the challenge to the route

Caddy enables HTTP-01 and TLS-ALPN-01 by default. HTTP-01 validation enters through public port 80, while TLS-ALPN-01 uses port 443 and a special TLS handshake. Caddy can choose between available challenges. A failed HTTP challenge does not imply that DNS validation is the only possible alternative; inspect which methods and routes are actually available.

A proxy or CDN can interfere with either path, but its presence does not automatically make issuance impossible. HTTP challenge traffic may be forwarded correctly even when the CDN terminates ordinary HTTPS. DNS-01 proves control through a TXT record and does not require the requesting server to accept inbound validation traffic. It needs a supported provider integration and appropriately scoped DNS credentials.

When it breaks

Stale A or AAAA records, blocked ports, incorrect forwarding, or a changed CDN rule can break validation after the first issuance succeeded. Check both address families and the actual failed challenge in logs. Repeatedly deleting storage and retrying is a poor diagnostic: it removes reusable certificate state and can increase unnecessary issuance attempts.

Persist Caddy’s data storage across replacement instances and protect it because it contains sensitive key material. Multiple instances need a storage arrangement with the coordination semantics Caddy expects. A directory merely bearing the same name on several machines is not shared storage. Test recovery of that configuration without exposing keys in logs or source control.

Automation can fail while the existing certificate still works. Monitor renewal errors and the expiry of the certificate clients actually receive, leaving enough time for response. A successful origin renewal does not prove the CDN’s public certificate changed. Likewise, CDN TLS does not remove the need to secure and validate a separate HTTPS origin connection.

Use a staging issuer for issuance experiments, and keep its untrusted test certificates out of production. DNS credential rotation needs a controlled validation check. Choose alert lead time around response requirements and certificate lifetime; a fixed thirty-day arithmetic example is not a specification of Caddy’s renewal schedule.

Caddy certificate lifecycle: mechanism and evidence boundaries.

Check it quickly

Save this as expiry_alert.py and run python3 expiry_alert.py. Use assertions enabled, without -O or PYTHONOPTIMIZE. It needs only the standard library and uses a fixed UTC observation time. The synthetic thirty-day threshold belongs to this operator alert, not to Caddy. No domain, certificate authority, network listener, key, or system trust setting is touched.

from datetime import datetime, timedelta, timezone

# An operator-chosen alert threshold, not Caddy's renewal scheduler.
ALERT_BEFORE = timedelta(days=30)
NOW = datetime(2026, 10, 15, 12, tzinfo=timezone.utc)


def expiry_alert(not_after, now):
 for value in (not_after, now):
 if value.tzinfo is None or value.utcoffset() is None:
 raise ValueError("timezone-aware timestamps required")
 remaining = not_after - now
 if remaining <= timedelta(0):
 return "expired"
 if remaining <= ALERT_BEFORE:
 return "warning"
 return "ok"


for name, days, expected in [
 ("fresh", 80, "ok"), ("due", 20, "warning"),
 ("boundary", 30, "warning"), ("expired", -1, "expired"),
]:
 not_after = NOW + timedelta(days=days)
 actual = expiry_alert(not_after, NOW)
 assert actual == expected
 print(f"{name}: remaining={days}d alert={actual}")
assert expiry_alert(NOW, NOW) == "expired"
try:
 expiry_alert(NOW.replace(tzinfo=None), NOW)
except ValueError:
 print("naive timestamp rejected")
else:
 raise AssertionError("missing timezone accepted")
print("PASS: expiry alert boundaries; no certificate issuance tested")

The four printed cases are ok, warning, warning, and expired. Exactly thirty days is included in the warning interval; expiry itself is classified separately. The negative case supplies a naive expiry timestamp; the function also checks the observation time, but that separate branch is not exercised. These assertions verify the alert’s declared boundaries rather than relying on today’s date or a particular certificate lifetime.

Retained exact records cover the unchanged arithmetic fixture; this source review did not rerun it. The exercise does not run Caddy, issue a certificate, or observe renewal. For deployment evidence, verify the served certificate and configured challenge route, then observe successful renewal using the intended persistent storage. Keep that operational evidence separate from this small, repeatable date-arithmetic check.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
Caddy automatic HTTPS, activation, local trust, challenge selection, and storage, checked 2026-10-15.
Caddy file-server command, default listener and the domain flag, checked 2026-10-15.
Let’s Encrypt challenge types, validation ports, DNS automation, and scoped credentials, checked 2026-10-15.
Let’s Encrypt rate limits, staging for experiments and issuance failures, checked 2026-10-15.

The Link header relates adjacent resources

Fri, 04 Dec 2026 00:00:00 +0000

The Link response field describes relationships between the current resource and other resources. An API returning JSON can advertise its next page without making clients scrape HTML or invent a query parameter. The field supplies a target and a relation; whether to follow it remains a client decision. It is useful metadata, not a command that every browser or crawler must execute.

Resolve the returned target against the effective request URI. Preserve relation grammar and apply destination and browser-exposure rules separately.

A link has a context and a target

A value such as </items?cursor=b>; rel="next" links the response’s context resource to a target URI reference. With no explicit anchor parameter, the context is normally the requested resource. Resolve relative targets against the effective request URI after redirects, not against the client’s current working directory or an assumed API root. A query-only target changes the query while preserving the path.

The rel parameter names the relationship. Registered relations include next, prev, alternate, and canonical; extension relations can use URI identifiers. Additional parameters describe the target, for example type="application/json". They do not change the actual response’s Content-Type, grant access, or prove that the target is trustworthy. An anchor can change the context, so clients that cannot honor that context must not silently attach the link to the original resource.

Relations also have different consumers. A crawler may interpret canonical when selecting a preferred indexing URL, while a browser can use supported preload hints to begin fetching assets. Neither relation redirects the current request. Test the behavior of the intended consumer separately from checking field syntax; an ordinary HTTP library may correctly expose the header and take no further action.

Follow pagination as a server contract

For pagination, the server decides whether a continuation exists and supplies its exact target. A cursor may encode ordering, filters, and a position in a snapshot. Treat it as opaque: guessing the next page number or changing cursor bytes can skip results, repeat rows, or invalidate the request. Preserve the returned query rather than rebuilding it from assumptions about today’s API implementation.

A missing next relation normally terminates that API’s traversal, subject to its documented pagination contract. It does not prove the collection is immutable. Concurrent insertions and deletions can still change membership between requests unless the API provides snapshot semantics. Bound requests and detect repeated continuation targets so a server bug cannot create an endless crawl.

When it breaks

Multiple links can appear in separate Link field lines or a comma-separated field value. Naively splitting on every comma is incorrect: quoted parameter values and URI references can contain commas. Use a parser implementing Web Linking, retain multiple values, and interpret relation tokens correctly. A proxy that overwrites repeated fields can silently remove the pagination link even while the response body remains valid.

Following an absolute target blindly is another failure. An authenticated API client must apply its destination and credential-forwarding policy before requesting a different origin; a link is not authorization to send tokens elsewhere. For browser code, successful CORS access to the body does not automatically expose Link. The server must include it in Access-Control-Expose-Headers when cross-origin JavaScript needs to read it.

Check it quickly

Earlier source-only review inspected this fixture without rerunning the local listener or browser CORS behavior. In the separate 2026-10-17 run, both complete original EN/ZH Python fences ran once on macOS 26.5.1 arm64 with Python 3.14.4, -I -B, optimization disabled and all four original assert statements retained. Both produced the exact confirmation line, exited zero and had empty stderr; their sockets, threads and owned temporary work were cleaned up.

Each language case recorded two successful loopback connections and two application-level GET/status events: /items?cursor=a then /items?cursor=b, both 200, matched to one owned listener generation. These observations are not packet capture or independent wire-status measurement. The second target is still the hardcoded ?cursor=b; no general Link parser, alternate-resource retrieval, redirect or credential policy, pagination-loop handling, or browser CORS behavior was tested.

This Python 3 experiment creates a two-page loopback API. The first response has separate next and alternate fields; the terminal response has no links. It examines the wire values and resolves the known fixture target with the standard URL library. It deliberately does not implement a general Link parser.

from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from urllib.parse import urljoin, urlsplit

class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 self.send_response(200)
 self.send_header("Content-Length", "2")
 if self.path == "/items?cursor=a":
 self.send_header("Link", '<?cursor=b>; rel="next"')
 self.send_header("Link", '</items.csv>; rel="alternate"')
 self.end_headers()
 self.wfile.write(b"[]")

 def log_message(self, *args):
 pass

with ThreadingHTTPServer(("127.0.0.1", 0), Handler) as server:
 worker = Thread(target=server.serve_forever, daemon=True)
 worker.start()
 client = HTTPConnection(*server.server_address, timeout=3)
 try:
 client.request("GET", "/items?cursor=a")
 response = client.getresponse()
 links = response.headers.get_all("Link")
 assert response.read() == b"[]"
 assert links == ['<?cursor=b>; rel="next"',
 '</items.csv>; rel="alternate"']
 base = f"http://127.0.0.1:{server.server_port}/items?cursor=a"
 target = urlsplit(urljoin(base, "?cursor=b"))
 client.request("GET", target.path + "?" + target.query)
 response = client.getresponse()
 assert response.read() == b"[]"
 assert response.headers.get_all("Link") is None
 print("two fields preserved; terminal page has no next link")
 finally:
 client.close()
 server.shutdown()
 worker.join()

Expect the final confirmation line without an assertion failure. The query-only target should resolve to /items?cursor=b, and both first-page fields should survive. This verifies response construction, repeated-field retrieval, and relative resolution. A production client additionally needs parser tests for quoted commas, redirects, multiple relation tokens, cross-origin targets, and pagination loops. Browser header exposure must be checked from the actual calling origin; this standard-library client does not enforce CORS.

Sources

RFC 8288: Web Linking, link context, relation types, and field grammar, checked 2026-09-06.
MDN: Link header, syntax and relative URL resolution, checked 2026-09-06.
MDN: Access-Control-Expose-Headers, exposing response metadata to cross-origin browser code, checked 2026-09-06.

Read status together with transport and application results

Thu, 03 Dec 2026 00:00:00 +0000

A caller needs more than a three-digit number to decide what happened. Did an HTTP response arrive, did its body arrive completely, and does the application result satisfy the requested operation? These are separate questions. A 200 response can carry a documented application-level failure, or its transfer can break after the headers. A connection failure before any response supplies no HTTP status at all. Preserve those distinctions in diagnostics and metrics.

Classify the evidence you received

An HTTP error response is still a response. It can contain a useful representation, authentication challenge, request identifier, or Retry-After field. Some client libraries raise exceptions for 4xx and 5xx while retaining access to those details. Catch the specific HTTP error before a broader transport exception, and read its relevant fields. Otherwise a meaningful 429 can be flattened into the same generic error as a failed connection.

A successful HTTP exchange can also be an application envelope. For example, a batch or RPC contract may return 200 while individual operations report failure in the body. Interpret the documented schema instead of assuming all such responses mean success or that every such API is necessarily violating HTTP. A monitoring check should validate the outcome it promises to monitor, including the body when that contract requires it.

When it breaks

A status recorded when headers arrive can conceal a later truncated body. Content-Length, stream termination, decompression, and parsing can all affect whether a usable representation was delivered. Report a received status and an incomplete transfer separately. Do not count a partially delivered 200 response as a successful application result simply because the status metric was incremented early.

Retry choices require more context than the first digit. 429 indicates rate limiting and may provide Retry-After. 503 can also suggest a delay. That field can use delay seconds or an HTTP date; neither form promises the server will be healthy at the requested time. Combine server guidance with a bounded policy, backoff, and the operation’s semantics instead of immediately replaying every error.

For a non-idempotent request, a lost response can leave its effect uncertain. The server might have committed before the client lost contact. Repeating without a suitable idempotency contract can duplicate work. Conversely, some 4xx conditions become recoverable after authentication, state, or rate-limit conditions change. Class-wide “always retry” and “never retry” rules erase the information needed to make that decision.

The status also does not identify which component generated it. An origin, gateway, or intermediary may send an error response. Correlate available request identifiers and logs before attributing the failure. A diagnostic value such as curl’s 000 is not a registered HTTP response status; it commonly indicates that no usable HTTP status was obtained.

HTTP: three separate outcomes. The retained Mac trial is scoped below; this diagram and adoption review did not rerun the examples.

Check it quickly

Save this as status-client.py with Python 3.14 assertions enabled. Five loopback responses and a bound-but-not-listening endpoint check classification. The private opener disables proxy discovery only for this fixture, without changing system proxies. The no-response key tests absence of a returned HTTP status, not a particular refusal error; the retained Mac trial observed a timeout on that held endpoint.

from http.client import IncompleteRead
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from urllib.error import HTTPError
from urllib.request import ProxyHandler, build_opener
import json
import socket


class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 if self.path == '/application-error':
 status, data, retry = 200, b'{"ok":false,"error":"demo"}', None
 elif self.path == '/busy':
 status, data, retry = 503, b'{"error":"unavailable"}', '3'
 elif self.path == '/limit':
 status, data, retry = 429, b'{"error":"limited"}', '2'
 else:
 status, data, retry = 404, b'{"error":"missing"}', None
 if self.path == '/truncated':
 self.send_response(200)
 self.send_header('Content-Length', '100')
 self.end_headers()
 self.wfile.write(b'partial')
 return
 self.send_response(status)
 self.send_header('Content-Type', 'application/json')
 self.send_header('Content-Length', str(len(data)))
 if retry:
 self.send_header('Retry-After', retry)
 self.end_headers()
 self.wfile.write(data)

 def log_message(self, *args):
 pass


opener = build_opener(ProxyHandler({}))


def inspect(url):
 try:
 response = opener.open(url, timeout=3)
 except HTTPError as error:
 response = error
 except OSError: # Includes URLError and socket timeout/connection errors.
 return {'http_status': None, 'transport_failure': True}
 with response:
 result = {'http_status': response.code, 'retry_after': response.headers.get('Retry-After')}
 try:
 body = response.read()
 except (IncompleteRead, OSError):
 return {**result, 'complete_body': False}
 return {**result, 'complete_body': True, 'payload': json.loads(body)}


server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever)
thread.start()
try:
 origin = f'http://127.0.0.1:{server.server_port}'
 results = {path: inspect(origin + path) for path in
 ('/application-error', '/busy', '/limit', '/missing', '/truncated')}
 with socket.socket() as reserved:
 reserved.bind(('127.0.0.1', 0)) # Still owned, never listening: no HTTP response expected.
 results['no-response'] = inspect(f'http://127.0.0.1:{reserved.getsockname()[1]}/')
 assert results['/application-error']['http_status'] == 200
 assert results['/application-error']['payload']['ok'] is False
 assert results['/busy']['http_status'] == 503 and results['/busy']['retry_after'] == '3'
 assert results['/limit']['http_status'] == 429 and results['/limit']['retry_after'] == '2'
 assert results['/missing']['http_status'] == 404
 assert all(results[path]['complete_body'] is True for path in
 ('/application-error', '/busy', '/limit', '/missing'))
 assert results['/truncated'] == {'http_status': 200, 'retry_after': None, 'complete_body': False}
 assert results['no-response'] == {'http_status': None, 'transport_failure': True}
 print(json.dumps(results))
finally:
 server.shutdown()
 thread.join()
 server.server_close()

The expected application envelope has HTTP 200 and ok: false. The busy and limited endpoints yield 503 and 429, retaining Retry-After values three and two. The missing endpoint remains HTTP 404. A normal return for each of these four cases requires a complete read and JSON parsing; the fixture does not wait or retry.

The truncated endpoint advertises one hundred bytes but sends only partial; IncompleteRead retains status 200 with complete_body: false. Opening catches HTTPError before OSError. During reading, IncompleteRead or OSError, including TimeoutError, retains status and Retry-After while marking an incomplete body. JSON parsing and other protocol failures can still escape; this is not general exception handling.

On 2026-10-18, one supervised macOS 26.5.1 arm64 trial used Python 3.14.4 with -I -B and assertions. A-en and A-zh each ran the complete, identical code body above once and returned all six expected results. Separate B-positive reused inspect, holding a 429 body open until read-phase TimeoutError; it retained 429, Retry-After 2 and complete_body: false. B is not another endpoint here. Four mutants failed their intended assertions: discarding late-body metadata, substituting a received response for no-response, discarding HTTPError metadata, and declaring truncation complete.

The seven cases completed in 19.1647 seconds with 32 observed inspect calls; this is not a benchmark. The inherited environment stayed unchanged, but isolated interpreter flags do not establish default-environment or startup-customization behavior. Cleanup was observed for this run only; no wire capture or production behavior was verified. The displayed socket timeout, shutdown() and unbounded thread.join() do not bound process duration or establish recovery and safe replay. Earlier uncorrected bytes, compile-only records and diagnostic failures remain history; this trial does not retroactively validate them.

Sources

RFC 9110: HTTP Semantics, status interpretation, retries, and Retry-After, checked 2026-09-13.
RFC 6585: additional HTTP status codes, the meaning of 429 Too Many Requests, checked 2026-09-13.
Python 3.14: urllib.request, HTTP response objects and error handling, checked 2026-09-13.
Python 3.14: urllib.error, HTTPError response access and URLError hierarchy, checked 2026-10-18.
Python 3.14: http.client, response reads and IncompleteRead, checked 2026-10-18.
Python 3.14: assert statements, optimization removes assertion checks, checked 2026-10-18.
Python 3.14: socket, TimeoutError/OSError and platform-dependent timeout behavior, checked 2026-10-19.

UTF-8 preserves text only when byte boundaries and decoding rules agree

Wed, 02 Dec 2026 00:00:00 +0000

UTF-8 encodes Unicode scalar values using one to four bytes. ASCII characters retain their familiar one-byte representation, while many other characters need longer sequences. This makes UTF-8 a useful interchange format, but it does not make arbitrary bytes valid text or every byte offset a character boundary. Correct handling requires an agreed encoding, strict validity rules where appropriate, and a clear distinction between bytes and decoded text.

A read boundary can split a character. Carry incomplete bytes, finalize strictly, and name the unit you are counting.

When it breaks

A file or network read can end halfway through a multibyte sequence. Decoding each chunk independently may reject a perfectly valid complete stream. Replacing errors separately in every chunk is worse when it destroys bytes that would have formed a character with the next read. A streaming decoder needs to retain incomplete input and validate it when more bytes arrive or the stream ends.

UTF-8’s recognizable leading and continuation-byte patterns do not make arbitrary slicing safe. A program that truncates an encoded byte string can cut a sequence in half. Nor does successful decoding under another codec prove that the chosen interpretation is correct. Latin-1 maps every byte, so wrongly decoding UTF-8 as Latin-1 can produce valid Unicode text whose content is nevertheless mojibake.

Encoding validity is separate from text equivalence and display width. A single visible accented letter can use one code point or a base letter plus a combining mark. Normalization can make selected forms comparable, but decoding UTF-8 does not automatically normalize them. Similarly, a byte-order mark may become a leading text character unless the selected codec intentionally consumes it.

Check it quickly

Save this as check-utf8.py and run python3 check-utf8.py with Docker. It uses standard-library codecs and Unicode operations in a pinned Python 3.11.16 container without network access. All examples are fixed in-memory bytes and strings. The script performs no database migration or file conversion and does not depend on the terminal correctly displaying an emoji:

import json,subprocess,uuid
image="python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84"
program=r"""
import codecs,json,unicodedata

text='A中🙂';encoded=text.encode('utf-8');assert len(text)==3 and len(encoded)==8
assert encoded.hex()=='41e4b8adf09f9982' and encoded.decode('utf-8')==text
pieces=[encoded[:2],encoded[2:5],encoded[5:]]
split_errors=0
for piece in pieces:
 try:piece.decode('utf-8')
 except UnicodeDecodeError:split_errors+=1
assert split_errors==3
decoder=codecs.getincrementaldecoder('utf-8')(errors='strict');decoded=[]
for piece in pieces:decoded.append(decoder.decode(piece,final=False))
decoded.append(decoder.decode(b'',final=True));assert ''.join(decoded)==text
rejected={}
for label,raw in {'overlong':bytes.fromhex('c0af'),'surrogate':bytes.fromhex('eda080'),
 'too_large':bytes.fromhex('f4908080'),'truncated':bytes.fromhex('e4b8')}.items():
 try:raw.decode('utf-8');raise AssertionError('invalid UTF-8 accepted')
 except UnicodeDecodeError as error:rejected[label]=error.reason
bad_tail=codecs.getincrementaldecoder('utf-8')();assert bad_tail.decode(bytes.fromhex('e4b8'),final=False)==''
try:bad_tail.decode(b'',final=True);raise AssertionError('incomplete final sequence accepted')
except UnicodeDecodeError:final_rejected=True
mojibake=encoded.decode('latin-1');assert mojibake!=text and mojibake.encode('latin-1')==encoded
decomposed='e\u0301';composed='é';assert decomposed!=composed and unicodedata.normalize('NFC',decomposed)==composed
bom=b'\xef\xbb\xbf'+encoded;assert bom.decode('utf-8').startswith('\ufeff') and bom.decode('utf-8-sig')==text
replacement=bytes.fromhex('ff41').decode('utf-8',errors='replace');assert replacement=='\ufffdA'
print(json.dumps({'text':text,'code_points':len(text),'bytes':len(encoded),'hex':encoded.hex(),
 'chunk_hex':[x.hex() for x in pieces],'independent_chunk_failures':split_errors,'incremental_output':decoded,
 'invalid_rejections':rejected,'final_incomplete_rejected':final_rejected,'latin1_code_points':[hex(ord(c)) for c in mojibake],
 'normalization_lengths':[len(decomposed),len(composed)],'bom_utf8_first':hex(ord(bom.decode('utf-8')[0])),
 'utf8_sig_strips_bom':bom.decode('utf-8-sig')==text,'replacement':replacement},ensure_ascii=False))
"""
name='xbcatm-socket-'+uuid.uuid4().hex[:10]
try:
 result=subprocess.run(['docker','run','--rm','-i','--name',name,'--network','none',
 '--memory','128m','--pids-limit','64',image,'python3','-'],input=program,
 capture_output=True,text=True,timeout=60)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),ensure_ascii=False,sort_keys=True))
finally:
 subprocess.run(['docker','rm','-f',name],capture_output=True,text=True,timeout=10)

The string A中🙂 has three Python string elements representing code points and encodes to eight bytes: 41e4b8adf09f9982. ASCII contributes one byte, the Chinese character three, and the emoji four. The round trip returns the original text. Counting encoded bytes and counting these code points therefore answer different questions; neither should automatically be used as a displayed-column count.

The bytes are split into 41e4, b8adf0, and 9f9982. Each piece fails when independently decoded as complete UTF-8. An incremental decoder instead produces A, then 中, then 🙂, retaining partial sequences between calls. A final empty call with final=True completes validation. A separate decoder holding only e4b8 rejects that incomplete sequence when finalized, rather than silently losing a trailing character.

Four negative cases reject an overlong encoding, a surrogate code point, a value above the Unicode limit, and an incomplete sequence. Valid UTF-8 excludes those forms even when some bytes resemble plausible continuation patterns. The example catches UnicodeDecodeError specifically. It does not accept an unrelated exception as proof that byte validation worked, and it does not use permissive surrogate handling.

Decoding the valid UTF-8 bytes as Latin-1 succeeds but yields different code points. In this controlled case, encoding that mistaken string back as Latin-1 recovers the original bytes. That reversible example is not a universal repair method: prior replacement, normalization, or lossy conversion may have discarded information. Preserve original bytes when diagnosing an encoding mismatch before attempting a corrective conversion.

The script also compares decomposed e plus a combining acute accent with composed é. The strings initially differ and have lengths two and one; NFC normalization makes them equal. Separately, plain UTF-8 decoding keeps an initial BOM as U+FEFF, while utf-8-sig consumes it. These are two different transformations, and neither should be applied merely because a text string looks unusual.

Define the byte contract at each boundary

An HTTP charset declaration or a file-format specification tells a consumer how to interpret bytes; changing the declaration does not transcode the payload. Ensure the producing encoder, transport metadata, parser, and storage interface agree. Test a multibyte sample across the actual boundaries, especially where code truncates values, buffers partial reads, or counts limits expressed in bytes rather than characters.

Choose error handling according to the data contract. The fixture shows that errors=‘replace’ turns invalid input into U+FFFD, a valid but lossy text result. That may suit a diagnostic display, but it cannot prove preservation of identifiers or signed data. Keep rejection, replacement, BOM removal, and normalization as deliberate decisions with different observable effects.

Finally, use a character-aware operation when limiting decoded text, and a specified framing rule when limiting encoded messages. Incremental decoding solves split byte sequences; it does not define grapheme clusters, line wrapping, or database comparison semantics. Verify the property the application actually requires instead of treating “we use UTF-8” as proof that every text boundary is correct.

Sources

RFC 3629 UTF-8, valid byte sequences, scalar range and invalid encodings, checked 2026-09-12.
Unicode UTF and BOM FAQ, encoding boundaries and byte-order-mark behavior, checked 2026-09-12.
Python 3.11 codecs, incremental decoding, finalization and error policies, checked 2026-09-12.
Python 3.11 unicodedata, Unicode normalization as a separate operation, checked 2026-09-12.

SNI selects a TLS certificate for the requested hostname

Sun, 08 Nov 2026 00:00:00 +0000

Server Name Indication gives a TLS server a hostname while it is choosing handshake parameters and a certificate. In the certificate-based, no-early-data path discussed here, this happens before ordinary HTTP bytes, not before any bytes cross the network. SNI is a selection hint; the client must still verify the selected identity.

One address, separate decisions

The client places server_name in ClientHello. A server hosting multiple names can use it to choose a certificate or security configuration. TLS 1.3 encrypts its certificate message after ServerHello, so describing the whole selection and certificate exchange as happening before encryption is inaccurate.

A DNS hostname in SNI is not the destination IP, and supplying it does not require the test name to resolve through DNS. RFC 6066 excludes literal IP addresses from its HostName field and treats DNS names case-insensitively. Clients connecting by IP may omit SNI and verify an IP identity instead.

Multiple sites do not necessarily need separate certificates. One certificate can cover multiple subject alternative names, and a server without SNI can choose a default certificate. Whether the client accepts that certificate depends on its expected name and trust policy. SNI enables selection; it does not create trust or prove ownership of the requested hostname.

When it breaks

A client can reach the correct IP yet receive a default certificate for another name. Disabling hostname verification would hide that routing error. Check the connection address, intended TLS name, selected certificate’s SANs, trust chain, and expiry independently.

HTTP Host or an HTTP/2 authority belongs to the application layer. Changing it does not redo the TLS handshake or replace its certificate. ALPN separately negotiates an application protocol such as HTTP/2; it does not verify a hostname or authorize a tenant. Certificate selection, identity verification, protocol negotiation, and application routing need separate checks.

Unknown SNI may lead to handshake rejection or a default certificate, depending on server policy; absence of SNI does not universally mean failure. Without ECH, the ordinary ClientHello exposes SNI on the network. TLS 1.3 alone does not hide it. ECH requires additional deployment support and does not conceal every traffic characteristic.

SNI certificate selection: mechanism and evidence boundaries.

Check it quickly

Save this as sni_probe.go and run go run sni_probe.go with Go 1.23 or newer. It generates a short-lived test CA and two leaf certificates entirely in memory. Only a local client trusts that CA. All requests connect to one loopback IP; ServerName supplies the intended TLS name without requiring public DNS. Sessions and HTTP connections are not reused.

The CA and leaf certificates expire after one hour. Their limited lifetime is a fixture choice, not a production renewal strategy. Creating a certificate does not make other clients trust it; the explicit root pool is what establishes trust for these requests.

package main

import (
 "crypto/ed25519"
 "crypto/rand"
 "crypto/tls"
 "crypto/x509"
 "errors"
 "fmt"
 "io"
 "log"
 "math/big"
 "net/http"
 "net/http/httptest"
 "runtime"
 "strings"
 "sync/atomic"
 "time"
)

func must(err error) {
 if err != nil {
 panic(err)
 }
}

func check(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func leaf(name string, serial int64, ca *x509.Certificate, signer ed25519.PrivateKey) tls.Certificate {
 public, private, err := ed25519.GenerateKey(rand.Reader)
 must(err)
 now := time.Now()
 template := &x509.Certificate{
 SerialNumber: big.NewInt(serial), DNSNames: []string{name},
 NotBefore: now.Add(-time.Minute), NotAfter: now.Add(time.Hour),
 KeyUsage: x509.KeyUsageDigitalSignature,
 ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
 }
 der, err := x509.CreateCertificate(rand.Reader, template, ca, public, signer)
 must(err)
 certificate, err := x509.ParseCertificate(der)
 must(err)
 return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: private, Leaf: certificate}
}

func main() {
 public, private, err := ed25519.GenerateKey(rand.Reader)
 must(err)
 now := time.Now()
 template := &x509.Certificate{
 SerialNumber: big.NewInt(1), IsCA: true, BasicConstraintsValid: true,
 NotBefore: now.Add(-time.Minute), NotAfter: now.Add(time.Hour),
 KeyUsage: x509.KeyUsageCertSign,
 }
 der, err := x509.CreateCertificate(rand.Reader, template, template, public, private)
 must(err)
 ca, err := x509.ParseCertificate(der)
 must(err)
 roots := x509.NewCertPool()
 roots.AddCert(ca)
 alpha := leaf("alpha.example.test", 2, ca, private)
 beta := leaf("beta.example.test", 3, ca, private)
 certificates := map[string]*tls.Certificate{
 "alpha.example.test": &alpha, "beta.example.test": &beta,
 }

 var handled, rejected atomic.Int64
 server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 handled.Add(1)
 fmt.Fprintf(w, "%s|%s", r.TLS.ServerName, r.Host)
 }))
 server.Config.ErrorLog = log.New(io.Discard, "", 0)
 server.TLS = &tls.Config{
 MinVersion: tls.VersionTLS13, MaxVersion: tls.VersionTLS13,
 Certificates: []tls.Certificate{alpha}, SessionTicketsDisabled: true,
 GetCertificate: func(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
 certificate, ok := certificates[strings.ToLower(hello.ServerName)]
 if !ok {
 rejected.Add(1)
 return nil, fmt.Errorf("unknown fixture SNI")
 }
 return certificate, nil
 },
 }
 server.StartTLS()
 defer server.Close()

 request := func(name, host string) (string, error) {
 config := &tls.Config{RootCAs: roots, ServerName: name, MinVersion: tls.VersionTLS13}
 transport := &http.Transport{TLSClientConfig: config, DisableKeepAlives: true}
 defer transport.CloseIdleConnections()
 client := &http.Client{Transport: transport, Timeout: 2 * time.Second}
 request, err := http.NewRequest(http.MethodGet, server.URL, nil)
 must(err)
 request.Host = host
 response, err := client.Do(request)
 if err != nil {
 return "", err
 }
 defer response.Body.Close()
 body, err := io.ReadAll(response.Body)
 if err != nil {
 return "", err
 }
 check(response.StatusCode == 200, "unexpected HTTP status")
 check(response.TLS.Version == tls.VersionTLS13 && len(response.TLS.VerifiedChains) > 0, "verified TLS 1.3 required")
 check(string(body) == name+"|"+host, "handler must observe separate SNI and Host")
 certificate := response.TLS.PeerCertificates[0]
 must(certificate.VerifyHostname(name))
 return certificate.DNSNames[0], nil
 }

 for _, name := range []string{"alpha.example.test", "beta.example.test"} {
 selected, err := request(name, name)
 must(err)
 check(selected == name, "SNI must select its matching certificate")
 fmt.Println("SNI", name, "selected", selected)
 }
 selected, err := request("ALPHA.EXAMPLE.TEST", "alpha.example.test")
 must(err)
 check(selected == "alpha.example.test", "DNS name selection must be case-insensitive")
 selected, err = request("alpha.example.test", "beta.example.test")
 must(err)
 check(selected == "alpha.example.test", "HTTP Host must not change the TLS certificate")
 _, err = request("missing.example.test", "missing.example.test")
 check(err != nil && rejected.Load() == 1, "unknown SNI must be rejected by fixture policy")
 _, err = request("", "alpha.example.test")
 var nameError x509.HostnameError
 check(errors.As(err, &nameError), "without DNS SNI, default certificate must fail IP identity")
 check(handled.Load() == 4, "only four verified requests may reach the handler")
 fmt.Println("uppercase name selects alpha; Host override keeps alpha; unknown SNI and IP-name mismatch rejected")
 fmt.Println(runtime.Version())
}

Alpha and beta select their respective certificates, and an uppercase alpha name still selects alpha. The client performs normal chain and hostname checks; the example additionally inspects the selected certificate. Four verified requests reach the echo handler, including the deliberate request with alpha SNI but beta Host. That request still authenticates alpha and demonstrates independent fields, not permission to serve another tenant.

An unknown SNI is rejected by this fixture’s exact-name policy. With ServerName empty, the IP URL leads to no DNS SNI; the configured default alpha certificate cannot authenticate that IP, so hostname verification fails even though Host says alpha. Neither negative case reaches HTTP. The echo handler intentionally exposes the two fields for inspection and is not a production virtual-host router.

Retained evidence establishes compilation on Go 1.24.4, with handshake execution blocked; the earlier macOS Go 1.27.0 success statement lacks a matching retained run here. No TLS exercise ran in this source review. Keys remain fixture-local by inspection. ECH, ALPN negotiation, wildcard or internationalized names, HTTP/2 coalescing, resumption and real edges need separate tests.

Sources

RFC 6066: TLS Extensions, server_name, hostname syntax, and server selection, checked 2026-09-09.
RFC 8446: TLS 1.3, encrypted handshake messages after ServerHello, checked 2026-09-09.
Go crypto/tls, ServerName, GetCertificate, and explicit ECH configuration, checked 2026-09-09.
Go crypto/x509, test certificate creation and SAN-based hostname verification, checked 2026-09-09.
Go net/http/httptest, temporary loopback TLS servers, checked 2026-09-09.
RFC 7301: ALPN, application-protocol selection distinct from server-name indication, checked 2026-10-18.

Server-Timing exposes named measurements to the browser

Sat, 07 Nov 2026 00:00:00 +0000

The Server-Timing response header lets a server attach named measurements to a response. Browser developer tools and performance APIs can then associate those measurements with the resource request. It is useful when the network waterfall shows a long wait but cannot explain whether backend work involved a database call, rendering, or another measured operation. The header supplies server-selected observations, not an independent browser measurement of those operations.

The duration syntax is db;dur=42, render;dur=81, with milliseconds the recommended unit. The browser parses the number without verifying its unit. A bare db=42 is not the duration syntax. Each metric has a name and can include duration and description parameters. Use stable names that readers can interpret, and derive values from actual instrumentation in production. A description can clarify the measurement, but should not contain raw queries, identifiers, or internal details that the response should not disclose.

Match the number to its interval

Server measurements do not necessarily add up to the browser’s request duration. They may cover nested operations, overlapping work, or only part of the handling path. Adding database and rendering time is valid only when their definitions justify that addition. Network transport, queuing outside the measured interval, caching, and client-side processing can explain differences without either clock being wrong.

The header also does not create a distributed trace. It names measurements on one response without automatically supplying parent-child relationships, propagation, or collector storage. Use it to expose a small set of useful request measurements, then follow trace or log correlation when a deeper explanation is needed. When responses pass through a cache or proxy, inspect the delivered header and establish which component produced the numbers.

When it breaks

For cross-origin resources, reading a response header in JavaScript and receiving detailed Resource Timing data are separate permissions. CORS allows the response to be read, and Access-Control-Expose-Headers can expose Server-Timing to response.headers. Timing-Allow-Origin controls the corresponding detailed timing exposure. Secure-context and browser privacy rules can further restrict timing exposure. A successful header read therefore does not prove that PerformanceResourceTiming.serverTiming will contain the same measurements.

Another failure is confusing a parser demonstration with measured performance. Sending a fixed duration establishes what the browser parses, not how long the handler ran. Likewise, an empty timing array does not prove that the server did no work. First check origin permissions and the actual wire header, then inspect measurement placement. Keep public names and descriptions appropriate for their audience; an operationally useful timing hint should not accidentally reveal sensitive backend structure.

The same synthetic Server-Timing values can be readable as a header while absent from browser timing entries. The source’s historical Chrome 152 comparison checks distinct exposure permissions; it is not backend performance measurement.

Check it quickly

Save this as server-timing.py and run python3 server-timing.py --serve. The earlier record used Python 3.14.4; this review did not repeat the socket or browser tests. It starts two loopback HTTP origins on separate temporary ports and prints the page URL to open. The page fetches two resources and displays the header values alongside the browser’s parsed timing entries. Press Control-C to close both servers. Without --serve, the script checks response headers and exits without running a browser.

import json
import sys
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Event, Thread
from urllib.request import Request, urlopen

timing = "db;dur=42, render;dur=81"


class API(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def do_GET(self):
 body = b"synthetic timing fixture\n"
 self.send_response(200)
 self.send_header("Content-Type", "text/plain; charset=utf-8")
 self.send_header("Content-Length", str(len(body)))
 self.send_header("Cache-Control", "no-store")
 self.send_header("Access-Control-Allow-Origin", app_origin)
 self.send_header("Access-Control-Expose-Headers", "Server-Timing")
 self.send_header("Server-Timing", timing)
 if self.path.split("?", 1)[0] == "/allow":
 self.send_header("Timing-Allow-Origin", app_origin)
 self.end_headers()
 self.wfile.write(body)


class App(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def do_GET(self):
 body = ("""<!doctype html><meta charset="utf-8"><title>Server-Timing lab</title>
<h1>Cross-origin Server-Timing</h1><pre id="result">Running...</pre>
<script>
const api = """ + json.dumps(api_origin) + """;
async function run() {
 performance.clearResourceTimings();
 const results = {};
 for (const path of ['deny', 'allow']) {
 const url = api + '/' + path;
 const response = await fetch(url, {cache:'no-store'});
 await response.text();
 await new Promise(resolve => requestAnimationFrame(resolve));
 const entry = performance.getEntriesByName(url).at(-1);
 results[path] = {header:response.headers.get('Server-Timing'),
 timing:entry.serverTiming.map(item => ({name:item.name, duration:item.duration}))};
 }
 window.labResult = results;
 document.getElementById('result').textContent = JSON.stringify(results, null, 2);
}
run().catch(error => {window.labError=String(error); document.getElementById('result').textContent=String(error);});
</script>""").encode()
 self.send_response(200)
 self.send_header("Content-Type", "text/html; charset=utf-8")
 self.send_header("Content-Length", str(len(body)))
 self.end_headers()
 self.wfile.write(body)


api = HTTPServer(("127.0.0.1", 0), API)
app = HTTPServer(("127.0.0.1", 0), App)
api_origin = f"http://127.0.0.1:{api.server_port}"
app_origin = f"http://127.0.0.1:{app.server_port}"
threads = [Thread(target=server.serve_forever, daemon=True) for server in (api, app)]
for thread in threads:
 thread.start()
try:
 for path in ("deny", "allow"):
 with urlopen(Request(api_origin + "/" + path, headers={"Origin": app_origin}), timeout=5) as response:
 assert response.headers["Server-Timing"] == timing
 assert response.headers["Access-Control-Allow-Origin"] == app_origin
 assert response.headers["Access-Control-Expose-Headers"] == "Server-Timing"
 assert response.headers.get("Timing-Allow-Origin") == (app_origin if path == "allow" else None)
 response.read()
 print(json.dumps({"app_url": app_origin, "api_url": api_origin,
 "wire_checks": "PASS", "durations_are_synthetic": True}), flush=True)
 if "--serve" in sys.argv:
 Event().wait()
except KeyboardInterrupt:
 pass
finally:
 for server in (api, app):
 server.shutdown()
 server.server_close()
 for thread in threads:
 thread.join(timeout=5)

Both responses allow the page’s origin through CORS, expose Server-Timing, and carry the same synthetic database and rendering durations. Only /allow adds Timing-Allow-Origin. In the historical Chrome 152 observation, both displayed header strings were db;dur=42, render;dur=81; the denied timing array was empty, while the allowed array contained db with duration 42 and render with duration 81. This browser result was checked separately from the Python wire assertions.

The fixed numbers intentionally test syntax and visibility rather than elapsed backend time. This local example does not verify another browser, production proxies, cached responses, or a real database measurement. To deploy the idea, measure well-defined intervals, emit the header before the response headers are committed, and repeat the check through the actual delivery path. Explain the timing names and overlap rules so readers can interpret the numbers without inventing an additive breakdown the server never promised.

Sources

W3C Server Timing Working Draft, header parameters and PerformanceServerTiming exposure, checked 2026-10-15.
MDN Server-Timing, duration syntax, browser use, and disclosure considerations, checked 2026-09-13.
MDN Timing-Allow-Origin, permission for detailed cross-origin resource timing, checked 2026-09-13.
WHATWG Fetch, separate header exposure, timing permission, and secure-context processing, checked 2026-10-15.

Web · XB Field Notes

nginx chooses the first matching regex, not a specificity score

When it breaks

Check it quickly

Make overlaps visible in review

Sources

Frame policy names who may embed your page

Put each restriction on the right response

When it breaks

Check it quickly

Sources

SSRF makes the server open the door

Identify the authority being delegated

Parse, resolve, and preserve the checked destination

When it breaks

Check it quickly

Sources

CSRF asks the browser where the request came from

Cookies travel without asking

A token ties the action to a page

SameSite adds a browser-side guard

Read Origin as a source signal

When it breaks

Check it quickly

Further reading

Sources

Enable gzip explicitly, then verify negotiation and decoded bytes

When it breaks

Check it quickly

Tune the policy using representative responses

Further reading

Sources

A request limit protects the budget named by its key

When it breaks

Check it quickly

Define identity, scope, and observation together

Sources

A reverse proxy needs an explicit header trust boundary

When it breaks

Check it quickly

Connect network access and header interpretation

Further reading

Sources

nginx matches normalized paths and can select a location again

When it breaks

Check it quickly

Verify the path contract across the full request chain

Sources

A 404 page answers honestly

Choose the status before rendering

Make the body independent and safe

When it breaks

Check it quickly

Further reading

Sources

Sitemaps guide crawler discovery

Generate from the publication inventory

When it breaks

Check it quickly

Sources

Canonical URLs name a preferred representation

Choose identity before writing the tag

When it breaks

Check it quickly

Sources

Well-known endpoints provide a discovery convention

Follow the suffix’s specification

When it breaks

Check it quickly

Sources

A charset tells the reader how the bytes become characters

The format determines the rules

When it breaks

Check it quickly

Sources

CORS preflight depends on the request shape

Match each permission to its stage

When it breaks

Check it quickly

Sources