Deployment · XB Field Notes

Change windows fit people, not servers

Thu, 29 Jul 2027 00:00:00 +0000

A change that races to zero traffic but lands when nobody is watching trades one risk for another. The best window is one where the operators responsible can actually attend.

Traffic is only part of the math

Deploying at the absolute quietest hour minimizes how many users feel the change, but it collides with the people problem: the operator is most likely asleep, unreachable, or already covering another incident. If the rollout breaks at 3 a.m. and nobody is watching, a silent failure can run for hours before a human reads it. Least-traffic is a good starting estimate, but it is a traffic answer, not a risk answer. Balance reduced exposure against the ability to react, and prefer a window where you can actually observe the result.

Watch the human clock

The window has to fit the humans, not just the chart. Consider who must notice a broken deploy: the SRE on call, the feature owner, the support team, and schedule their attention into the slot. Also watch what else is scheduled — a parallel release, a job run, an adjacent change — that could tangle with yours and turn a clean rollout into a confusing blame-fest. A slot where the people who must read the result are awake, reachable, and not already overloaded beats a quieter one where nobody is watching.

Let the window be a habit

A recurring staffed window can help coordinate risky changes and make observation predictable. It is not a universal requirement to batch every release: small, automated, independently reversible changes may use a continuous-delivery workflow. Choose the cadence from actual risk and response coverage. Automate builds, smoke tests, and canaries, while preserving an accountable owner and a tested recovery path. A calendar slot should support those safeguards, not replace them.

When it breaks

An urgent fix arrives outside the window and there is no emergency lane. The team either waits while the incident burns, or someone pushes straight to production with no watcher and no rollback note. Without a documented break-glass path — who approves, who watches, what gets written down — every exception becomes a secret deploy. Rehearse an explicit emergency procedure alongside the normal schedule.

A second failure is the stampede inside the window. Four services land in the same two hours, their signals blend together, and when the error rate climbs nobody can name the culprit. Rollback becomes a negotiation instead of a button press. Serialize changes that share failure signals or dependencies, and require every entry to name its owner, signal, and way back.

A third failure is the cross-timezone misread. The window says Wednesday 10:00 in Berlin, but a teammate in Los Angeles reads it as local time and ships while responders sleep. Daylight saving quietly shifts the slot by another hour. Store every window with its zone, render it locally, and confirm a human is present before counting a slot as staffed.

One instant, different human clocks. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

Use Python 3.9 or newer with IANA time-zone data available to zoneinfo; the module does not bundle that data. Save the following as change_window_check.py and run python3 change_window_check.py. Run without -O, -OO, or a nonzero PYTHONOPTIMIZE setting so the assertions execute. It uses only the standard library, treats every window as timezone-aware, and decides between allow and needs-approval. The table holds same-day Wednesday slots in Berlin and New York. Freeze dates use an explicitly chosen UTC policy day, regardless of the timezone used to express the input instant. The bad samples are negative controls.

from datetime import datetime, time
from zoneinfo import ZoneInfo

WINDOWS = [
 {"name": "berlin-wed", "weekday": 2, "start": "09:00", "end": "11:00", "tz": "Europe/Berlin"},
 {"name": "newyork-wed", "weekday": 2, "start": "09:00", "end": "11:00", "tz": "America/New_York"},
]
FREEZE_DATES = frozenset({"2027-12-24"})
FREEZE_ZONE = ZoneInfo("UTC") # Organization-wide policy, independent of input representation.


def _hhmm(value):
 hour, minute = map(int, value.split(":"))
 return time(hour, minute)


def decide(moment, windows=WINDOWS, freeze=FREEZE_DATES):
 if moment.tzinfo is None or moment.utcoffset() is None:
 return "needs-approval", "naive datetime carries no zone; attach a ZoneInfo first"
 freeze_day = moment.astimezone(FREEZE_ZONE).date()
 if freeze_day.isoformat() in freeze:
 return "needs-approval", f"freeze date {freeze_day} in UTC; emergency path only"
 for window in windows:
 local = moment.astimezone(ZoneInfo(window["tz"]))
 if local.weekday() == window["weekday"] and _hhmm(window["start"]) <= local.time() < _hhmm(window["end"]):
 return "allow", f"inside {window['name']} ({local:%Y-%m-%d %H:%M %Z})"
 return "needs-approval", f"outside all windows ({moment:%Y-%m-%d %H:%M %Z})"


def check(actual, expected, label):
 status, reason = actual
 assert status == expected, f"{label}: got {status} ({reason})"
 print(f"{label}: {status} -- {reason}")


def main():
 berlin = ZoneInfo("Europe/Berlin")
 newyork = ZoneInfo("America/New_York")
 good_berlin = datetime(2027, 8, 4, 9, 30, tzinfo=berlin)
 good_ny = datetime(2027, 8, 4, 9, 30, tzinfo=newyork)
 check(decide(good_berlin), "allow", "good/berlin-wed-morning")
 check(decide(good_ny), "allow", "good/newyork-wed-morning")
 # Same UTC instant, different local reading: 07:30 UTC is 09:30 in Berlin but 03:30 in New York.
 same_instant = datetime(2027, 8, 4, 7, 30, tzinfo=ZoneInfo("UTC"))
 check(decide(same_instant), "allow", "good/same-instant-via-berlin")
 evening = datetime(2027, 8, 4, 22, 10, tzinfo=berlin)
 check(decide(evening), "needs-approval", "bad/wednesday-night")
 wrong_day = datetime(2027, 8, 5, 9, 30, tzinfo=berlin)
 check(decide(wrong_day), "needs-approval", "bad/thursday-morning")
 frozen = datetime(2027, 12, 24, 9, 30, tzinfo=berlin)
 check(decide(frozen), "needs-approval", "bad/freeze-date")
 naive = datetime(2027, 8, 4, 9, 30)
 check(decide(naive), "needs-approval", "bad/naive-datetime")
 misread = datetime(2027, 8, 4, 9, 30, tzinfo=ZoneInfo("America/Los_Angeles"))
 check(decide(misread), "needs-approval", "bad/timezone-misread")
 check(decide(datetime(2027, 8, 4, 9, 0, tzinfo=berlin)), "allow", "good/start-inclusive")
 check(decide(datetime(2027, 8, 4, 11, 0, tzinfo=berlin)), "needs-approval", "bad/end-exclusive")
 winter = datetime(2027, 1, 6, 8, 30, tzinfo=ZoneInfo("UTC"))
 check(decide(winter), "allow", "good/winter-berlin")
 # This Wednesday would otherwise pass: the freeze test cannot pass by accident.
 frozen_wednesday = frozenset({"2027-08-04"})
 shifted = same_instant.astimezone(ZoneInfo("Etc/GMT+12"))
 assert shifted == same_instant and shifted.date() != same_instant.date()
 check(decide(same_instant, freeze=frozen_wednesday), "needs-approval", "bad/UTC-freeze")
 check(decide(shifted, freeze=frozen_wednesday), "needs-approval", "bad/same-freeze-other-zone")
 assert decide(same_instant, freeze=frozen_wednesday) == decide(shifted, freeze=frozen_wednesday)
 print("PASS: 5 allows and 8 approvals; policy dates do not depend on caller timezone")


if __name__ == "__main__":
 main()

The script prints five allow results and eight needs-approval results. Window starts are inclusive and ends exclusive. The summer and winter UTC cases reach Berlin at 09:30 despite different offsets. A Wednesday freeze rejects an otherwise eligible instant both in UTC and in a representation whose local date is the previous day. Checking the input date directly would incorrectly allow that second representation.

These revised assertions passed with Python 3.12.14. Allow means only that this schedule predicate passed; it grants no deployment authorization and verifies no staffing. This same-day fixture does not handle overnight windows or validate ambiguous local-time construction. Convert a known instant, then separately check approvals, the on-call roster, observation time, and recovery readiness.

Sources

Google SRE: release engineering, release as an engineered practice, checked 2026-10-15.
12Factor: build release run, separating build from release, checked 2026-10-15.
Python datetime, aware timestamps and conversion between timezone representations, checked 2026-10-15.
Python zoneinfo, IANA timezone rules and daylight-saving transitions, checked 2026-10-15.

Rolling updates replace pods in small waves

Mon, 07 Jun 2027 00:00:00 +0000

A rolling update is a controlled replacement, not a promise of uninterrupted service. Its useful property is that the controller can retain old replicas while introducing new ones. Whether users notice the change also depends on spare capacity, meaningful health checks, and shutdown behavior. This article tunes that process; the companion strategy article asks whether versions should overlap at all.

Ten plus two is a target of twelve; ten minus one is a planning budget of nine. Account separately for terminating Pods, stable readiness and real request behavior.

The surge and the gap

With ten desired replicas, maxSurge: 2 permits a rollout target of twelve, not eleven. maxUnavailable: 1 gives the controller an availability budget of nine. It is not a limit of one old Pod deleted per wave: several old replicas can be removed once sufficient replacements are available. Terminating Pods can remain and consume resources beyond the surge target, so twelve is not a hard count of every Pod or process.

Percentages need rounding: surge rounds upward, unavailable downward. For three replicas, the default 25% yields one surge and zero unavailable. Kubernetes 1.34.0 has a boundary exception: zero surge plus a nonzero unavailable percentage that rounds to zero resolves to one unavailable replica, so replacement can progress. Do not infer a zero-disruption policy from that rounded zero.

Neither knob reserves resources. Compare requests with schedulable capacity, quota, placement constraints, and the shutdown tail before choosing faster replacement. A budget that looks affordable on paper may still leave every new Pod Pending.

Health gates replacement

Check that a readiness probe is actually configured. Readiness should answer whether this instance can serve its intended traffic. A listening port alone may miss application failure; checking a required backend can help, but a shared dependency failure can also withdraw every replica. Design that response deliberately rather than blindly weakening a failing probe.

minReadySeconds adds a stable-ready interval before a new Pod counts as available. It does not delay Service traffic until that interval finishes. A startup probe can protect slow initialization from premature liveness restarts. None of these checks proves correct business behavior or sustained throughput.

Shutdown needs its own test. Endpoint changes, connection draining, application signal handling, and the termination grace period must work together. A green readiness endpoint cannot prevent an application from dropping an in-flight request when it exits.

Choose a deliberate pace

One surge and zero unavailable is a useful starting point for a small fleet with spare capacity; it prevents intentional availability reduction by the rollout, not node failures or request errors. A PodDisruptionBudget does not constrain a Deployment’s own rolling upgrade.

Manual pause is useful for inspection, but it does not reserve an exact one-Pod canary or a fixed traffic percentage: reconciliation may advance before the pause takes effect. Use separately controlled canary capacity and routing when that exposure limit matters. Define error-rate, latency, restart, and business-check thresholds before starting, with an observation period after completion.

When it breaks

A stuck rollout needs evidence from the failing layer. Scheduling events point to capacity or placement; image-pull events point to registry access or image references; container logs and probe events distinguish crashes from readiness failures. Do not change probes until you know whether they are detecting a real fault.

A bad new version may consume the configured unavailable allowance before progress stalls. progressDeadlineSeconds reports stalled progress; it neither automatically rolls back nor freezes all reconciliation. Alert on the condition and decide whether to repair or revert. A completed rollout still needs a bake period for memory growth, connection exhaustion, and errors under realistic traffic. Retain usable revision history and the previous image through that period.

Check it quickly

Save this standard-library exercise as rounding_window.py and run python3 rounding_window.py. It checks percentage conversion using integer arithmetic, including a small fleet, an exact boundary, and 100%. It is not a Deployment controller simulation or a complete API validator. Its 0–100% input range is an exercise limit, not a maxSurge API restriction.

"""Check percentage rounding only; no Kubernetes API calls."""


def round_percent(replicas, percent, *, up):
 if type(replicas) is not int or replicas <= 0:
 raise ValueError("this exercise requires positive integer replicas")
 if type(percent) is not int or not 0 <= percent <= 100:
 raise ValueError("this exercise accepts integer percentages 0..100")
 numerator = replicas * percent
 return (numerator + 99) // 100 if up else numerator // 100


def main():
 cases = [(1, 1, 0, 1), (3, 25, 0, 1), (4, 50, 2, 2),
 (10, 20, 2, 2), (10, 10, 1, 1), (4, 100, 4, 4)]
 for replicas, percent, down, up in cases:
 assert round_percent(replicas, percent, up=False) == down
 assert round_percent(replicas, percent, up=True) == up
 print(f"replicas={replicas}, percent={percent}: down={down}, up={up}")
 print("PASS: percentage rounding only; no cluster exercised")


if __name__ == "__main__":
 main()

Expect six case lines and a final arithmetic-only PASS. Three replicas at 25% produce zero rounded down and one rounded up; ten at 20% produce two either way. The assertions were executed with Python 3.12.14 in Linux during this revision. They establish arithmetic, not service availability.

Before applying these numbers, select the intended test context and namespace and confirm the actual Deployment strategy. With an existing api Deployment and app=api Pods, kubectl rollout status deployment/api --timeout=120s observes convergence, while kubectl get pods -l app=api -w watches replacements in a separate terminal. A client timeout does not cancel the rollout. If releases overlap, use --revision=N to follow the intended revision. Stop the watch with Ctrl-C.

Those cluster commands were not executed here. Collect Pod events, logs, and Deployment status during a staging upgrade, together with continuous requests through the real Service or ingress. Record failures and latency during shutdown. This is the integration evidence still needed; a successful controller status alone cannot establish zero downtime.

Sources

Kubernetes Deployments, rollout budgets, readiness intervals, pause, and progress conditions, checked 2026-10-15.
Kubernetes probes, startup, liveness, and readiness responsibilities, checked 2026-10-15.
Kubernetes Pod lifecycle, termination and connection draining, checked 2026-10-15.
Kubernetes resource management, requests and scheduling capacity, checked 2026-10-15.
Kubernetes disruptions, limits of PodDisruptionBudgets during rolling upgrades, checked 2026-10-15.
kubectl reference, rollout status and timeout behavior, checked 2026-10-15.
Kubernetes controller rounding implementation, versioned ResolveFenceposts behavior when both rounded budgets are zero, checked 2026-10-15.

Promote a verified artifact through environments

Wed, 17 Feb 2027 00:00:00 +0000

A staging test establishes facts about the artifact and environment that actually ran. Building again before production creates another output whose identity must be checked or whose behavior must be revalidated. Promotion carries an already verified artifact to the next environment and records that decision. It removes one source of uncertainty, but it does not make staging and production equivalent or eliminate the need to verify the resulting deployment.

Name the unit that moves

Use an immutable package identifier or content digest to identify the build. A moving tag, branch name, or source commit alone does not uniquely establish the deployed bytes. Toolchains, dependency resolution, embedded settings, or packaging can change between builds of the same source. A reproducible rebuild might yield identical bytes; compare the result before treating it as the previously tested artifact.

Keep the build stage separate from the release stage. A release combines a particular artifact with the configuration required for a deployment. Runtime then executes that combination against actual dependencies and data. Record the artifact identity and a suitable configuration revision, without exposing secrets. If frontend settings are compiled into JavaScript during packaging, changing them creates different bytes and therefore a different artifact requiring validation.

When it breaks

A pipeline can say it promotes a release while a deployment script quietly recompiles it. Another pipeline can download the correct archive and overwrite a configuration file inside it. Both change the tested input. Decide which settings belong outside the immutable payload and make the deployment read them through the application’s existing configuration interface. Do not assume a checksum checked before a later mutation still describes the running files.

The same artifact can behave differently with a changed database schema, feature flag, dependency version, permission, resource limit, or network path. Promotion establishes continuity of bytes, not continuity of the entire execution context. Validate important production behavior after deployment and compare the relevant environment differences when staging results do not reproduce.

Rollback also has prerequisites. An older executable may not understand newly written records or a completed destructive schema migration. Returning a pointer to an earlier artifact cannot undo those external effects. Keep rollback instructions tied to data compatibility, configuration, and dependency state. When reversal is unavailable, plan the necessary forward repair rather than promising that selecting the old image always restores service.

Promotion keeps the tested artifact identity. It does not freeze data, dependencies or configuration, and selecting an old artifact cannot undo incompatible data changes. The original EN/ZH local file-copy and configuration exercises passed in the recorded 2026-10-17 run; production deployment and other real release-system behavior remain untested.

Check it quickly

Recorded check, 2026-10-17: both complete original EN/ZH Python fences ran once on macOS 26.5.1 arm64 with Python 3.14.4, -I -B, optimization disabled and all three original assert statements retained. Each produced the exact expected JSON: matching artifact digests, local-build-41 with the two distinct tier labels and configuration digests, and rejection of the deliberately modified copy. Both exited zero with empty stderr; generated files, sockets, threads and owned temporary work were cleaned up.

Each language case recorded four successful loopback connections and four application-level GET/status events: /artifact then / for staging-local, followed by /artifact then / for production-simulation, all 200, matched to two successive owned listener generations. These observations are not packet capture or independent wire-status measurement. This remains a synthetic local file-copy and configuration exercise: no production deployment, registry, secrets, migration, rollback, signing, reproducible-build pipeline or real release-system enforcement was tested.

Save this as promote.py and run python3 promote.py. The script uses Python’s standard library, temporary directories, and HTTP servers bound only to loopback. It writes one tiny application artifact, records its digest, and copies it to two local environments. The label production-simulation names a temporary directory; no external deployment occurs.

from hashlib import sha256
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from tempfile import TemporaryDirectory
from threading import Thread
from urllib.request import ProxyHandler, build_opener
import json
import runpy
import shutil

source = 'def render(config):\n return {"build": "local-build-41", "tier": config["tier"]}\n'
digest = sha256(source.encode()).hexdigest()


def verify(path):
 if sha256(path.read_bytes()).hexdigest() != digest:
 raise ValueError('artifact changed')


def handler(path, config):
 verify(path)
 render = runpy.run_path(str(path))['render']

 class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 data = path.read_bytes() if self.path == '/artifact' else json.dumps(render(config)).encode()
 self.send_response(200)
 self.send_header('Content-Length', str(len(data)))
 self.end_headers()
 self.wfile.write(data)

 def log_message(self, *args):
 pass

 return Handler


with TemporaryDirectory() as directory:
 root = Path(directory)
 artifact = root / 'build.py'
 artifact.write_text(source)
 opener = build_opener(ProxyHandler({}))
 observed = []
 for tier in ('staging-local', 'production-simulation'):
 destination = root / tier
 destination.mkdir()
 deployed = destination / 'app.py'
 shutil.copyfile(artifact, deployed)
 config = {'tier': tier}
 server = ThreadingHTTPServer(('127.0.0.1', 0), handler(deployed, config))
 thread = Thread(target=server.serve_forever)
 thread.start()
 try:
 origin = f'http://127.0.0.1:{server.server_port}'
 with opener.open(origin + '/artifact', timeout=5) as response:
 received_digest = sha256(response.read()).hexdigest()
 with opener.open(origin + '/', timeout=5) as response:
 rendered = json.load(response)
 assert received_digest == digest
 assert rendered == {'build': 'local-build-41', 'tier': tier}
 observed.append({'artifact_sha256': received_digest, 'response': rendered,
 'config_sha256': sha256(json.dumps(config, sort_keys=True).encode()).hexdigest()})
 finally:
 server.shutdown()
 thread.join()
 server.server_close()
 deployed.write_text(source + '# rebuilt\n')
 try:
 verify(deployed)
 except ValueError:
 changed_rejected = True
 else:
 changed_rejected = False
 assert changed_rejected
 print(json.dumps({'environments': observed, 'changed_copy_rejected': changed_rejected}))

Before starting each server, the script verifies the copied file against the original digest and loads its application function. Fetching /artifact returns the deployed bytes; fetching / executes that function with the environment’s separate configuration. Both HTTP downloads have the same artifact hash, and both responses name local-build-41.

The response’s tier differs between environments, and the recorded configuration hashes differ as well. Those configurations contain only harmless labels. The result demonstrates how identical application bytes can intentionally produce different behavior. After the servers stop, a changed copy is rejected by the same digest check, so a silent rebuild cannot pass this particular identity gate.

The fixture does not model a registry, authentication, database migration, production traffic, or rollback. It establishes the local transfer, validation, and configuration boundary using actual file copies and HTTP responses. It cannot establish that a real deployment system enforces the same rules.

Verify the release that actually runs

Attach the promotion decision to the artifact and configuration revision, then check the deployed process or served files instead of relying solely on a successful upload. Retain the approved artifact for the required recovery window. If a later step generates a different package, give that output a new identity and its own relevant validation. A useful promotion record answers which bytes moved, what configuration accompanied them, and what behavior was observed after the move.

Sources

The Twelve-Factor App: build, release, run, separate build artifacts from releases and execution, checked 2026-09-13.
The Twelve-Factor App: config, configuration that varies between deploys, checked 2026-09-13.

Feature flags decouple release from exposure

Tue, 09 Feb 2027 00:00:00 +0000

Deploying code and exposing it are different acts. A feature flag can select a runtime path after the code is deployed. This supports gradual exposure and fast disablement, provided the old path remains compatible and the configuration reaches the processes that evaluate it. The flag controls a decision; it does not reverse everything the deployment or the new behavior has already done.

A flag chooses later work; it does not reverse completed work. The cohort count belongs to a fixed local fixture and does not establish request share or distributed delivery.

Ship behind a defined gate

A release flag commonly defaults to the established path while a new implementation is tested. Deploy-time behavior still changes: imports, startup hooks, schema migrations, dependencies, and resource use may run outside the guarded branch. An off flag therefore cannot promise zero user-visible effects. Test the deployed artifact with the flag off as well as on, including startup and compatibility with existing data.

Turning off the flag can stop later requests from entering the new path. It does not undo completed writes, messages sent, migrations, or work already in flight. Keep data formats compatible with both branches during the rollback window and define any repair required after partial execution. A kill switch is useful only when its propagation delay and actual stopping boundary match the incident response plan.

Choose defaults by meaning

For an unfinished release feature, an unknown configuration may safely select the old path. That is a decision about this flag’s semantics, not a universal rule that false is always safe. Disabling a protective check or reversing a flag named disable-validation can have the opposite effect. Specify fallback behavior, configuration freshness, and what happens when the service cannot evaluate the flag.

Evaluate a release decision once at the appropriate operation boundary and reuse it within that operation. Multiple services or separate requests need an explicit way to carry compatible decisions; local evaluation alone does not create global consistency. Keep authorization enforced independently. A client-side display flag must not become permission to perform a protected server-side action.

Ramp exposure with stable identities

Hashing a stable user identifier with a stable flag key gives repeatable cohorts. Changing the identifier, salt, or bucketing algorithm can reshuffle users even when the percentage stays unchanged. A percentage of users is also not necessarily the same percentage of requests: a small cohort can generate disproportionate traffic. Record the unit of assignment before interpreting rollout metrics.

A kill switch should override the cohort selection and have a tested delivery path. SDK caches and configuration outages can leave some instances on older values. Observe errors and latency by actual evaluated variant, rather than treating a control-panel change as proof that every request has switched. Choose a rollout pace from the service’s risk and observation delay, not an arbitrary universal schedule.

When it breaks

Temporary release flags can become permanent branches when nobody owns their removal. Give each one an owner and a retirement condition, and remove obsolete paths after the rollback window closes. Operational switches and entitlement controls may deliberately remain long-lived; they need ongoing tests and ownership rather than an unconditional short expiry.

Contradictory flags can create combinations nobody tested. Centralize the decision rule where practical, define precedence, and exercise both the fallback and enabled paths. Logs should record enough nonsecret evaluation context to diagnose disagreement, while avoiding unbounded retention of real user identifiers. The example below uses synthetic names only.

Check it quickly

Save the block as flag_check.py and run it with Python 3.10 or newer. The original fixture reports Python 3.14.4; the retained 2026-10-15 review reran its decision logic with Python 3.12.14. It models one percentage release flag, a kill switch, stable user bucketing, and an in-memory audit list. The percentage is assumed to be a validated value from zero through one hundred.

import hashlib


class Flag:
 """One percentage flag with kill switch, default-closed, and audit log."""

 def __init__(self, key, percent):
 self.key = key
 self.percent = percent
 self.killed = False
 self.audit = []

 def _bucket(self, user):
 digest = hashlib.sha256(f"{self.key}:{user}".encode()).hexdigest()
 return int(digest, 16) % 100

 def evaluate(self, user):
 if self.killed:
 decision, reason = False, "killed"
 elif not user:
 decision, reason = False, "default-closed"
 else:
 hit = self._bucket(user) < self.percent
 decision, reason = hit, ("in-bucket" if hit else "out-of-bucket")
 self.audit.append({"user": user, "decision": decision, "reason": reason})
 return decision


flag = Flag("new-checkout", 20)
users = [f"user-{i}" for i in range(200)]

# 1. Default closed: a missing user never gets the new path.
assert flag.evaluate("") is False
assert flag.evaluate(None) is False

# 2. Stickiness: the same user gets the same answer twice.
first = flag.evaluate("user-1024")
second = flag.evaluate("user-1024")
assert first == second

# 3. Kill switch overrides every bucket.
flag.killed = True
assert flag.evaluate("user-1024") is False
assert flag.evaluate("user-7") is False
flag.killed = False

# 4. The dial moves exposure for a fixed fixture set.
open_20 = sum(flag.evaluate(u) for u in users)
flag.percent = 100
open_100 = sum(flag.evaluate(u) for u in users)
flag.percent = 0
open_0 = sum(flag.evaluate(u) for u in users)
flag.percent = 20
assert open_0 == 0
assert 0 < open_20 < len(users)
assert open_100 == len(users)

# 5. A different bucketing key disagrees somewhere: the inconsistency to avoid.
session_flag = Flag("new-checkout-session", 20)
mismatch = [u for u in users if flag.evaluate(u) != session_flag.evaluate(u)]
assert mismatch, "expected at least one user where two flag keys disagree"

# 6. Audit log holds one entry per evaluation with its reason.
assert len(flag.audit) == 2 + 2 + 2 + 200 * 3 + 200
assert len(session_flag.audit) == 200
assert all(set(e) == {"user", "decision", "reason"} for e in flag.audit)
print(f"default-closed empty user: off; sticky user-1024: {'on' if first else 'off'} twice")
print(f"dial 0/20/100 over 200 users: {open_0}/{open_20}/{open_100}")
print(f"killed flag forces off; different flag keys disagree on {len(mismatch)} users, e.g. {mismatch[0]}")
print(f"audit entries: flag={len(flag.audit)}, session_flag={len(session_flag.audit)}")
print("PASS: default-closed, sticky bucketing, kill switch, dial, and audit log behave as asserted")

At twenty percent, missing users stay off, the kill switch overrides every named user, and repeated evaluation is stable. The fixed two-hundred-user cohort produces zero, thirty-five, and two hundred enabled users at zero, twenty, and one hundred percent. Thirty-five is a deterministic fixture result, not a guarantee of exactly twenty percent in every finite population.

The second evaluator changes the flag key while retaining the same user IDs. Its different decisions demonstrate salt/key drift, not an actual session-ID experiment. The audit counts verify one entry per evaluation. No distributed propagation, request-wide reuse, database rollback, or authorization is tested by this local model; those remain separate integration responsibilities.

Sources

Feature Toggles (Pete Hodgson), the toggle taxonomy, checked 2026-10-15.
LaunchDarkly: what are feature flags, progressive exposure, checked 2026-10-15.
12 Factor: build/release/run separation, where config and flags sit, checked 2026-10-15.

Blue-green deployment keeps two environments ready

Mon, 02 Nov 2026 00:00:00 +0000

Blue-green deployment prepares a new application version in a separate environment while the previous version serves traffic. After readiness checks, routing directs new work toward the new environment. Keeping the old environment available can shorten rollback, because recovery need not wait for another build. The pattern does not make every dependency duplicated, every switch instantaneous, or every data change reversible.

Routing selection and state compatibility are distinct. The model explains overlap and schema limits; it is not a measured router rollout or a guarantee of safe recovery.

Define what the switch controls

The routing mechanism might be a load-balancer backend, a service selector, or DNS. Each has different propagation and connection semantics. Updating one control-plane object does not imply that all data-plane instances change at the same instant. DNS caches and existing connections can continue using old destinations after the authoritative configuration changes.

Decide whether routing applies per request, connection, session, or queued job. A persistent connection can remain on blue while new connections reach green. Some proxies select an upstream for each request instead. The longest-lived client connection alone therefore does not define every drain window. Document the actual mechanism and its timeout and termination behavior.

Both versions should support their assigned traffic and dependencies. That does not require identical software versions or permanently paying exactly twice the total system cost. Capacity can be staged, and databases or supporting services may be shared. Savings have consequences: an old environment scaled down too far may no longer provide a quick, full-capacity rollback target.

Keep data compatible across the overlap

A shared database is common, but is a design choice rather than the definition of blue-green. Separate databases introduce replication, synchronization, and reconciliation questions. A routing reversal cannot erase writes already accepted by green or safely invent transactions missing from blue. Define what happens to data as carefully as what happens to requests.

With a shared store, expand-and-contract changes keep both versions working during overlap. Add compatible fields, deploy readers and writers that tolerate both representations, backfill where needed, and only later remove obsolete structure. Waiting for old HTTP connections to end is insufficient if old workers, scheduled jobs, rollback binaries, or external consumers still need the old schema.

Check semantic compatibility as well as column names. Both versions might parse a field while interpreting its units differently. A migration can cause rejected queries, inconsistent writes, or later corruption; corruption is not inevitable, but neither is a clean error. Exercise realistic read and write paths and validate representative business results before declaring rollback safe.

When it breaks

Mixed-version responses during cutover are not automatically a deployment failure. They may be the expected result of draining, gradual propagation, or intentional traffic splitting. They become a problem when clients or shared state cannot tolerate that overlap. Include release identity in useful diagnostics, then correlate failures with routing state, connection age, and application compatibility.

Conversely, fifty responses from one client all reporting green do not prove global atomicity. That client may repeatedly hit one edge or reuse one connection. Readiness replica counts also do not establish that a service selector, endpoint list, ingress controller, or external load balancer points at those replicas. Inspect the complete path and test representative client locations.

Finally, do not retire blue merely because the switch command succeeded. Define a rollback window, monitor error and latency changes, and confirm old workers have stopped before incompatible cleanup. If rollback depends on state that has already changed irreversibly, choose a forward repair or a rehearsed data-recovery procedure instead of repeatedly flipping the route.

Check it quickly

Run the following Python 3 model with assertions enabled. It changes only in-memory dictionaries. Two routers update separately, existing sessions retain their selected environment, and a schema contract determines whether either version can run. This is an explicit model, not a measured load-balancer rollout or a Kubernetes integration test.

routers = {"edge-a": "blue", "edge-b": "blue"}
existing = {"session-1": "blue", "session-2": "blue"}
routers["edge-a"] = "green"
assert set(routers.values()) == {"blue", "green"}
routers["edge-b"] = "green"
assert set(routers.values()) == {"green"}
assert set(existing.values()) == {"blue"}
print("new routes green; existing sessions still blue")

required = {"blue": {"amount_cents"}, "green": {"amount_cents", "currency"}}
def compatible(schema, colour):
 return required[colour] <= schema
schema = {"amount_cents", "currency"}
assert all(compatible(schema, colour) for colour in required)
routers = dict.fromkeys(routers, "blue")
assert compatible(schema, "blue")
print("rollback remains schema-compatible")

# A destructive replacement can invalidate the old version.
required["green"] = {"amount_minor", "currency"}
schema = {"amount_minor", "currency"}
assert compatible(schema, "green")
assert not compatible(schema, "blue")
print("routing rollback cannot restore a removed column")

The first assertion exposes a mixed routing state. Even after both routers select green, the modeled sessions remain blue. The additive schema supports a routing rollback; the later destructive replacement does not. The three output lines identify those boundaries without inventing a universal cutover duration.

Repeat these checks in an isolated deployment using the real router, representative traffic, and reversible test data. Measure propagation, draining, and rollback separately. Preserve evidence of both routing convergence and compatible application behavior; neither substitutes for the other.

Sources

Martin Fowler: Blue-Green Deployment, the original pattern write-up, including the shared-database caveat, checked 2026-09-16.
Kubernetes: Deployments, rollout, readiness and rollback semantics, checked 2026-09-16.
AWS: Blue/Green Deployments whitepaper, routing-level cutover and its costs, checked 2026-09-16.
AWS Route 53 cutover, DNS caching, weighted transitions and capacity tradeoffs, checked 2026-10-15.

Progressive rollouts need evidence and a recovery path

Fri, 18 Sep 2026 00:00:00 +0000

A progressive rollout increases exposure while checking whether the new version behaves acceptably. It can discover a regression before every request uses that version. It does not make a change reversible by itself, guarantee that only the selected percentage is affected, or require a feature flag. Traffic routing, instance replacement, and feature exposure are different controls that can participate in the same release plan.

These invented counts and thresholds explain a decision rule, not a deployment experiment or production SLO. Real rollout needs suitable cohorts, fresh representative measurements, observation time and a confirmed recovery path.

Define the population and its dependencies

A percentage needs a denominator: requests, users, tenants, or instances. One percent of instances need not serve one percent of traffic, and one busy customer can dominate a request sample. Choose a population that exercises relevant workloads without treating its size as a safety proof. Preserve consistent assignment when a workflow needs the same version across several requests.

The old and new populations may share databases, queues, caches, or quotas. A faulty canary can exhaust a shared dependency or write incompatible data that affects the control population too. Consequently, a one-percent canary does not cap incident impact at one percent. Identify these shared failure paths before deciding which rollout steps are acceptable.

Measure enough to make a decision

Compare canary-specific error and latency signals with a suitable control, while retaining absolute service objectives. If a common dependency harms both versions, similar results do not establish health. Define the observation window, minimum traffic, relevant routes, and behavior for missing or stale measurements before promotion. Zero observed errors among ten requests is weaker evidence than the same result among many representative requests.

Stage duration should cover the effects being evaluated: startup, cache warming, long-running jobs, delayed writes, and business cycles can all matter. A thirty-second stage is neither automatically adequate nor inherently useless. Half the traffic also does not prove full-load capacity; nonlinear bottlenecks can emerge later. Keep monitoring through full exposure and establish when the release is considered complete.

When it breaks

A timer-only rollout promotes even when no useful measurements arrived. Treat insufficient evidence as a pause, with an owner and a deadline for investigation, rather than silently counting it as success. Automatic rollback can shorten response time when its trigger and recovery action are well tested; some failures instead require a deliberate operator decision. Automation is a mechanism, not proof that recovery is safe.

Another failure is assuming that restoring old binaries restores old state. Schema changes, persisted formats, messages already published, and external side effects can survive a traffic reversal. Keep compatible old artifacts available and test the actual recovery path. Where reversal is unsafe, define a roll-forward or reconciliation procedure rather than advertising a rollback button that cannot restore service.

Fleet-wide averages can hide a localized regression. With a zero-error control, a canary handling ten percent of requests at a fifteen-percent error rate contributes only 1.5 percent to the total. That arithmetic does not make the failing population acceptable. Keep cohort breakdowns and business outcomes visible alongside aggregate health.

Check it quickly

Save the Python block as rollout_check.py and run python3 rollout_check.py with Python 3.10 or newer. The complete unchanged program passed on 2026-10-19 with Python 3.12.14. It is a deterministic decision model using invented request counts, not a deployment controller. It does not route traffic, wait for telemetry, or carry out a rollback.

from dataclasses import dataclass

STAGES = (1, 10, 50, 100)
MIN_REQUESTS = 1000
MAX_ERROR_RATE = 0.02 # Illustrative gate, not an SLO error budget.


@dataclass(frozen=True)
class Sample:
 requests: int
 errors: int


def decide(sample):
 if not 0 <= sample.errors <= sample.requests:
 raise ValueError("invalid request/error counts")
 if sample.requests < MIN_REQUESTS:
 return "PAUSE", None
 rate = sample.errors / sample.requests
 return ("ROLLBACK_REQUESTED" if rate > MAX_ERROR_RATE else "PROMOTE"), rate


def plan(samples):
 history = []
 for stage in STAGES:
 decision, rate = decide(samples[stage])
 history.append((stage, decision, rate))
 if decision != "PROMOTE":
 break
 return history


healthy = plan({stage: Sample(2000, 10) for stage in STAGES})
faulty = plan({stage: Sample(2000, 300) for stage in STAGES})
missing = plan({stage: Sample(0, 0) for stage in STAGES})
sparse = plan({stage: Sample(100, 0) for stage in STAGES})
assert [row[0] for row in healthy] == list(STAGES)
assert all(row[1] == "PROMOTE" for row in healthy)
assert faulty == [(1, "ROLLBACK_REQUESTED", 0.15)]
assert missing == sparse == [(1, "PAUSE", None)]
assert decide(Sample(1000, 20)) == ("PROMOTE", 0.02)
assert decide(Sample(1000, 21)) == ("ROLLBACK_REQUESTED", 0.021)
for sample in (Sample(-1, 0), Sample(10, -1), Sample(10, 11)):
 try:
 decide(sample)
 except ValueError:
 pass
 else:
 raise AssertionError("invalid sample accepted")

# A 10% canary at 15%, with a zero-error control, gives 1.5% fleet errors.
fleet_rate = 0.1 * 0.15 + 0.9 * 0.0
assert fleet_rate < MAX_ERROR_RATE
for label, history in (("healthy", healthy), ("faulty", faulty),
 ("missing", missing), ("sparse", sparse)):
 print(label, history)
print(f"fleet dilution: {fleet_rate:.1%}; rollback was requested, not executed")

The healthy sample promotes through all four stages. The faulty sample requests rollback at the first stage. Missing measurements and a small zero-error sample both pause. Boundary assertions accept exactly two percent and reject 2.1 percent under the deliberately chosen policy; invalid counters fail validation. These checks explain the decision rule without claiming the threshold is statistically justified.

The final calculation demonstrates dilution using its explicitly zero-error control. A real rollout must additionally compare suitable populations, validate telemetry freshness, enforce observation time, and confirm that its requested recovery actually happened. The distinction matters: returning ROLLBACK_REQUESTED is only a decision, while restored service requires observable execution and health evidence.

Sources

Google SRE: Canarying releases, cohort comparison, shared failure domains, and observation intervals, checked 2026-10-15.
Argo Rollouts: analysis, successful, failed, and inconclusive decisions, checked 2026-10-15.
Google SRE: Release engineering, reproducible artifacts and release automation, checked 2026-10-15.

HSTS should be rolled out with a recovery plan

Tue, 23 Jun 2026 00:00:00 +0000

HTTP Strict Transport Security tells a supporting browser to use HTTPS for a host during a cached period. That persistence is the feature and the risk: removing the header does not immediately erase policy already stored by clients.

Extend HSTS only after observation. Decide descendant scope and preload separately. Recovery needs valid HTTPS and cannot instantly erase every stored or preloaded policy.

Verify the HTTPS path first

Before sending HSTS, confirm that the canonical hostname serves a valid certificate and that HTTP redirects to the intended HTTPS URL without depending on user input:

curl -sS -D - -o /dev/null --max-redirs 0 http://www.example.test/
curl -sS -D - -o /dev/null https://www.example.test/

Check representative deep routes and error pages. HSTS cannot repair a certificate error because the browser requires a valid HTTPS connection before accepting the policy.

Start with a short max-age

A staged rollout limits the recovery window while configuration is new:

Strict-Transport-Security: max-age=300

After observing the site and renewal path, increase the duration deliberately. Record the date, chosen value, and rollback conditions. A final policy might use a year, but the number is a commitment rather than a scanner score.

Treat includeSubDomains as a separate decision

includeSubDomains extends policy to every subdomain. Inventory names first, including old services, delegated zones, and hostnames used only on internal networks. One HTTP-only descendant is enough to make the option disruptive for clients that received the parent policy.

Preload is not ordinary header configuration

Browser preload programs can distribute policy outside the normal response cycle. Their requirements and removal delays are stricter than setting a header. Do not request preload merely because a scanner recommends it; confirm long-term ownership of the domain and all covered names.

Verify the public response

curl -sS -D - -o /dev/null https://www.example.test/ \
 | awk 'tolower($1) == "strict-transport-security:" { print }'

Test the canonical public edge, not only an origin port. Proxies may add, remove, or duplicate the header.

Know what rollback can clear

A valid HTTPS response with max-age=0 clears that host’s learned policy only for clients that receive it. Removing the header or sending it over HTTP does not clear a cached policy. A child cannot opt out of a parent’s still-active includeSubDomains, and clearing a parent does not delete policies independently learned for children.

Scope follows the host that sends the header: policy from www.example.test covers that host and, with includeSubDomains, its descendants, not the sibling api.example.test or the parent example.test. Keep HTTPS working during recovery. Preload removal is a separate browser-update process, not an immediate response-header rollback.

Sources

RFC 6797 — HTTP Strict Transport Security, policy processing and scope, checked 2026-10-15.
MDN: Strict-Transport-Security, directives and deployment cautions, checked 2026-10-15.
HSTS preload program, submission requirements and removal constraints, checked 2026-10-15.

DNS changes need a cache-aware rollback window

Thu, 18 Jun 2026 00:00:00 +0000

Changing an address record is easy; predicting when every resolver stops using the old answer is not. The time-to-live value travels with a DNS response and controls its ordinary cache freshness. Lowering the TTL at the moment of a migration does not rewrite answers that were cached earlier.

TTL travels with each answer. Prepare before the switch, keep both endpoints valid, and use old-endpoint traffic plus a margin to guide retirement and rollback.

Lower the TTL before the event

If an existing record has a one-day TTL, lower it at least one old TTL before the planned switch. That gives caches time to fetch the shorter value while the authoritative record still points to the old endpoint. The exact interval belongs in the change plan rather than in an improvised command during the migration.

Record authoritative answers separately. The following two probes compare the system-configured resolver with a public recursive resolver; neither is inherently an authoritative query:

dig +noall +answer www.example.test A
dig @1.1.1.1 +noall +answer www.example.test A

For authoritative evidence, query the zone’s actual nameserver without recursion and inspect the AA flag, status, answer, and authority sections. The remaining TTL shown by a recursive resolver is evidence of its current cache state; different resolvers can legitimately differ. Repeat for AAAA and any CNAME chain used by the service, not only A.

Keep both sides valid

During the switch, keep the old and new endpoints capable of serving the intended hostname. TLS names, redirects, and application configuration must agree on both sides. Removing the old endpoint immediately after the authoritative record changes creates an avoidable outage for clients that still hold the prior answer.

Negative answers are cached too. Creating a name that recently returned NXDOMAIN can take time to become visible because the negative response may remain in recursive caches according to the zone’s authority data.

Raise the TTL after observation

Once authoritative and representative recursive answers point to the new endpoint and the old cache window has elapsed, restore the normal TTL. Keep the change record with the old value, new value, authoritative answer, and rollback condition.

TTL is not a universal shutdown deadline. Resolvers can serve stale data during resolution failures, and long-lived connections or application caches may outlast DNS freshness. Observe old-endpoint traffic and retain a margin; changing DNS back also has a cache window. For negative answers, inspect the SOA-derived negative TTL rather than assuming the new A record’s TTL applies.

Sources

RFC 1034 — Domain Names: Concepts and Facilities, caching model, checked 2026-10-15.
RFC 1035 — Domain Names: Implementation and Specification, TTL field, checked 2026-10-15.
RFC 2308 — Negative Caching of DNS Queries, negative caching model and retained error TTL, checked 2026-10-15.
RFC 8767 — Serving Stale Data, stale answers during DNS resolution failures, checked 2026-10-15.
BIND 9 dig manual, server selection and recursion controls, checked 2026-10-15.

A static server should have one obvious root

Thu, 04 Jun 2026 00:00:00 +0000

The most useful property of a static deployment is that an operator can answer “which directory served this response?” without reading application code. Caddy’s root and file_server directives provide that boundary directly.

Make the serving directory explicit. Audit symlinks and read/traverse permissions, then verify the actual response at the edge.

Make the root explicit

A minimal site block can be kept close to the artifact layout:

www.example.test {
 root * /srv/example-site
 encode zstd gzip

 header {
 -Server
 X-Content-Type-Options nosniff
 X-Frame-Options DENY
 Referrer-Policy strict-origin-when-cross-origin
 }

 file_server
}

The hostname and path are placeholders; www.example.test is not a publicly issuable production name. Use an owned, resolvable hostname for production and plan local TLS trust separately for a test environment. The release command should publish a complete directory, verify it, and only then make that directory the configured root. Keep the prior known-good directory until the new one has passed smoke tests.

Check the edge, not only the file tree

Request an HTML page, a downloadable text file, and a missing path. Compare content types and status codes with the local artifact. A 404 page that returns 200 is a deployment bug even if the page looks polished.

When Caddy manages HTTPS, certificate provisioning and HTTP-to-HTTPS redirects are part of the edge behavior. Test the redirect from the bare HTTP listener and the certificate name from the public hostname. Do not infer success from a healthy process alone.

A root is a publication boundary, not a sandbox

Keep source control metadata, backups, credentials, and private configuration outside the published tree. Caddy follows symlinks inside the root, so a link can expose files outside it. Check both the artifact contents and the serving user’s read/traverse permissions. Also, merely placing 404.html in the tree does not configure a custom error handler; verify the missing-route status and error-page behavior you intend.

Sources

Caddy file_server directive, file serving, root, and index defaults, checked 2026-10-15.
Caddy Automatic HTTPS, automatic TLS and issuance behavior, checked 2026-10-15.
Static-site release checklist, local companion resource.
Caddy header directive, response header operations, checked 2026-10-15.
RFC 7034 — X-Frame-Options, the DENY framing policy, checked 2026-10-15.
MDN: X-Frame-Options, framing options, checked 2026-10-15.
RFC 6761 — Special-Use Domain Names, reserved .test names, checked 2026-10-15.

Publish a static site by switching complete directories

Mon, 01 Jun 2026 00:00:00 +0000

Copying new files directly over a live static directory creates a period where visitors can receive HTML from one release and assets from another. A safer pattern stages one complete tree beside the live tree, verifies it, then changes directory names.

These two moves leave live absent between them; crashes can prolong the gap. Restore must be attempted and checked. Old asset URLs must remain reachable across separate browser requests.

Keep all release directories on one filesystem

Keep the live tree, staged tree, and rollback tree on the same filesystem. These fragments assume one Bash session, GNU Coreutils, and a single deployer with exclusive access to these names. Stop on a failed command or either side of the copy pipeline:

set -euo pipefail
df -P /srv/www

Create a uniquely named stage directory under that parent. Do not reuse an unresolved variable or broad wildcard as a deletion target.

stage=$(mktemp -d /srv/www/site.new.XXXXXX)

Populate and verify the stage

Copy the generated artifact into the empty directory, then check representative files. The final command displays a count; compare it with the verified artifact’s expected count or manifest rather than treating any number as a pass:

tar -C public -cf - . | tar -C "$stage" -xf -

test -s "$stage/index.html"
test -s "$stage/404.html"
find "$stage" -type f | wc -l

Run link and checksum checks against the staged path when the tooling supports a directory argument. Also verify that the serving user can traverse the directories and read the files: mktemp -d starts with restrictive permissions, and the extracted artifact must supply the intended modes. A stage that has not passed verification must never acquire the live name.

Switch names with an explicit rollback path

Resolve every path before the first move:

live=/srv/www/site
stamp=$(date -u +%Y%m%dT%H%M%SZ)
rollback=/srv/www/site.old-$stamp

test -d "$live" && test ! -L "$live" || exit 1
test -d "$stage" && test ! -L "$stage" || exit 1
test ! -e "$rollback" && test ! -L "$rollback" || exit 1
df -P "$live" "$stage"
live_device=$(stat -c %d -- "$live") || exit 1
stage_device=$(stat -c %d -- "$stage") || exit 1
test "$live_device" = "$stage_device" || exit 1

mv -T -- "$live" "$rollback" || exit 1
if mv -T -- "$stage" "$live"; then
 test -s "$live/index.html"
else
 mv -T -- "$rollback" "$live" || {
 printf 'Restore failed; inspect %s and %s before retrying.\n' "$rollback" "$live" >&2
 exit 1
 }
 exit 1
fi

This is a two-step switch, not one atomic replacement: the live path is absent between moves, and a crash can leave it absent until recovery. GNU mv -T prevents an existing directory from silently becoming a container for the moved tree; it does not serialize competing deployers. Do not proceed if the filesystem check differs. Platform-specific atomic exchange or symlink replacement can remove the missing-name interval, but must be tested with the actual server.

Verify the public edge

After the switch, request the homepage, one deep route, one fingerprinted asset, and a missing path. Compare public bytes with the staged artifact. Keep the rollback directory until that check and the observation window complete.

Even an atomic name switch does not make a browser’s HTML and asset requests one transaction. Keep older fingerprinted assets reachable for cached or in-flight HTML; merely retaining an unserved rollback directory does not do that. A filesystem rename also makes no promise of crash durability without the platform-specific persistence steps.

Sources

POSIX rename(), name replacement semantics, checked 2026-06-01.
Linux rename(2), filesystem and replacement behavior, checked 2026-10-15.
GNU tar manual, archive creation and extraction, checked 2026-06-01.
GNU Coreutils: Target directory, -T and directory nesting hazards, checked 2026-10-15.
GNU Coreutils: stat, filesystem device-number checks, checked 2026-10-15.

Deployment · XB Field Notes

Change windows fit people, not servers

Traffic is only part of the math

Watch the human clock

Let the window be a habit

When it breaks

Check it quickly

Further reading

Sources

Rolling updates replace pods in small waves

The surge and the gap

Health gates replacement

Choose a deliberate pace

When it breaks

Check it quickly

Further reading

Sources

Promote a verified artifact through environments

Name the unit that moves

When it breaks

Check it quickly

Verify the release that actually runs

Further reading

Sources

Feature flags decouple release from exposure

Ship behind a defined gate

Choose defaults by meaning

Ramp exposure with stable identities

When it breaks

Check it quickly

Further reading

Sources

Blue-green deployment keeps two environments ready

Define what the switch controls

Keep data compatible across the overlap

When it breaks

Check it quickly

Sources

Progressive rollouts need evidence and a recovery path

Define the population and its dependencies

Measure enough to make a decision

When it breaks

Check it quickly

Further reading

Sources

HSTS should be rolled out with a recovery plan

Verify the HTTPS path first

Start with a short max-age

Treat includeSubDomains as a separate decision

Preload is not ordinary header configuration

Verify the public response

Know what rollback can clear

Sources

DNS changes need a cache-aware rollback window

Lower the TTL before the event

Keep both sides valid

Raise the TTL after observation

Sources

A static server should have one obvious root

Make the root explicit

Check the edge, not only the file tree

A root is a publication boundary, not a sandbox

Sources

Publish a static site by switching complete directories

Keep all release directories on one filesystem

Populate and verify the stage

Switch names with an explicit rollback path

Verify the public edge

Sources