A progressive rollout increases exposure while checking whether the new version behaves acceptably. It can discover a regression before every request uses that version. It does not make a change reversible by itself, guarantee that only the selected percentage is affected, or require a feature flag. Traffic routing, instance replacement, and feature exposure are different controls that can participate in the same release plan.

The teaching model pauses below 1000 requests, promotes at an error rate up to 2 percent, and requests rollback above 2 percent. It stops at the first non-promotion. Shared dependencies can widen harm; fleet averages can hide a failing cohort. A recovery request still needs execution and health evidence.
These invented counts and thresholds explain a decision rule, not a deployment experiment or production SLO. Real rollout needs suitable cohorts, fresh representative measurements, observation time and a confirmed recovery path.

Define the population and its dependencies

A percentage needs a denominator: requests, users, tenants, or instances. One percent of instances need not serve one percent of traffic, and one busy customer can dominate a request sample. Choose a population that exercises relevant workloads without treating its size as a safety proof. Preserve consistent assignment when a workflow needs the same version across several requests.

The old and new populations may share databases, queues, caches, or quotas. A faulty canary can exhaust a shared dependency or write incompatible data that affects the control population too. Consequently, a one-percent canary does not cap incident impact at one percent. Identify these shared failure paths before deciding which rollout steps are acceptable.

Measure enough to make a decision

Compare canary-specific error and latency signals with a suitable control, while retaining absolute service objectives. If a common dependency harms both versions, similar results do not establish health. Define the observation window, minimum traffic, relevant routes, and behavior for missing or stale measurements before promotion. Zero observed errors among ten requests is weaker evidence than the same result among many representative requests.

Stage duration should cover the effects being evaluated: startup, cache warming, long-running jobs, delayed writes, and business cycles can all matter. A thirty-second stage is neither automatically adequate nor inherently useless. Half the traffic also does not prove full-load capacity; nonlinear bottlenecks can emerge later. Keep monitoring through full exposure and establish when the release is considered complete.

When it breaks

A timer-only rollout promotes even when no useful measurements arrived. Treat insufficient evidence as a pause, with an owner and a deadline for investigation, rather than silently counting it as success. Automatic rollback can shorten response time when its trigger and recovery action are well tested; some failures instead require a deliberate operator decision. Automation is a mechanism, not proof that recovery is safe.

Another failure is assuming that restoring old binaries restores old state. Schema changes, persisted formats, messages already published, and external side effects can survive a traffic reversal. Keep compatible old artifacts available and test the actual recovery path. Where reversal is unsafe, define a roll-forward or reconciliation procedure rather than advertising a rollback button that cannot restore service.

Fleet-wide averages can hide a localized regression. With a zero-error control, a canary handling ten percent of requests at a fifteen-percent error rate contributes only 1.5 percent to the total. That arithmetic does not make the failing population acceptable. Keep cohort breakdowns and business outcomes visible alongside aggregate health.

Check it quickly

Save the Python block as rollout_check.py and run python3 rollout_check.py with Python 3.10 or newer. The complete unchanged program passed on 2026-10-19 with Python 3.12.14. It is a deterministic decision model using invented request counts, not a deployment controller. It does not route traffic, wait for telemetry, or carry out a rollback.

from dataclasses import dataclass

STAGES = (1, 10, 50, 100)
MIN_REQUESTS = 1000
MAX_ERROR_RATE = 0.02  # Illustrative gate, not an SLO error budget.


@dataclass(frozen=True)
class Sample:
    requests: int
    errors: int


def decide(sample):
    if not 0 <= sample.errors <= sample.requests:
        raise ValueError("invalid request/error counts")
    if sample.requests < MIN_REQUESTS:
        return "PAUSE", None
    rate = sample.errors / sample.requests
    return ("ROLLBACK_REQUESTED" if rate > MAX_ERROR_RATE else "PROMOTE"), rate


def plan(samples):
    history = []
    for stage in STAGES:
        decision, rate = decide(samples[stage])
        history.append((stage, decision, rate))
        if decision != "PROMOTE":
            break
    return history


healthy = plan({stage: Sample(2000, 10) for stage in STAGES})
faulty = plan({stage: Sample(2000, 300) for stage in STAGES})
missing = plan({stage: Sample(0, 0) for stage in STAGES})
sparse = plan({stage: Sample(100, 0) for stage in STAGES})
assert [row[0] for row in healthy] == list(STAGES)
assert all(row[1] == "PROMOTE" for row in healthy)
assert faulty == [(1, "ROLLBACK_REQUESTED", 0.15)]
assert missing == sparse == [(1, "PAUSE", None)]
assert decide(Sample(1000, 20)) == ("PROMOTE", 0.02)
assert decide(Sample(1000, 21)) == ("ROLLBACK_REQUESTED", 0.021)
for sample in (Sample(-1, 0), Sample(10, -1), Sample(10, 11)):
    try:
        decide(sample)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid sample accepted")

# A 10% canary at 15%, with a zero-error control, gives 1.5% fleet errors.
fleet_rate = 0.1 * 0.15 + 0.9 * 0.0
assert fleet_rate < MAX_ERROR_RATE
for label, history in (("healthy", healthy), ("faulty", faulty),
                       ("missing", missing), ("sparse", sparse)):
    print(label, history)
print(f"fleet dilution: {fleet_rate:.1%}; rollback was requested, not executed")

The healthy sample promotes through all four stages. The faulty sample requests rollback at the first stage. Missing measurements and a small zero-error sample both pause. Boundary assertions accept exactly two percent and reject 2.1 percent under the deliberately chosen policy; invalid counters fail validation. These checks explain the decision rule without claiming the threshold is statistically justified.

The final calculation demonstrates dilution using its explicitly zero-error control. A real rollout must additionally compare suitable populations, validate telemetry freshness, enforce observation time, and confirm that its requested recovery actually happened. The distinction matters: returning ROLLBACK_REQUESTED is only a decision, while restored service requires observable execution and health evidence.

Further reading

Sources