A standby helps only if the promotion path works and the old writer cannot corrupt the new history. Failover is a decision, a procedure, and a rehearsal. A healthy-looking replica is not proof that clients can reach it or that it contains every acknowledged write. Define the required recovery time and acceptable data loss before deciding when to switch.

Design the decision in advance

Choose who can authorize promotion, what evidence demonstrates failure, and whether automation or an operator acts. A lost heartbeat can mean a network partition rather than a dead primary. Failure detection alone cannot authorize two independent writers. The system needs an enforced ownership mechanism, such as verified node isolation or resource-side fencing, with guarantees appropriate to its replication protocol.

For a single-writer service, the safety goal is at most one accepted writer, not uninterrupted availability of exactly one. Fencing may deliberately leave zero writers during a transition. The runbook should identify the candidate, its replication position, the acceptable loss boundary, how clients reconnect, and who stops the procedure if prerequisites fail. Automatic failover can shorten an outage but cannot manufacture missing replicated data.

Rehearse the actual path

Practice at a cadence matched to risk and after relevant changes. A drill validates the paths it actually exercises: a database promotion that bypasses DNS says little about cached addresses, and a test client using a different identity says little about production permissions. Include realistic client reconnection and checks of acknowledged writes where safe. Record scope, timings, lag, refusals, and recovery evidence, rather than treating the date of the last successful drill as a complete readiness score.

Plan the safe stopping point and recovery procedure before starting. Once the promoted node has accepted new writes, switching back is not necessarily a reversible configuration edit. The former primary may need reconciliation or rebuilding as a replica. Backups provide another recovery path, but their existence alone does not prove a usable restore.

When it breaks

Split brain appears when the old primary remains writable after promotion. Merely recording “fenced” in a controller is insufficient; the storage or isolation mechanism must enforce it. A stale candidate can lose acknowledged writes even though promotion succeeds. Express lag thresholds in meaningful units and tie them to the data-loss objective; a nonzero threshold is not a zero-loss promise.

Flapping health checks can provoke repeated transitions. Use deliberate failure thresholds and recovery criteria, and avoid automatic failback before resynchronization. DNS changes affect later lookups: existing connections and answers cached under the previous TTL may persist. Lowering the TTL during an incident does not retroactively expire those cached answers. Test the actual client’s retry and reconnection behavior as well as the authoritative DNS response.

The sequential model moves from writable a, through zero writable nodes after fencing a, to writable b after eligibility checks. Failback resynchronizes a, fences b and explicitly reactivates a. Health and lag are synthetic.
Fence before promoting a writer. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

With Python 3.9 or newer, save this as failover_drill.py and run python3 failover_drill.py in a scratch directory. Run without -O, -OO, or a nonzero PYTHONOPTIMIZE setting so the assertions execute. It changes only in-memory objects. Health flags and lag values are synthetic; fence() is a set update, not a real fence command. The model keeps one designated primary while allowing zero writable nodes during isolation. It explicitly reactivates an eligible candidate, including during failback, instead of accidentally leaving the new primary in the fenced set.

"""Failover drill: single-primary switch with health and lag guards (stdlib only)."""
from dataclasses import dataclass

MAX_LAG = 128  # Synthetic units, not seconds or a zero-data-loss guarantee.

class FailoverRefused(Exception):
    pass

@dataclass
class Node:
    name: str
    healthy: bool = True
    lag: int = 0
    is_primary: bool = False

class Cluster:
    def __init__(self):
        self.nodes = {"a": Node("a", is_primary=True), "b": Node("b")}
        self.fenced: set[str] = set()

    @property
    def primary(self) -> str:
        primaries = [n for n in self.nodes.values() if n.is_primary]
        assert len(primaries) == 1, f"must keep exactly one primary, saw {[p.name for p in primaries]}"
        return primaries[0].name

    def can_write(self, name: str) -> bool:
        node = self.nodes[name]
        return node.is_primary and node.healthy and name not in self.fenced

    def health_check(self, name: str) -> bool:
        node = self.nodes[name]
        return node.healthy and node.lag <= MAX_LAG

    def fence(self, name: str) -> None:
        self.fenced.add(name)
        assert not self.can_write(name)

    def failover(self, target: str, *, fence_old: bool) -> str:
        old = self.primary
        if target == old:
            raise FailoverRefused("target is already primary")
        if target not in self.nodes:
            raise FailoverRefused(f"unknown target {target}")
        candidate = self.nodes[target]
        if not fence_old or old not in self.fenced:
            raise FailoverRefused(f"refuse: old primary {old} is not fenced; fence first to avoid split brain")
        if not candidate.healthy:
            raise FailoverRefused(f"refuse: standby {target} is unhealthy")
        if candidate.lag > MAX_LAG:
            raise FailoverRefused(f"refuse: standby {target} lags by {candidate.lag} (> {MAX_LAG})")
        self.nodes[old].is_primary = False
        # Model explicit reactivation after the old writer is fenced.
        self.fenced.discard(target)
        candidate.is_primary = True
        assert self.primary == target and self.can_write(target)
        assert sum(self.can_write(name) for name in self.nodes) <= 1
        return target

def expect_refused(fn, label: str) -> None:
    try:
        fn()
    except FailoverRefused as exc:
        print(f"refused as expected ({label}): {exc}")
        return
    raise AssertionError(f"should have refused: {label}")

def main() -> None:
    # 1. Clean drill: healthy standby, caught-up replica, old primary fenced.
    cluster = Cluster()
    assert cluster.primary == "a"
    assert cluster.health_check("b")
    cluster.fence("a")
    assert cluster.failover("b", fence_old=True) == "b"
    print("clean switch: a -> b")

    # 2. Refuse lag beyond this policy; even accepted positive lag may lose writes.
    cluster = Cluster()
    cluster.nodes["b"].lag = 900
    assert not cluster.health_check("b")
    cluster.fence("a")
    expect_refused(lambda: cluster.failover("b", fence_old=True), "stale replica")
    assert cluster.primary == "a"

    # 3. Unhealthy standby must not promote.
    cluster.nodes["b"].lag = 0
    cluster.nodes["b"].healthy = False
    expect_refused(lambda: cluster.failover("b", fence_old=True), "unhealthy standby")
    assert cluster.primary == "a"

    # 4. Unfenced old primary must block promotion (split-brain guard).
    cluster.nodes["b"].healthy = True
    unfenced = Cluster()  # nobody fenced yet
    expect_refused(lambda: unfenced.failover("b", fence_old=False), "split brain")
    expect_refused(lambda: unfenced.failover("b", fence_old=True), "split brain without fence record")
    assert unfenced.primary == "a"

    # 5. Failback: resynchronize the candidate, then fence the CURRENT writer.
    cluster = Cluster()
    cluster.fence("a")
    cluster.failover("b", fence_old=True)
    cluster.nodes["a"].healthy = True
    cluster.nodes["a"].lag = 0
    cluster.fence("b")
    assert cluster.failover("a", fence_old=True) == "a"
    print("failback: b -> a")

    print("PASS: at most one writable node; 4 refusals + clean switch + failback passed")

if __name__ == "__main__":
    main()

The run prints a clean switch, four refusals in three categories, and failback, ending with PASS: at most one writable node; 4 refusals + clean switch + failback passed. Refused promotion leaves the designated primary unchanged, but a previously fenced primary remains unavailable. The clean-switch and failback fixtures use zero lag; the policy also permits positive lag up to 128 synthetic units, which carries a possible loss trade-off.

The earlier fixture recorded Python 3.14.4 on macOS. The revised fixture passed with Python 3.12.14 on Linux. These assertions test sequential decision logic, including the formerly fenced target’s writable state. They do not validate concurrent promotion, real replication, permissions, DNS propagation, storage isolation, or production recovery. Those require a separately authorized staging drill with the actual mechanisms and representative clients.

Further reading

Sources