Failover · XB Field Notes

Failover is a practiced switch

Thu, 25 Feb 2027 00:00:00 +0000

A standby helps only if the promotion path works and the old writer cannot corrupt the new history. Failover is a decision, a procedure, and a rehearsal. A healthy-looking replica is not proof that clients can reach it or that it contains every acknowledged write. Define the required recovery time and acceptable data loss before deciding when to switch.

Design the decision in advance

Choose who can authorize promotion, what evidence demonstrates failure, and whether automation or an operator acts. A lost heartbeat can mean a network partition rather than a dead primary. Failure detection alone cannot authorize two independent writers. The system needs an enforced ownership mechanism, such as verified node isolation or resource-side fencing, with guarantees appropriate to its replication protocol.

For a single-writer service, the safety goal is at most one accepted writer, not uninterrupted availability of exactly one. Fencing may deliberately leave zero writers during a transition. The runbook should identify the candidate, its replication position, the acceptable loss boundary, how clients reconnect, and who stops the procedure if prerequisites fail. Automatic failover can shorten an outage but cannot manufacture missing replicated data.

Rehearse the actual path

Practice at a cadence matched to risk and after relevant changes. A drill validates the paths it actually exercises: a database promotion that bypasses DNS says little about cached addresses, and a test client using a different identity says little about production permissions. Include realistic client reconnection and checks of acknowledged writes where safe. Record scope, timings, lag, refusals, and recovery evidence, rather than treating the date of the last successful drill as a complete readiness score.

Plan the safe stopping point and recovery procedure before starting. Once the promoted node has accepted new writes, switching back is not necessarily a reversible configuration edit. The former primary may need reconciliation or rebuilding as a replica. Backups provide another recovery path, but their existence alone does not prove a usable restore.

When it breaks

Split brain appears when the old primary remains writable after promotion. Merely recording “fenced” in a controller is insufficient; the storage or isolation mechanism must enforce it. A stale candidate can lose acknowledged writes even though promotion succeeds. Express lag thresholds in meaningful units and tie them to the data-loss objective; a nonzero threshold is not a zero-loss promise.

Flapping health checks can provoke repeated transitions. Use deliberate failure thresholds and recovery criteria, and avoid automatic failback before resynchronization. DNS changes affect later lookups: existing connections and answers cached under the previous TTL may persist. Lowering the TTL during an incident does not retroactively expire those cached answers. Test the actual client’s retry and reconnection behavior as well as the authoritative DNS response.

Fence before promoting a writer. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

With Python 3.9 or newer, save this as failover_drill.py and run python3 failover_drill.py in a scratch directory. Run without -O, -OO, or a nonzero PYTHONOPTIMIZE setting so the assertions execute. It changes only in-memory objects. Health flags and lag values are synthetic; fence() is a set update, not a real fence command. The model keeps one designated primary while allowing zero writable nodes during isolation. It explicitly reactivates an eligible candidate, including during failback, instead of accidentally leaving the new primary in the fenced set.

"""Failover drill: single-primary switch with health and lag guards (stdlib only)."""
from dataclasses import dataclass

MAX_LAG = 128 # Synthetic units, not seconds or a zero-data-loss guarantee.

class FailoverRefused(Exception):
 pass

@dataclass
class Node:
 name: str
 healthy: bool = True
 lag: int = 0
 is_primary: bool = False

class Cluster:
 def __init__(self):
 self.nodes = {"a": Node("a", is_primary=True), "b": Node("b")}
 self.fenced: set[str] = set()

 @property
 def primary(self) -> str:
 primaries = [n for n in self.nodes.values() if n.is_primary]
 assert len(primaries) == 1, f"must keep exactly one primary, saw {[p.name for p in primaries]}"
 return primaries[0].name

 def can_write(self, name: str) -> bool:
 node = self.nodes[name]
 return node.is_primary and node.healthy and name not in self.fenced

 def health_check(self, name: str) -> bool:
 node = self.nodes[name]
 return node.healthy and node.lag <= MAX_LAG

 def fence(self, name: str) -> None:
 self.fenced.add(name)
 assert not self.can_write(name)

 def failover(self, target: str, *, fence_old: bool) -> str:
 old = self.primary
 if target == old:
 raise FailoverRefused("target is already primary")
 if target not in self.nodes:
 raise FailoverRefused(f"unknown target {target}")
 candidate = self.nodes[target]
 if not fence_old or old not in self.fenced:
 raise FailoverRefused(f"refuse: old primary {old} is not fenced; fence first to avoid split brain")
 if not candidate.healthy:
 raise FailoverRefused(f"refuse: standby {target} is unhealthy")
 if candidate.lag > MAX_LAG:
 raise FailoverRefused(f"refuse: standby {target} lags by {candidate.lag} (> {MAX_LAG})")
 self.nodes[old].is_primary = False
 # Model explicit reactivation after the old writer is fenced.
 self.fenced.discard(target)
 candidate.is_primary = True
 assert self.primary == target and self.can_write(target)
 assert sum(self.can_write(name) for name in self.nodes) <= 1
 return target

def expect_refused(fn, label: str) -> None:
 try:
 fn()
 except FailoverRefused as exc:
 print(f"refused as expected ({label}): {exc}")
 return
 raise AssertionError(f"should have refused: {label}")

def main() -> None:
 # 1. Clean drill: healthy standby, caught-up replica, old primary fenced.
 cluster = Cluster()
 assert cluster.primary == "a"
 assert cluster.health_check("b")
 cluster.fence("a")
 assert cluster.failover("b", fence_old=True) == "b"
 print("clean switch: a -> b")

 # 2. Refuse lag beyond this policy; even accepted positive lag may lose writes.
 cluster = Cluster()
 cluster.nodes["b"].lag = 900
 assert not cluster.health_check("b")
 cluster.fence("a")
 expect_refused(lambda: cluster.failover("b", fence_old=True), "stale replica")
 assert cluster.primary == "a"

 # 3. Unhealthy standby must not promote.
 cluster.nodes["b"].lag = 0
 cluster.nodes["b"].healthy = False
 expect_refused(lambda: cluster.failover("b", fence_old=True), "unhealthy standby")
 assert cluster.primary == "a"

 # 4. Unfenced old primary must block promotion (split-brain guard).
 cluster.nodes["b"].healthy = True
 unfenced = Cluster() # nobody fenced yet
 expect_refused(lambda: unfenced.failover("b", fence_old=False), "split brain")
 expect_refused(lambda: unfenced.failover("b", fence_old=True), "split brain without fence record")
 assert unfenced.primary == "a"

 # 5. Failback: resynchronize the candidate, then fence the CURRENT writer.
 cluster = Cluster()
 cluster.fence("a")
 cluster.failover("b", fence_old=True)
 cluster.nodes["a"].healthy = True
 cluster.nodes["a"].lag = 0
 cluster.fence("b")
 assert cluster.failover("a", fence_old=True) == "a"
 print("failback: b -> a")

 print("PASS: at most one writable node; 4 refusals + clean switch + failback passed")

if __name__ == "__main__":
 main()

The run prints a clean switch, four refusals in three categories, and failback, ending with PASS: at most one writable node; 4 refusals + clean switch + failback passed. Refused promotion leaves the designated primary unchanged, but a previously fenced primary remains unavailable. The clean-switch and failback fixtures use zero lag; the policy also permits positive lag up to 128 synthetic units, which carries a possible loss trade-off.

The earlier fixture recorded Python 3.14.4 on macOS. The revised fixture passed with Python 3.12.14 on Linux. These assertions test sequential decision logic, including the formerly fenced target’s writable state. They do not validate concurrent promotion, real replication, permissions, DNS propagation, storage isolation, or production recovery. Those require a separately authorized staging drill with the actual mechanisms and representative clients.

Sources

PostgreSQL 18: failover, old-primary isolation, rebuilding redundancy, and written switch procedures, checked 2026-10-15.
AWS: configuring DNS failover, health checks and routing configuration, checked 2026-10-15.
Google SRE: testing for reliability, production-test scope, automation barriers, and disaster recovery testing, checked 2026-10-15.
AWS: record TTL, resolver caching and planning TTL changes before a switch, checked 2026-10-15.

Redis Sentinel coordinates primary failover

Sat, 09 Jan 2027 00:00:00 +0000

Redis Sentinel supervises a non-Cluster primary/replica deployment and publishes the current primary’s address. It does not store the application’s dataset or proxy its commands. Automatic promotion can restore write availability after a primary fails, but discovery, replica freshness, client reconnection, and protection from stale writers remain separate parts of the system.

Failure agreement and election are different gates

A Sentinel first marks an instance subjectively down when acceptable replies are absent long enough. A primary becomes objectively down when the configured quorum agrees. That failure-detection threshold is not the same as the majority needed to authorize a failover leader. Election also needs at least the configured quorum; setting a higher quorum can make the requirement stricter.

For five known Sentinels with quorum two, two observers can agree about failure, but a majority of three is needed to authorize promotion. Conversely, quorum one does not let a single surviving observer authorize failover when it still knows about two others. SENTINEL CKQUORUM checks both reachability requirements; a healthy-looking Redis process does not establish either one.

Use at least three Sentinels in independent failure domains for a robust deployment. Three processes on one laptop help test protocol behavior but all fail with that machine. Each Sentinel needs a writable configuration file to persist discovered topology and epochs. Treat those files as operational state and protect any credentials they contain.

Promotion does not fence every old writer

Redis replication is asynchronous. A replica selected for promotion may lack a write that the old primary acknowledged. During a network partition, clients on the old primary’s side may even continue writing there while another side promotes a replacement. Sentinel’s election coordination does not physically revoke every old connection or turn Redis into a strongly consistent store.

Settings such as min-replicas-to-write and min-replicas-max-lag can restrict writes when replication health degrades, at an availability cost; they are not a universal fencing protocol. Independent backups and an explicit application recovery policy remain necessary. Test partitions and replica lag separately from merely killing a process.

Clients must rediscover and reconnect

A Sentinel-aware client asks observers for the primary address and then connects directly to Redis. It must handle temporary discovery failures, broken connections, and changed roles. A client pinned to the old address does not become healthy because an observer has a new answer. Likewise, a write timeout does not prove that its operation was never applied.

The example explicitly disconnects the application’s old pool after discovery reports the promoted replica, then checks a new connection. This isolates the reconnection step instead of promising invisible, interruption-free failover. Do not blindly repeat nonidempotent writes during that interval; promotion is not a business deduplication mechanism.

When it breaks

Insufficient reachable observers, no eligible replica, unwritable Sentinel state, or unusable announced addresses can prevent recovery. Authentication and network rules must permit observer-to-observer, observer-to-Redis, replication, and application connections as appropriate. Seeing one port respond to PING is not proof that all these paths work.

Monitor CKQUORUM, replica health, failover events, discovery results, and actual client operations. Detection and election take time; aggressive timeouts can mistake pauses for failures. Rehearse recovery with the deployed clients, topology, and security configuration before claiming an outage budget.

Five observers illustrate separate quorum and majority gates. Discovery, reconnection, write loss and fencing are distinct; the three-observer diagnostic fixture remains unrun.

Check it quickly

The article reports macOS, Python 3.14.4, Redis 8.10.1 and redis-py 8.1.0; no matching run log was located among reviewed evidence, and nothing was rerun here. Require POSIX, a C compiler, make, curl, tar and Python 3.10+. Keep assertions enabled: -O or PYTHONOPTIMIZE skips checks and required calls.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run with "$redis_lab/venv/bin/python"; verify setup succeeds. It starts one primary, one replica and three Sentinels on temporarily selected loopback ports. Reservations close before process binding, leaving a port race. Private Unix sockets provide control; Sentinel configurations remain writable. Scheduled Redis snapshots and AOF are disabled. Cleanup attempts to close clients, stop owned processes and remove runtime files; the build/venv remains. Printed summaries precede cleanup. Short detection settings are experimental.

from contextlib import contextmanager, ExitStack
import os
from pathlib import Path
import signal
import socket
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry
from redis.sentinel import MasterNotFoundError, Sentinel


def until(check):
 deadline = time.monotonic() + 20
 while not check():
 assert time.monotonic() < deadline, "Sentinel state did not converge"
 time.sleep(0.05)


@contextmanager
def instance(directory, port, monitor=None):
 directory.mkdir()
 endpoint = str(directory / "redis.sock")
 config = directory / "redis.conf"
 lines = ["bind 127.0.0.1", f"port {port}", "protected-mode yes",
 f"unixsocket {endpoint}", "unixsocketperm 700", f"dir {directory}"]
 if monitor is None:
 lines += ['save ""', "appendonly no"]
 else:
 lines += [f"sentinel monitor lab 127.0.0.1 {monitor} 2",
 "sentinel down-after-milliseconds lab 500",
 "sentinel failover-timeout lab 5000",
 "sentinel parallel-syncs lab 1",
 "sentinel announce-ip 127.0.0.1"]
 config.write_text("\n".join(lines) + "\n")
 command = [os.environ["REDIS_SERVER"], str(config)]
 if monitor is not None:
 command.append("--sentinel")
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 process = subprocess.Popen(command, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 assert time.monotonic() < deadline, "Redis did not start"
 time.sleep(0.02)
 assert client.info("server")["process_id"] == process.pid
 yield client, process
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode in (0, -signal.SIGKILL), errors.decode()


def crash(process):
 assert process.poll() is None
 process.kill()
 assert process.wait(timeout=5) == -signal.SIGKILL


with TemporaryDirectory(prefix="redis-sentinel-", dir="/tmp") as directory, ExitStack() as stack:
 leases = [stack.enter_context(socket.socket()) for _ in range(5)]
 for lease in leases:
 lease.bind(("127.0.0.1", 0))
 ports = [lease.getsockname()[1] for lease in leases]
 nodes = []
 for index in range(5):
 leases[index].close()
 nodes.append(stack.enter_context(instance(
 Path(directory) / str(index), ports[index], ports[0] if index >= 2 else None,
 )))
 primary, replica = nodes[0][0], nodes[1][0]
 observers = [item[0] for item in nodes[2:]]
 assert replica.replicaof("127.0.0.1", ports[0])
 until(lambda: replica.info("replication")["master_link_status"] == "up")
 until(lambda: all(observer.sentinel_master("lab")["num-other-sentinels"] == 2
 and len(observer.execute_command("SENTINEL", "REPLICAS", "lab")) == 1
 for observer in observers))
 for observer in observers:
 assert observer.execute_command("SENTINEL", "CKQUORUM", "lab").startswith(b"OK 3 usable")
 with primary.client() as writer:
 assert writer.set("marker", "replicated-before-failure")
 assert writer.wait(1, 5000) == 1
 assert replica.get("marker") == b"replicated-before-failure"
 print("three Sentinels discover each other and a replica; WAIT confirms the fixture write")

 manager = Sentinel([("127.0.0.1", port) for port in ports[2:]],
 socket_timeout=1, retry=Retry(NoBackoff(), 0),
 sentinel_kwargs={"socket_timeout": 1, "retry": Retry(NoBackoff(), 0)})
 application = manager.master_for("lab")
 try:
 assert manager.discover_master("lab") == ("127.0.0.1", ports[0])
 assert application.get("marker") == b"replicated-before-failure"
 crash(nodes[0][1])
 try:
 primary.get("marker")
 except redis.ConnectionError:
 pass
 else:
 raise AssertionError("the original primary connection must fail")

 def switched():
 try:
 return (manager.discover_master("lab") == ("127.0.0.1", ports[1])
 and replica.info("replication")["role"] == "master")
 except MasterNotFoundError:
 return False

 until(switched)
 application.connection_pool.disconnect()
 assert application.get("marker") == b"replicated-before-failure"
 assert application.set("after-failover", "new-primary")
 assert replica.get("after-failover") == b"new-primary"
 print("automatic failover changes discovery; a reconnected client uses the promoted replica")
 finally:
 application.close()
 for sentinel_client in manager.sentinels:
 sentinel_client.close()

 for item in nodes[3:]:
 crash(item[1])
 until(lambda: len(observers[0].sentinel_sentinels("lab")) == 2
 and all(peer["is_sdown"] for peer in observers[0].sentinel_sentinels("lab")))
 assert observers[0].execute_command("SENTINEL", "SET", "lab", "quorum", 1) == b"OK"
 assert observers[0].sentinel_master("lab")["quorum"] == 1
 try:
 observers[0].execute_command("SENTINEL", "CKQUORUM", "lab")
 except redis.ResponseError as error:
 assert "majority" in str(error) and "specified quorum" not in str(error)
 print("one of three known Sentinels meets quorum=1 but cannot authorize failover")
 else:
 raise AssertionError("one surviving observer is not a majority of three")

The fixture waits for peer and replica discovery, then writes a marker and checks WAIT on that same connection. After the primary is killed, its old connection fails, discovery identifies the promoted replica, and a reconnected application reads the marker and writes new data.

Finally, two observers are killed and the survivor’s quorum is deliberately lowered to one. CKQUORUM still rejects authorization because one of three known observers is not a majority. This checks the diagnostic gate, not a second promotion attempt. WAIT confirms the fixture’s replication acknowledgment, not disk durability or strong consistency. Network partitions, stalled replicas, independent-host failures, and production recovery latency remain untested.

Sources

Redis Sentinel, failure detection, authorization, discovery, and partition risks, checked 2026-09-08.
Redis replication, asynchronous replication and write restrictions, checked 2026-09-08.
Redis WAIT, connection-scoped acknowledgment and consistency limits, checked 2026-09-08.
Redis 8.10.1 Sentinel source, separate quorum and majority checks, checked 2026-09-08.
redis-py 8.1.0 Sentinel client, discovery and primary connection pools, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.
Python assert, optimization removes assertion evaluation, checked 2026-10-18.

Failover · XB Field Notes

Failover is a practiced switch

Design the decision in advance

Rehearse the actual path

When it breaks

Check it quickly

Further reading

Sources

Redis Sentinel coordinates primary failover

Failure agreement and election are different gates

Promotion does not fence every old writer

Clients must rediscover and reconnect

When it breaks

Check it quickly

Sources