etcd elects a leader so writes happen in order

Thu, 09 Mar 2028 00:00:00 +0000

etcd uses Raft to order changes to its replicated state. Replicas can temporarily lag; consensus does not mean every member holds identical bytes at every instant. A leader coordinates replication, while the voting majority determines progress. Keep election, commitment, local application, and client acknowledgment separate when diagnosing whether an operation succeeded.

One leader per term, with voting restrictions

A candidate needs a majority of votes, and each voter grants at most one vote in a term. Log freshness also matters: a newer term alone does not make an outdated candidate eligible for every vote. Randomized election timeouts reduce repeated split votes. Terms order leadership attempts; they do not measure elapsed time or directly identify a network partition.

For an entry from its current term, a leader can advance commitment after replication to a majority. Earlier-term entries become committed indirectly as that current-term entry commits; merely counting old copies is not the complete Raft rule. Within the protocol’s storage and failure assumptions, later leaders preserve committed entries. Simultaneous loss of persistent data or restoring an old backup is a different disaster-recovery problem.

An election interrupts progress until a leader is established. Client requests can wait, time out, or fail according to their deadlines and connectivity. A timeout does not establish that a proposed operation never committed. Applications need an appropriate retry or transaction design rather than interpreting every failed response as permission to repeat a non-idempotent effect.

Count voters rather than processes

A voting configuration of N members needs floor(N/2)+1 for a majority. Three voters tolerate one unavailable member; four still tolerate one but need three participants. Five and six each tolerate two. An even count does not add another tolerated failure over the preceding odd count, although temporary even configurations can occur during controlled membership changes.

A learner receives replication but does not vote. Adding it leaves the voting quorum unchanged; promotion changes that count after catch-up checks. This avoids immediately depending on an empty new replica for voting, but catch-up traffic still consumes resources and can affect availability. An unchanged quorum is not a guarantee that adding a learner cannot contribute to timeouts or an election.

When it breaks

Without a reachable majority, the cluster cannot make normal write progress. Serializable reads can use one member’s local state and may return stale values. Linearizable reads need coordination that establishes current state; they are not simply local reads with a nicer label. They also need not append a new log entry for every read, so calling all reads replicated writes obscures the implementation.

The minority of a network partition cannot independently commit a competing history. Recovery behavior depends on the actual configuration. etcd 3.5 enables pre-vote by default to reduce disruption when a disconnected member rejoins. A healed partition therefore does not inevitably force a higher-term election and write outage. Inspect configuration, timing, and logs instead of asserting that every isolated member must have raised its term.

Losing a follower may leave enough voters to continue, but lag still reduces redundancy and affects local reads. Do not dismiss it until the majority disappears. Likewise, a failed endpoint probe can reflect client routing, TLS, permissions, timeouts, or an active alarm rather than a dead voter. Client reachability and peer-to-peer quorum connectivity are different observations.

Replication, then commitment. Source-based mechanism illustration. Model values and source-reported observations are not a new runtime test.

Check it quickly

Run this Python 3 arithmetic fixture without contacting a cluster. It prints voting thresholds and checks the learner distinction; it does not implement election or replication:

def quorum(voters):
 return voters // 2 + 1

for voters in (1, 2, 3, 4, 5, 6):
 required = quorum(voters)
 print(f"voters={voters}, quorum={required}, tolerated={voters-required}")
assert quorum(3) == 2 and quorum(4) == 3
assert 2 < quorum(4) # Neither side of a 2+2 partition has a majority.
# A learner replicates data but is not included in the voting count.
voters, learners = 3, 1
assert quorum(voters) == 2
print("PASS: adding a learner leaves the voting quorum unchanged")

The output gives quorums one, two, two, three, three, four for one through six voters. A two-plus-two split of four voters has no majority. Three voters plus one learner still require two votes. PASS establishes these calculations only, not the behavior of a live cluster under failure.

The original operational example below is retained, not rerun. These private addresses and the /registry/ prefix are placeholders: adapt them only to an authorized test cluster, with its TLS/authentication settings and non-sensitive fixture keys. No authenticated cluster or partition test was executed in this review:

etcdctl --endpoints=https://10.0.1.11:2379 endpoint status --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379,https://10.0.1.12:2379,https://10.0.1.13:2379 endpoint health --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 member list --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 --consistency=s get /registry/ --prefix --keys-only | head

The first command queries only one endpoint, so its table cannot prove that the cluster has exactly one leader. Collect status from every member before comparing leader identity, terms, and applied-index progress; samples need not be simultaneous. The final query intentionally uses local serializable semantics. head limits displayed lines, not the server-side range read. Success does not certify quorum health. Preserve probe errors and avoid changing membership merely to make a table look healthy.

Sources

etcd failure modes, leader loss, majority loss, and partitions, checked 2026-10-15.
Runtime reconfiguration, voting membership and learner promotion, checked 2026-10-15.
etcd learner design, non-voting replication and catch-up load, checked 2026-10-15.
etcd API guarantees, linearizable and serializable operations, checked 2026-10-15.
etcd 3.5 configuration, pre-vote enabled by default, checked 2026-10-15.
Raft paper, sections 5.2 and 5.4 on voting and current-term commitment, checked 2026-10-15.
etcdctl 3.5.21 endpoint checks, health read and alarm checks, checked 2026-10-15.