Distributed · XB Field Notes

etcd elects a leader so writes happen in order

Thu, 09 Mar 2028 00:00:00 +0000

etcd uses Raft to order changes to its replicated state. Replicas can temporarily lag; consensus does not mean every member holds identical bytes at every instant. A leader coordinates replication, while the voting majority determines progress. Keep election, commitment, local application, and client acknowledgment separate when diagnosing whether an operation succeeded.

One leader per term, with voting restrictions

A candidate needs a majority of votes, and each voter grants at most one vote in a term. Log freshness also matters: a newer term alone does not make an outdated candidate eligible for every vote. Randomized election timeouts reduce repeated split votes. Terms order leadership attempts; they do not measure elapsed time or directly identify a network partition.

For an entry from its current term, a leader can advance commitment after replication to a majority. Earlier-term entries become committed indirectly as that current-term entry commits; merely counting old copies is not the complete Raft rule. Within the protocol’s storage and failure assumptions, later leaders preserve committed entries. Simultaneous loss of persistent data or restoring an old backup is a different disaster-recovery problem.

An election interrupts progress until a leader is established. Client requests can wait, time out, or fail according to their deadlines and connectivity. A timeout does not establish that a proposed operation never committed. Applications need an appropriate retry or transaction design rather than interpreting every failed response as permission to repeat a non-idempotent effect.

Count voters rather than processes

A voting configuration of N members needs floor(N/2)+1 for a majority. Three voters tolerate one unavailable member; four still tolerate one but need three participants. Five and six each tolerate two. An even count does not add another tolerated failure over the preceding odd count, although temporary even configurations can occur during controlled membership changes.

A learner receives replication but does not vote. Adding it leaves the voting quorum unchanged; promotion changes that count after catch-up checks. This avoids immediately depending on an empty new replica for voting, but catch-up traffic still consumes resources and can affect availability. An unchanged quorum is not a guarantee that adding a learner cannot contribute to timeouts or an election.

When it breaks

Without a reachable majority, the cluster cannot make normal write progress. Serializable reads can use one member’s local state and may return stale values. Linearizable reads need coordination that establishes current state; they are not simply local reads with a nicer label. They also need not append a new log entry for every read, so calling all reads replicated writes obscures the implementation.

The minority of a network partition cannot independently commit a competing history. Recovery behavior depends on the actual configuration. etcd 3.5 enables pre-vote by default to reduce disruption when a disconnected member rejoins. A healed partition therefore does not inevitably force a higher-term election and write outage. Inspect configuration, timing, and logs instead of asserting that every isolated member must have raised its term.

Losing a follower may leave enough voters to continue, but lag still reduces redundancy and affects local reads. Do not dismiss it until the majority disappears. Likewise, a failed endpoint probe can reflect client routing, TLS, permissions, timeouts, or an active alarm rather than a dead voter. Client reachability and peer-to-peer quorum connectivity are different observations.

Replication, then commitment. Source-based mechanism illustration. Model values and source-reported observations are not a new runtime test.

Check it quickly

Run this Python 3 arithmetic fixture without contacting a cluster. It prints voting thresholds and checks the learner distinction; it does not implement election or replication:

def quorum(voters):
 return voters // 2 + 1

for voters in (1, 2, 3, 4, 5, 6):
 required = quorum(voters)
 print(f"voters={voters}, quorum={required}, tolerated={voters-required}")
assert quorum(3) == 2 and quorum(4) == 3
assert 2 < quorum(4) # Neither side of a 2+2 partition has a majority.
# A learner replicates data but is not included in the voting count.
voters, learners = 3, 1
assert quorum(voters) == 2
print("PASS: adding a learner leaves the voting quorum unchanged")

The output gives quorums one, two, two, three, three, four for one through six voters. A two-plus-two split of four voters has no majority. Three voters plus one learner still require two votes. PASS establishes these calculations only, not the behavior of a live cluster under failure.

The original operational example below is retained, not rerun. These private addresses and the /registry/ prefix are placeholders: adapt them only to an authorized test cluster, with its TLS/authentication settings and non-sensitive fixture keys. No authenticated cluster or partition test was executed in this review:

etcdctl --endpoints=https://10.0.1.11:2379 endpoint status --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379,https://10.0.1.12:2379,https://10.0.1.13:2379 endpoint health --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 member list --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 --consistency=s get /registry/ --prefix --keys-only | head

The first command queries only one endpoint, so its table cannot prove that the cluster has exactly one leader. Collect status from every member before comparing leader identity, terms, and applied-index progress; samples need not be simultaneous. The final query intentionally uses local serializable semantics. head limits displayed lines, not the server-side range read. Success does not certify quorum health. Preserve probe errors and avoid changing membership merely to make a table look healthy.

Sources

etcd failure modes, leader loss, majority loss, and partitions, checked 2026-10-15.
Runtime reconfiguration, voting membership and learner promotion, checked 2026-10-15.
etcd learner design, non-voting replication and catch-up load, checked 2026-10-15.
etcd API guarantees, linearizable and serializable operations, checked 2026-10-15.
etcd 3.5 configuration, pre-vote enabled by default, checked 2026-10-15.
Raft paper, sections 5.2 and 5.4 on voting and current-term commitment, checked 2026-10-15.
etcdctl 3.5.21 endpoint checks, health read and alarm checks, checked 2026-10-15.

Raft elections need a majority of voters

Tue, 21 Sep 2027 00:00:00 +0000

Raft leader elections require a strict majority of the configured voting members. With fixed membership and one durable vote per server per term, two candidates cannot both win the same term: their majorities would overlap at a voter that cannot support both. This is an election guarantee, not a claim that every node always agrees about the current leader.

One leader per term

Count the configured voters, not just reachable machines. Five voters require three votes even when only two can communicate. Learners and non-voting replicas do not automatically enlarge that denominator. Membership changes require their own safe protocol; independently replacing the voter list in each partition defeats the intersection argument.

The vote-once rule matters as much as the arithmetic. A server must retain its current term and vote across crashes before acknowledging them. Repeating a response from one voter does not produce extra votes. Authentication and correct membership identity are also assumed; this crash-fault model does not establish protection against malicious voters inventing messages.

Authority differs from local belief

A majority partition can elect a leader in a newer term while an isolated old leader still believes it leads the earlier term. Basic Raft steps down when it learns of a higher term; immediate resignation merely because heartbeats receive no replies is not the universal rule. Implementations may add quorum checking, but that is a separate mechanism.

The minority cannot commit new client commands by itself. Election restrictions, log matching, and commitment rules work together to protect replicated state across terms. A majority count alone does not make arbitrary external writes safe. A worker acting on a previously granted leadership belief may need resource-side fencing, and linearizable reads need the protocol’s current-authority checks rather than trust in a local leader flag.

Terms and timers have different jobs

A term is a logical epoch of arbitrary length, not a fixed-duration lease. Heartbeats normally prevent followers from starting elections while they hear from a valid leader. Missing them triggers suspicion and a later election; it does not prove the old machine stopped executing. Raft’s safety argument does not depend on synchronized wall clocks.

Randomized election timeouts help competing candidates avoid repeatedly splitting votes. Timely communication and sufficient processing capacity are still needed for progress. Overloaded machines, packet delays, and aggressive timeouts can cause election churn even without a permanent partition. Tune from observed latency and scheduling behavior, not an assumption that a shorter timeout is always better.

When it breaks

Four voters require three votes and cannot progress after a two-versus-two split. Three and four voters both tolerate one unavailable voter; adding a fourth does not buy a second failure. Five voters tolerate two. Odd sizes improve this particular capacity tradeoff, but even sizes still retain the same quorum-intersection safety property.

A minority may receive or buffer a request, but must not report a new uncommitted command as successfully committed. Client timeouts also leave uncertainty about whether a command committed elsewhere. Retrying safely therefore depends on the application contract, including request deduplication where needed, rather than assuming every unavailable response proves no effect occurred.

A candidate also needs a sufficiently up-to-date log. In Raft, voters compare the last log term and then its index, rather than simply choosing the longest log. Removing this restriction can violate leader completeness even when the candidate collects a numerical majority. The ballot exercise deliberately omits this log rule and cannot validate a consensus implementation.

Three of five, in one term. Source-based mechanism illustration. Model values and source-reported observations are not a new runtime test.

Check it quickly

Save this as majority.py and run python3 majority.py. It uses a shared in-memory ballot box, fixed voter identities, and explicit terms. Repeated messages are deduplicated. It performs no network operations or durable writes.

def quorum(size):
 if size < 1:
 raise ValueError("at least one voter is required")
 return size // 2 + 1


class Ballots:
 """Fixed membership, unique votes, and in-memory per-term voting only."""
 def __init__(self, size):
 self.size = size
 quorum(size)
 self.terms = [0] * size
 self.voted_for = [None] * size

 def elect(self, candidate, term, reachable):
 nodes = sorted(set(reachable))
 if not 0 <= candidate < self.size or term < 1:
 raise ValueError("invalid candidate or term")
 if any(not 0 <= node < self.size for node in nodes):
 raise ValueError("unknown voter")
 votes = []
 for node in nodes:
 if term < self.terms[node]:
 continue
 if term > self.terms[node]:
 self.terms[node] = term
 self.voted_for[node] = None
 if self.voted_for[node] in (None, candidate):
 self.voted_for[node] = candidate
 votes.append(node)
 return len(votes) >= quorum(self.size), votes


assert [quorum(n) for n in (3, 4, 5)] == [2, 3, 3]
cluster = Ballots(5)
assert cluster.elect(0, 1, [0, 1, 2]) == (True, [0, 1, 2])
assert cluster.elect(3, 1, [3, 4]) == (False, [3, 4])
assert cluster.elect(3, 1, range(5)) == (False, [3, 4])
assert cluster.elect(0, 1, [0, 1, 2]) == (True, [0, 1, 2])
print("one winner per term; repeated same-candidate requests preserve votes")

assert cluster.elect(3, 2, range(5)) == (True, list(range(5)))
assert cluster.elect(0, 1, range(5)) == (False, [])
print("a newer term can elect another candidate; older requests are rejected")

even = Ballots(4)
assert not even.elect(0, 1, [0, 1])[0]
assert not even.elect(2, 1, [2, 3])[0]
assert not Ballots(5).elect(0, 1, [0, 0, 0])[0]
print("2-2 split stalls; repeating one voter cannot manufacture a majority")
print("PASS: fixed-membership ballot rules only; no replicated log or network")

Expect four confirmation lines. A three-member side wins, a two-member side loses, and a second candidate cannot reuse votes already given in the same term. A later term can elect a different candidate. Older requests then fail, a two-two partition stalls, and three copies of one identity remain one vote.

The exact revised example passed with Python 3.12.14 on Linux. Those assertions establish only this ballot model’s behavior. They do not demonstrate log replication, crash persistence, membership reconfiguration, timing, or a live cluster recovering from partition.

Sources

In Search of an Understandable Consensus Algorithm (Raft), leader election and majority quorum in section 5, checked 2026-09-09.
etcd: etcd versus other key-value stores, leader election and distributed coordination over Raft, checked 2026-09-09.
ZooKeeper Internals, majority quorums for atomic broadcast and leader election, checked 2026-09-09.

Gossip converges under explicit assumptions

Mon, 20 Sep 2027 00:00:00 +0000

Gossip spreads information through repeated peer contact. It can tolerate some lost exchanges because surviving copies can be sent again. Eventual convergence still needs assumptions: relevant peers must become reachable, updates must remain available, and the merge rule must handle the states that can arise. Random contact alone does not guarantee agreement by a fixed deadline.

Separate dissemination from reconciliation

A node can periodically push state, pull another node’s state, or exchange in both directions. These are protocol choices, not a requirement that every gossip message be symmetrical. A bounded number of contacts per node can spread a small update rapidly through a well-connected population. Logarithmic dissemination estimates describe particular probabilistic models; they are not a worst-case timer derived from one network round trip.

Membership gossip also does not make application writes consistent by itself. Dynamo uses gossip for membership, vector clocks for object versions, and Merkle-tree anti-entropy for repairing replicas. Those mechanisms solve different problems. A membership protocol’s incarnation number distinguishes states of one member; it is not automatically a causal version for arbitrary business data.

The model below intentionally assumes one writer per key and increasing versions. Selecting the larger version is enough for its single update. Equal versions containing different values expose an undefined conflict policy: retaining whichever value arrived first makes the merge order-dependent. Production reconciliation needs a declared policy for concurrency, duplicate delivery, deletion, and restart identity.

When it breaks

A lasting partition prevents an update from crossing between disconnected components. Repetition inside each component cannot repair that missing path. After connectivity returns, reconciliation can resume if the necessary state still exists. A deleted item also needs a deletion representation and retention policy; expiring tombstones too early can let stale data return. Version vectors and tombstones are complementary mechanisms, not interchangeable guarantees against lost writes.

A slow or paused member can resemble a failed one. SWIM separates failure detection from infection-style dissemination and adds suspicion so a member can refute a suspected failure with a newer incarnation. This reduces false declarations under its assumptions; it does not provide a perfect failure detector during arbitrary delays. Re-replication and ownership changes are decisions of the surrounding system, not inevitable consequences of every gossip suspicion.

Repair consumes resources too. Full-state exchanges, aggressive intervals, and simultaneous rejoins can overload links or receivers. Bounded fanout, incremental exchange, and controlled recovery help, but costs depend on state size as well as the number of contacted peers. Restart versioning must follow the implementation’s identity rules: resetting a counter can make new announcements appear stale or create ambiguity; a lower counter does not inherently overwrite a higher one.

A missing path blocks gossip. Source-based mechanism illustration. Model values and source-reported observations are not a new runtime test.

Check it quickly

Save this as check_gossip.py and run python3 check_gossip.py. It uses only Python’s standard library and no sockets. Twenty dictionaries model peers; each sweep visits nodes sequentially and performs push-pull merges. A fixed seed reproduces the random choices in this runtime. Knowledge can cross several contacts during one sweep, so this is not a synchronized network-round simulator.

import random

NODES, ROUNDS, SEED = 20, 12, 7

def merge(a, b):
 # One writer per key, with strictly increasing versions in this fixture.
 merged = dict(a)
 for key, (version, value) in b.items():
 if key not in merged or version > merged[key][0]:
 merged[key] = (version, value)
 return merged

def spread(peer_for, nodes=None):
 rng = random.Random(SEED)
 if nodes is None:
 nodes = [{} for _ in range(NODES)]
 nodes[0]["config"] = (1, "blue")
 history = []
 for _ in range(ROUNDS):
 # Sequential contacts: knowledge can travel several hops in one sweep.
 for i in range(NODES):
 peer = peer_for(i, rng)
 if peer == i:
 continue
 united = merge(nodes[i], nodes[peer])
 nodes[i], nodes[peer] = dict(united), dict(united)
 history.append(sum(state.get("config") == (1, "blue") for state in nodes))
 return nodes, history

random_peer = lambda i, rng: rng.randrange(NODES)
nodes, history = spread(random_peer)
assert history[-1] == NODES
assert all(a <= b for a, b in zip(history, history[1:]))
print("random contacts:", history)

# A fixed hub is connected; fixed contacts do not inherently prevent convergence.
_, hub = spread(lambda i, rng: 0)
assert hub[0] == NODES
print("fixed hub after one sweep:", hub[0])

# Two disconnected peer sets retain different views until contacts can cross.
partitioned, split = spread(lambda i, rng: (i // 10) * 10 + rng.randrange(10))
assert split[-1] == 10 and all(not node for node in partitioned[10:])
_, healed = spread(random_peer, partitioned)
assert healed[-1] == NODES
print("partition:", split[-1], "after healing:", healed[-1])

# Dissemination alone cannot repair an ambiguous concurrent-write merge rule.
a, b = {"config": (2, "red")}, {"config": (2, "green")}
assert merge(a, b) != merge(b, a)
print("equal-version conflict is order-dependent: not a multi-writer merge")
print("PASS: dissemination, topology, partition, and merge controls")

The random-contact history is 5, 14, 19, then 20 informed peers, with later sweeps staying at 20. That reproduces the original single-update observation, which was recorded with CPython 3.12 on macOS. It is evidence about this input and schedule, not a probability bound for other seeds or deployed clusters. The current model adds explicit counterexamples rather than inferring reliability from that successful run alone.

Every node contacting the same live hub spreads the update to all twenty in one sweep. Fixed partners therefore do not necessarily stall dissemination; connectivity and scheduling matter. The partition control confines contact to two groups of ten, leaving the second group uninformed. Restoring unrestricted contacts eventually informs everyone in this deterministic run.

Finally, merging equal-version red and green values in opposite orders produces different results. That assertion demonstrates the fixture’s single-writer limitation instead of silently calling it general conflict resolution. No network loss, suspicion timeout, tombstone lifecycle, bandwidth budget, or real membership implementation is exercised. Measure those separately before choosing operational convergence targets.

Sources

Dynamo: Amazon’s highly available key-value store, gossip-based membership and anti-entropy with version reconciliation, checked 2026-09-12.
SWIM: Scalable Weakly-consistent Infection-style process group Membership protocol, suspicion mechanism and incarnation numbers for failure detection, checked 2026-09-12.
Cassandra internode communications (gossip), peer-to-peer state exchange with versioned gossip messages, checked 2026-09-12.

Consistent hashing shrinks the rebalance

Sun, 19 Sep 2027 00:00:00 +0000

Consistent hashing limits placement changes when membership changes; it does not itself move data, make replicas agree, or distribute a hot key’s requests. In the classic clockwise ring, a key belongs to the first node point at or after its hash position, wrapping to the smallest point when necessary.

Get the ownership interval right

A point owns the interval after its predecessor through itself: (predecessor, point]. With points A at 20, B at 50, and C at 80, B owns positions above 20 through 50. A key beyond 80 wraps to A. This is not the arc from B to its successor.

Adding D at 35 takes positions above 20 through 35 from B, the new point’s clockwise successor. Removing B transfers B’s former interval to C. Unaffected points retain their positions and ownership. That preservation is the useful property; saying “only a small arc moves” is insufficient when an existing arc might already cover a large fraction of the ring.

By contrast, directly choosing node index hash(key) modulo node_count changes the divisor when a node is added. Many keys then choose another index. A balanced ring with four equal-capacity nodes expanded to five would ideally move about one fifth of keys, but exact movement depends on the actual points and key distribution, not a universal percentage.

Virtual nodes spread intervals, not individual keys

Assigning many ring points to each physical node usually reduces imbalance from a few randomly placed points. Each physical node owns several smaller intervals, and adding another can take pieces from multiple neighbors. Weights can allocate different point counts for unequal capacity. More points also cost routing-table memory and maintenance work.

One key still resolves to one physical owner in this primary-placement model. If that key receives most requests, increasing virtual-node count does not split those requests across machines. Even equal key counts do not imply equal bytes, request rates, or processing cost. Replication, read distribution, and hot-key handling require additional policies.

Stability alone also does not imply balance. All participants must agree on membership, persistent node identities, key encoding, hash rules, and point counts. A stable digest helps them reproduce positions, but cannot compensate for different membership views or a skewed workload. Rebuilding every node’s points on each membership change would defeat the placement-preservation property.

Placement is not migration or consistency

A new routing answer does not transfer cached or persistent values to the new owner. A cache may experience misses and an origin-load spike; a persistent store needs a migration and ownership protocol. Clients using old and new rings can disagree during rollout. Recovery and synchronization behavior belongs to the surrounding system.

Real products also choose different algorithms and metadata. Redis Cluster uses fixed hash slots whose ownership is reassigned; it is not this example’s direct node-point ring. Cassandra adds replica placement and token-management rules, while Envoy uses hashing to select upstream hosts. Do not infer one product’s failure guarantees from another product’s placement diagram.

When it breaks

A single point per node can be badly unbalanced, and renaming a node can unexpectedly relocate its points. Changing byte encoding or hash configuration can likewise remap keys without any physical machine joining. Record the mapping inputs as part of deployment configuration and compare ownership before rollout.

Measure moved keys separately from moved bytes and real traffic. Virtual nodes improve statistical placement, not a guaranteed maximum load. During membership churn, budget for migration work or cold-cache demand, and verify how clients learn the new view. A placement experiment is not an availability or data-integrity test.

Only the affected interval moves. Source-based mechanism illustration. Model values and source-reported observations are not a new runtime test.

Check it quickly

Save the following as ring_check.py and run python3 ring_check.py with Python 3.10 or newer. This ran with Python 3.14.4. It uses only the standard library, starts no servers, and performs no network or filesystem writes. SHA-256 and explicit UTF-8 inputs make the generated fixture reproducible.

from bisect import bisect_left
from collections import Counter
from hashlib import sha256


def token(value):
 return int.from_bytes(sha256(value.encode("utf-8")).digest(), "big")


def ring(nodes, copies):
 entries = sorted(
 (token(f"node:{node}:{copy}"), node)
 for node in nodes for copy in range(copies)
 )
 positions, owners = zip(*entries)
 assert len(set(positions)) == len(positions), "fixture token collision"
 return positions, owners


def lookup(layout, position):
 positions, owners = layout
 return owners[bisect_left(positions, position) % len(positions)]


simple = ((20, 50, 80), ("A", "B", "C"))
joined = ((20, 35, 50, 80), ("A", "D", "B", "C"))
removed = ((20, 80), ("A", "C"))
assert [lookup(simple, value) for value in (0, 20, 21, 50, 51, 80, 81)] == [
 "A", "A", "B", "B", "C", "C", "A",
]
for value in range(100):
 assert lookup(joined, value) == ("D" if 20 < value <= 35 else lookup(simple, value))
 assert lookup(removed, value) == ("C" if 20 < value <= 50 else lookup(simple, value))
assert lookup(joined, 35) == "D"
assert lookup(removed, 50) == "C"
print("a point owns (predecessor, point]; lookups wrap clockwise")

nodes = ("A", "B", "C", "D")
positions = [token(f"key:{index}") for index in range(20000)]
base = ring(nodes, 256)
expanded = ring(nodes + ("E",), 256)
contracted = ring(tuple(node for node in nodes if node != "B"), 256)
assert base == ring(tuple(reversed(nodes)), 256)
before = [lookup(base, position) for position in positions]
after_add = [lookup(expanded, position) for position in positions]
after_remove = [lookup(contracted, position) for position in positions]
add_count = sum(old != new for old, new in zip(before, after_add))
remove_count = sum(old != new for old, new in zip(before, after_remove))
assert 0 < add_count < len(positions)
assert add_count == after_add.count("E")
assert all(old == new or new == "E" for old, new in zip(before, after_add))
assert remove_count == before.count("B")
assert all(old == new or old == "B" for old, new in zip(before, after_remove))
modulo_count = sum(
 nodes[position % 4] != (nodes + ("E",))[position % 5]
 for position in positions
)
print(f"add E: {add_count}/20000 keys move ({add_count / 200:.2f}%)")
print(f"remove B: {remove_count}/20000 keys move ({remove_count / 200:.2f}%)")
print(f"modulo 4 -> 5: {modulo_count}/20000 keys move ({modulo_count / 200:.2f}%)")

single = ring(nodes, 1)
for label, layout in (("one point", single), ("256 points", base)):
 counts = Counter(lookup(layout, position) for position in positions)
 print(label, {node: counts[node] for node in nodes})

hot_owner = lookup(base, token("key:hot"))
synthetic_requests = Counter(before)
synthetic_requests[hot_owner] += 100000
assert synthetic_requests[hot_owner] == before.count(hot_owner) + 100000
print("100000 extra requests for one hot key all target", hot_owner)
print("synthetic request counts", dict(sorted(synthetic_requests.items())))

The integer example checks both interval endpoints and wraparound across every position from zero through 99. For 20,000 deterministic keys with 256 points per node, adding E moves 4,023 keys, removing B moves 4,895, and modulo resizing moves 16,013. Assertions also require every addition-related move to end at E and every removal-related move to originate at B.

The printed counts show this fixture’s improved balance with 256 points, not a universal bound. Assigning 100,000 extra synthetic requests to one hot key still directs them all to D. These are calculated placement and demand counts, not measured production requests, transferred bytes, or recovery timings.

Sources

Apache Cassandra’s Dynamo architecture, clockwise ownership, token ranges, and virtual nodes, checked 2026-09-08.
Envoy load-balancing algorithms, ring hashing, stable host identity, weights, and distribution, checked 2026-09-08.
Python hashlib, portable SHA-256 digest construction, checked 2026-09-08.
Redis Cluster specification, fixed slot mapping as a distinct placement design, checked 2026-09-08.

Vector clocks distinguish causal order from concurrency

Sat, 18 Sep 2027 00:00:00 +0000

A vector records progress for each tracked participant. Comparing these counters distinguishes causal order from concurrency; the application still decides how to reconcile concurrent values.

One timestamp is not enough

A scalar timestamp can impose an order, but that order alone cannot prove causality. With stable participant identities and the usual event-clock rules, each process increments its own component for a local event. Receiving a message merges components by maximum and then increments the receiver’s component. These rules record information flow. They do not measure elapsed time or establish a physical-time ordering between machines that exchanged no information.

Compare vectors to see causality

Compare every component, treating an absent participant as zero. A is strictly before B only when all A components are at most B and at least one is smaller. Reverse domination means after; identical components mean equal. Only when neither vector dominates the other are they concurrent. Because a causal order accumulates everywhere the two paths crossed, a vector captures the transitive and direct edges that a single scalar time forfeits. The moment a message passes from A to B, B’s vector reflects both B’s own progress and everything A had seen, which is how the model traces causality instead of guessing it.

Concurrent means someone decides

Concurrency does not select a winner or necessarily imply a business conflict. Independent field updates might merge; incompatible replacements might require retaining siblings or asking a user. A deterministic tie-breaker can deliberately discard one value, but that is an application policy, not causality discovered by the clock. Equality means equal represented context; only an additional event-identity contract makes it proof that two payloads are the same event.

When it breaks

Clock size grows with the identities tracked by the chosen scheme. In Dynamo, components identify coordinating storage nodes, not every client that ever touched a key. Ephemeral participant identities and counter reuse after restart need explicit handling: reusing an old identity with a reset counter can make genuinely new work look obsolete. Stable identity is part of the algorithm’s assumptions, not an incidental implementation detail.

Pruning a component can lose a real ancestor relationship. Compare ancestor [A:2, C:4] with descendant [A:3, B:2, C:5]. Remove C from the descendant and the remaining vector now appears concurrent with the ancestor. Missing-as-zero is meaningful for an unseen participant; it cannot restore deliberately discarded history. Dynamo explicitly notes reconciliation imprecision from truncation. Dotted version vectors use a different causal representation; they are not permission to delete arbitrary components safely.

Comparing only sums, maxima, or nearby wall-clock values also loses information. [A:2, B:1] and [A:1, B:2] have equal sums but are concurrent. Carry the read context into a subsequent write so the store knows which versions it supersedes. Omitting that context can create siblings depending on the store and current versions; it does not guarantee a new sibling on every write. Retention and reconciliation require the store’s actual protocol.

Concurrent histories need a policy. Source-based mechanism illustration. Model values and source-reported observations are not a new runtime test.

Check it quickly

Save this as check_vector_clocks.py and run python3 check_vector_clocks.py. This in-memory comparison model covers all four relations, a merge followed by a receive-event increment, and a pruning counterexample. It creates no network, store, or persistent participant identity.

def compare(a, b):
 """Compare causal contexts with absent participants interpreted as zero."""
 nodes = set(a) | set(b)
 le = all(a.get(n, 0) <= b.get(n, 0) for n in nodes)
 ge = all(a.get(n, 0) >= b.get(n, 0) for n in nodes)
 if le and ge:
 return "equal"
 if le:
 return "before"
 if ge:
 return "after"
 return "concurrent"


def merge(a, b):
 return {n: max(a.get(n, 0), b.get(n, 0)) for n in set(a) | set(b)}


def tick(clock, node):
 result = dict(clock)
 result[node] = result.get(node, 0) + 1
 return result


a = {"A": 1}
b = tick(a, "B")
assert compare(a, b) == "before" and compare(b, a) == "after"
assert compare(a, a) == "equal"
assert compare({"A": 1}, {"A": 1, "B": 0}) == "equal"
assert a == {"A": 1} # tick does not mutate the input
c, d = {"A": 2, "B": 1}, {"A": 1, "B": 2}
assert compare(c, d) == compare(d, c) == "concurrent"
assert sum(c.values()) == sum(d.values()) # equal sums do not mean equal history
m = merge(c, d)
assert m == {"A": 2, "B": 2}
assert compare(c, m) == compare(d, m) == "before"
received = tick(m, "B")
assert received == {"A": 2, "B": 3} and compare(m, received) == "before"

# Truncation loses a genuine ancestor relationship.
ancestor = {"A": 2, "C": 4}
descendant = {"A": 3, "B": 2, "C": 5}
pruned = {"A": 3, "B": 2}
assert compare(ancestor, descendant) == "before"
assert compare(ancestor, pruned) == "concurrent"
print("PASS: four relations, merge-then-tick, zero slots, and false concurrency after pruning")

All assertions pass on Python 3.12.14. The merged concurrent histories are [A:2, B:2]; recording B’s receive event produces [A:2, B:3]. The pruning test changes a genuine before relation into concurrent, demonstrating a concrete loss of information rather than merely observing a shorter vector.

The model assumes valid nonnegative counters and a shared identity scheme. It does not test crash recovery, membership churn, client/server context exchange, or dotted versions. Those are separate integration properties. Passing comparison arithmetic cannot prove that a distributed store increments, persists, or retires its metadata correctly.

Sources

Dynamo: Amazon’s highly available key-value store, reconciliation with version vectors and sibling retention, checked 2026-10-15.
Preguiça et al.: Dotted Version Vectors, causal histories with bounded server identities and explicit update dots, checked 2026-10-15.

Idempotency keys scope one logical action

Mon, 12 Oct 2026 00:00:00 +0000

An idempotency key identifies one logical action only within a service’s defined scope and retention window. Reusing the same text is not a universal exactly-once guarantee. The service must associate that identity with the right principal and operation, coordinate competing attempts, and retain enough authoritative information to recognize the action later.

The diagram follows the SQLite fixture, not a universal API contract. Scope and retention are service-defined. Unknown state is not absence; database atomicity excludes external effects and does not promise permanent exactly-once execution.

Scope is part of the identity

A key looked up globally can accidentally merge unrelated tenants or expose another caller’s stored result. Authenticate and authorize before consulting or replaying the record. Build the lookup identity from verified scope, a stable operation identifier, and the client’s action key; preserve component boundaries instead of ambiguous string concatenation.

Which dimensions belong in scope is an API decision. EC2 explicitly distinguishes regional and zonal idempotency. The same client token can permit separate actions in different regions or availability zones under those documented rules. Do not assume that moving a retry to another region preserves a global deduplication promise.

Request meaning matters too. The same scoped key with different parameters must follow a declared conflict policy. The example fingerprints one positive integer quantity; it does not compare arbitrary HTTP bodies, perform authentication, or infer business equivalence from a hash.

Couple the effect to its receipt

A lookup followed by an independent write can race: two workers both see no record and both execute. A unique key on the receipt table is insufficient if both business effects have already occurred before one receipt insert fails. The effect and remembered result need an appropriate atomic boundary.

When both live in one transactional database, they can be committed together. This fixture uses SQLite BEGIN IMMEDIATE before lookup and writes its business row and receipt in that transaction. SQLite permits one simultaneous writer, so competing connections serialize; the composite primary key also expresses the lookup identity.

The transaction’s protection ends at that database. An external payment, email, or message publication does not roll back because a local transaction fails. Such effects need their own end-to-end delivery and deduplication contract. Calling the whole workflow exactly once does not supply one.

Retention is a correctness boundary

If a remembered key expires while an old request can still return, that request may execute again. Stripe API v1 documents that a key reused after its record is pruned creates a new request; its minimum twenty-four-hour retention statement is provider-specific, not a universal timer.

Keep recovery windows and retention aligned, and distinguish a temporary deduplication receipt from a lasting business uniqueness rule. Expiring a receipt does not undo the original order. If the business forbids that order from ever being recreated, it needs an authoritative invariant that outlives a short retry cache.

Restoring an older receipt database, clearing state, or switching to an independent store can also forget completed work. Preserve the relationship between business data and operation records when planning recovery, not merely their individual file availability.

When it breaks

A lost response after commit leaves the client uncertain but does not remove the recorded result. A failed transaction before commit should leave neither a business effect nor a successful receipt. Those are different cases and require different evidence.

Do not treat database contention, unavailable storage, or unknown outcomes as an empty lookup. Nor should an in-progress attempt automatically permit another effect. Define whether duplicates wait, receive an in-progress response, or use another documented recovery path; successful replay is only one possible state.

Check it quickly

Save the block as scope_check.py and run python3 scope_check.py with Python 3.10 or newer. The unchanged exercise was rerun with Python 3.12.14 and SQLite 3.53.1 on 2026-10-18. The standard-library fixture creates a database in a private temporary directory and removes it afterward. Each attempt opens its own connection; tenant and operation inputs represent already verified fixture identities.

from concurrent.futures import ThreadPoolExecutor
from contextlib import closing
import json
from pathlib import Path
import sqlite3
from tempfile import TemporaryDirectory
from threading import Barrier


with TemporaryDirectory(prefix="idempotency-scope-") as directory:
 database = Path(directory) / "orders.sqlite"
 with closing(sqlite3.connect(database)) as db:
 db.executescript("""
 CREATE TABLE effects (
 id INTEGER PRIMARY KEY, tenant TEXT NOT NULL,
 operation TEXT NOT NULL, quantity INTEGER NOT NULL CHECK (quantity > 0)
 );
 CREATE TABLE receipts (
 tenant TEXT NOT NULL, operation TEXT NOT NULL, request_key TEXT NOT NULL,
 fingerprint TEXT NOT NULL, result TEXT NOT NULL, expires_at INTEGER NOT NULL,
 PRIMARY KEY (tenant, operation, request_key)
 );
 """)

 def execute(tenant, operation, key, quantity, now, fail_before_receipt=False):
 # Tenant/operation are already authenticated/routed fixture identifiers.
 assert type(quantity) is int and quantity > 0
 fingerprint = json.dumps({"quantity": quantity}, separators=(",", ":"))
 scope = (tenant, operation, key)
 with closing(sqlite3.connect(database, timeout=5)) as db:
 with db:
 db.execute("BEGIN IMMEDIATE")
 previous = db.execute(
 "SELECT fingerprint, result, expires_at FROM receipts "
 "WHERE tenant=? AND operation=? AND request_key=?", scope,
 ).fetchone()
 if previous is not None and previous[2] > now:
 if previous[0] != fingerprint:
 return "conflict", None
 return "replay", json.loads(previous[1])
 if previous is not None:
 db.execute("DELETE FROM receipts WHERE tenant=? AND operation=? AND request_key=?", scope)
 cursor = db.execute(
 "INSERT INTO effects (tenant, operation, quantity) VALUES (?, ?, ?)",
 (tenant, operation, quantity),
 )
 result = {"effect_id": cursor.lastrowid, "quantity": quantity}
 if fail_before_receipt:
 raise RuntimeError("injected failure after effect insertion, before receipt insertion")
 db.execute("INSERT INTO receipts VALUES (?, ?, ?, ?, ?, ?)",
 (*scope, fingerprint, json.dumps(result), now + 60))
 return "created", result

 def counts():
 with closing(sqlite3.connect(database)) as db:
 return db.execute("SELECT (SELECT count(*) FROM effects), (SELECT count(*) FROM receipts)").fetchone()

 gate = Barrier(8)

 def attempt(_):
 gate.wait(timeout=5)
 return execute("north", "web-order", "intent-a", 2, 0)

 with ThreadPoolExecutor(max_workers=8) as pool:
 outcomes = list(pool.map(attempt, range(8)))
 assert [kind for kind, _ in outcomes].count("created") == 1
 assert [kind for kind, _ in outcomes].count("replay") == 7
 original = outcomes[0][1]
 assert all(result == original for _, result in outcomes) and counts() == (1, 1)
 print("eight independent connections produce one effect, one receipt, and seven replays")

 assert execute("north", "web-order", "intent-a", 3, 1) == ("conflict", None)
 assert counts() == (1, 1)
 assert execute("south", "web-order", "intent-a", 2, 1)[0] == "created"
 assert execute("north", "batch-order", "intent-a", 2, 1)[0] == "created"
 assert execute("north", "web-order", "intent-b", 2, 1)[0] == "created"
 assert counts() == (4, 4)
 print("changed parameters conflict; a different tenant, operation, or intent key has separate scope")

 try:
 execute("north", "web-order", "rolled-back", 1, 2, fail_before_receipt=True)
 except RuntimeError as error:
 assert str(error).startswith("injected failure")
 else:
 raise AssertionError("expected the transaction to fail")
 assert counts() == (4, 4)
 assert execute("north", "web-order", "rolled-back", 1, 2)[0] == "created"
 assert counts() == (5, 5)
 print("a raised exception rolls back the effect and receipt together; retry then creates one effect")

 _, committed = execute("north", "web-order", "committed-no-reply", 1, 3)
 # Discard the reply, then query through a fresh connection; no network loss is simulated here.
 assert execute("north", "web-order", "committed-no-reply", 1, 3) == ("replay", committed)
 assert counts() == (6, 6)
 print("a fresh connection replays a committed result without adding another effect")

 assert execute("north", "web-order", "intent-a", 2, 59) == ("replay", original)
 kind, after_expiry = execute("north", "web-order", "intent-a", 2, 60)
 assert kind == "created" and after_expiry["effect_id"] != original["effect_id"]
 assert counts() == (7, 6)
 print("at the explicit 60-second retention boundary, the same key creates a new effect")
 print("SQLite", sqlite3.sqlite_version, "verified counts: 7 effects, 6 retained receipts")

Eight competing connections produce one effect, one receipt, and seven replays with identical results. Changed quantity conflicts. Changing tenant, operation, or action key creates separate work under the declared scope, reaching four effects and four receipts.

An exception after inserting an effect but before its receipt rolls back both. Retrying then creates one effect. A committed result is available through a fresh connection without adding work; this discards an application reply but does not simulate network loss.

Injected times verify replay at second fifty-nine and a new effect at the exact sixty-second expiration boundary. Final counts are seven effects and six retained receipts: the expired record disappears, not its original business row.

The sixty-second window is an explicit test policy, not a recommendation. This experiment does not test process crashes, power loss, distributed databases, external side effects, authentication, or production contention. BEGIN IMMEDIATE can still fail with database-busy errors outside the small successful workload.

Sources

EC2 API idempotency, regional and zonal scope plus parameter mismatch behavior, checked 2026-09-08.
Stripe: Idempotent requests, API v1 retention, pruning, parameter checks, and concurrent-request behavior, checked 2026-10-15.
SQLite transactions, one simultaneous writer, BEGIN IMMEDIATE, and SQLITE_BUSY, checked 2026-09-08.
SQLite atomic commit, the atomic boundary around database changes and its storage assumptions, checked 2026-09-08.
Python sqlite3, connection context-manager commit/rollback and explicit connection closing, checked 2026-09-08.

Distributed · XB Field Notes

etcd elects a leader so writes happen in order

One leader per term, with voting restrictions

Count voters rather than processes

When it breaks

Check it quickly

Sources

Raft elections need a majority of voters

One leader per term

Authority differs from local belief

Terms and timers have different jobs

When it breaks

Check it quickly

Further reading

Sources

Gossip converges under explicit assumptions

Separate dissemination from reconciliation

When it breaks

Check it quickly

Further reading

Sources

Consistent hashing shrinks the rebalance

Get the ownership interval right

Virtual nodes spread intervals, not individual keys

Placement is not migration or consistency

When it breaks

Check it quickly

Further reading

Sources

Vector clocks distinguish causal order from concurrency

One timestamp is not enough

Compare vectors to see causality

Concurrent means someone decides

When it breaks

Check it quickly

Further reading

Sources

Idempotency keys scope one logical action

Scope is part of the identity

Couple the effect to its receipt

Retention is a correctness boundary

When it breaks

Check it quickly

Further reading

Sources