重试意味着再增加一份工作,不是自动提高可靠性的操作。它可能帮助恢复暂时失败,也可能重复一个不安全的业务行动,或把额外需求送到已经过载的依赖。选择延迟公式之前,应先确定哪些操作允许重复、总共能投入多少次尝试与多长时间,以及这些额外工作由哪一层负责。
圆点为示意取样,不是实际到达时刻。抖动不保证容量或逐次延长等待。检查重放资格、次数、总期限、更广预算和 Retry-After;必需等待放不进期限时应放弃。 先约束尝试次数,再安排时间 需要区分尝试和重试:总共三次尝试,是第一次调用加最多两次重试。还要确定恢复责任属于哪一层。如果嵌套的三层各自允许三次尝试,一次外层行动就可能对持续失败的最底层发起二十七次调用。抖动只能改变这些调用的时间,不能消除次数相乘的结果。
Google SRE 同时讨论单请求预算与更广范围的重试预算。即使每个请求都有限次尝试,大量失败的客户端仍可能制造相当可观的额外工作。应把重试流量与原始需求分别统计,并核对 SDK、代理和应用层已经存在的政策,避免每一层都独立认为自己只增加了少量调用。
指数退避增长的是上限 封顶指数退避让等待上限逐步增大,直到达到最大值。没有抖动时,同时失败的客户端可能算出相同的下次时间,甚至在到达上限后继续保持同样间隔。等待更久,并不必然打破同步,不能把周期变长当成请求已经分散的证据。
完全抖动会在零与当前上限之间采样。因为取值随机,即使上限正在增长,下一次实际等待仍可能比上一次短。因此,“每次失败后一定比之前多等一会儿”不是这种算法的准确描述;需要区分退避上限的变化与本次真正取到的延迟。
AWS 的文章比较了完全抖动、等量抖动和去相关抖动,并报告特定竞争模拟的结果。这些结果能帮助理解取舍,却不保证某种政策在所有工作负载下都具有最低延迟,也不保证完全没有突发。相同随机种子、定时器粒度和大量客户端,仍可能形成聚集。
整条操作共用一个截止期限 预算必须同时包括执行、等待和后续尝试。每次调用都重新拿到完整超时时长,会悄悄扩大原调用方允许的生命周期。应把剩余预算传给下游,并在准备进行更多工作之前再次检查,而不是只限制两次调用之间睡多久。
进程内计算经过时间,应使用单调时钟。客户端超时,也不证明服务器已经停止或从未提交;取消与远端截止期限需要真实的传输层支持,业务仍然需要重试或幂等契约。不能因为本地函数返回错误,就把未知结果当成未执行。
Retry-After 等服务端等待提示会增加另一项约束。不能因为本地抖动公式允许较小值,就采样出早于该边界的请求时间。如果必须等待的时间已经无法放进剩余期限,应在当前操作中放弃重试,而不是为了凑预算缩短服务端要求。
坏掉的样子 永久性的参数验证错误,不会因为重复发送就变得合法。非幂等操作在结果不明时重放,可能再次产生同一业务效果。应明确选择可重试的失败类别;捕获所有异常后循环重试,既会掩盖编程错误,也可能把一次局部故障推成持续过载。
只有退避上限,没有次数或总耗时限制,仍然允许无限工作;但期限过短,又可能让所有昂贵却合法的操作一直失败。需要一起观察完成工作量、重试次数、总延迟和放弃原因。仅看到错误率下降,可能忽略为了获得成功而越来越高的重试成本。
快速验证 保存为 retry_check.py,用 Python 3.10 以上版本执行 python3 retry_check.py。2026-10-19 以 Python 3.12.14 完整运行原样程序并通过,只使用标准库。时钟和下游调用都是确定性模型,不真实等待,不启动服务,也不访问依赖。示例政策只允许 GET、HEAD 在超时或 503 后重试,不是所有 HTTP 客户端行为的实现。
from collections import Counter import random class Clock: def __init__(self): self.now = 0.0 def advance(self, seconds): self.now += seconds class ScriptedCall: def __init__(self, clock, steps): self.clock, self.steps, self.timeouts = clock, iter(steps), [] def __call__(self, timeout): self.timeouts.append(timeout) status, duration = next(self.steps) self.clock.advance(min(duration, timeout)) if duration >= timeout: raise TimeoutError("the cooperative fixture used the remaining budget") return status def retry(call, clock, budget, sample, method="GET", max_attempts=3): deadline = clock.now + budget for attempt in range(1, max_attempts + 1): remaining = deadline - clock.now if remaining <= 0: return "deadline", attempt - 1 try: status = call(remaining) except TimeoutError: status = None if clock.now >= deadline: return "deadline", attempt if status == 200: return "success", attempt if status not in (None, 503) or method not in ("GET", "HEAD"): return "not-retryable", attempt if attempt == max_attempts: return "attempt-limit", attempt ceiling = 2.0 ** min(attempt - 1, 3) delay = ceiling * sample() assert 0 <= delay <= ceiling if delay >= deadline - clock.now: return "wait-exceeds-budget", attempt clock.advance(delay) raise AssertionError("positive attempt limit required") clock = Clock() call = ScriptedCall(clock, [(503, 0.25), (503, 0.25), (200, 0.25)]) assert retry(call, clock, 5, lambda: 0.5) == ("success", 3) assert clock.now == 2.25 and call.timeouts == [5, 4.25, 3] print("three attempts finish at 2.25; propagated timeouts shrink from 5 to 4.25 to 3") class EarlyTimeout: def __init__(self, clock): self.clock, self.timeouts = clock, [] def __call__(self, timeout): self.timeouts.append(timeout) self.clock.advance(0.25) if len(self.timeouts) == 1: raise TimeoutError("a transient failure before the deadline") return 200 clock = Clock() call = EarlyTimeout(clock) assert retry(call, clock, 5, lambda: 0.5) == ("success", 2) assert clock.now == 1.0 and call.timeouts == [5, 4.25] print("an early timeout can recover without resetting the shared deadline") for method, status in (("POST", 503), ("GET", 400)): clock = Clock() call = ScriptedCall(clock, [(status, 0.25)]) assert retry(call, clock, 5, lambda: 0.5, method) == ("not-retryable", 1) clock = Clock() call = ScriptedCall(clock, [(503, 0.25)] * 4) assert retry(call, clock, 5, lambda: 0.5) == ("attempt-limit", 3) assert len(call.timeouts) == 3 print("POST, permanent failure, and the total-attempt cap stop further work") clock = Clock() call = ScriptedCall(clock, [(503, 0.25)] * 7) assert retry(call, clock, 100, lambda: 1.0, max_attempts=7) == ("attempt-limit", 7) assert clock.now == 32.75 # Seven 0.25 calls plus waits 1, 2, 4, 8, 8, 8. print("the exponential ceiling stops at 8 rather than growing without bound") clock = Clock() call = ScriptedCall(clock, [(503, 0.1)]) assert retry(call, clock, 0.5, lambda: 1.0) == ("wait-exceeds-budget", 1) assert clock.now == 0.1 and len(call.timeouts) == 1 clock = Clock() call = ScriptedCall(clock, [(503, 99)]) assert retry(call, clock, 0.5, lambda: 0.5) == ("deadline", 1) assert clock.now == 0.5 clock = Clock() call = ScriptedCall(clock, []) assert retry(call, clock, 0, lambda: 0.5) == ("deadline", 0) and call.timeouts == [] print("a wait that cannot fit is abandoned; cooperative execution cannot consume a fresh budget per attempt") rng = random.Random(7) # Repeatable samples for this experiment, not a production seed. delays = [rng.random() for _ in range(100)] bins = Counter(int(delay * 10) for delay in delays) fixed_bins = Counter(int(1.0 * 10) for _ in range(100)) assert all(0 <= delay < 1 for delay in delays) and sum(bins.values()) == 100 assert max(bins.values()) < 100 assert max(fixed_bins.values()) == 100 print("first-retry 0.1-second-bin peak: fixed delay", max(fixed_bins.values()), "; full-jitter sample", max(bins.values())) leaf_calls = 0 class Unavailable(Exception): pass def nested(depth): global leaf_calls if depth == 0: leaf_calls += 1 raise Unavailable for _ in range(3): try: return nested(depth - 1) except Unavailable: pass raise Unavailable try: nested(3) except Unavailable: pass assert leaf_calls == 27 print("three retrying layers, each allowing three total attempts, invoke the failing leaf 27 times") 两次失败后成功,最终模型时间为 2.25。传给各次调用的超时依次缩短为五秒、4.25 秒和三秒,没有每次重新领取一份完整预算。额外用例确认提前超时仍能在同一预算内恢复;指数先封顶再计算,避免总尝试上限很大时先生成过大的幂。其他断言覆盖总尝试次数、八秒退避上限、永久错误、POST 不自动重放、放不进预算的等待,以及开始时就已经耗尽的期限。
...