Graceful degradation preserves a deliberately supported subset of a service when another part fails. An article page might omit recommendations while still showing the requested article. The important word is supported: a partial answer must remain useful and truthful under its product contract. Returning something is not automatically safer than returning an explicit failure.
Choose what may be missing
Identify optional regions and required outcomes before writing fallback code. A missing sidebar can be acceptable; silently bypassing authorization, presenting an uncertain payment as successful, or substituting an unverified price can violate the core promise. Those cases require an explicit failure or a separate approved workflow rather than a generic “keep the page green” rule.
Keep the optional dependency outside the core response’s critical path where possible. Give its call a bounded budget, but remember that a caller timeout does not necessarily stop underlying work. Bound concurrency and queued work as well. A placeholder should name the missing capability without exposing internal errors, and should not imply that fresh data was obtained when it was not.
Make stale-data acceptance explicit
Last-known-good data is useful only within a declared scope and age policy. Record when the data was obtained, preserve its identity and authorization scope, and decide what happens when its age is unknown, excessive, or the cache is empty. A timestamp alone does not establish correctness: some data is unsafe to reuse even briefly after a relevant state change.
Serving stale after an upstream failure is not identical to stale-while-revalidate, which can serve an eligible stale response while validation proceeds. Either approach needs bounded freshness and resource behavior. Successful refresh should actually replace the stored value and its age. Otherwise a service can repeatedly claim to use its latest good result while returning an older snapshot forever.
When it breaks
Fallback may amplify an outage. AWS’s account of avoiding fallback describes a cache failure that redirected traffic into a database unable to absorb it. That source is a warning about difficult, rarely exercised alternate paths, not a blanket endorsement of partial answers. Prefer predictable primary behavior and regularly exercised recovery paths over a complicated fallback whose resource demands are unknown.
Synchronized cache expiry and simultaneous recovery probes can create bursts. Spread refresh work, limit retries and concurrency, and measure whether the fallback competes with the same exhausted resource. A circuit breaker decides whether to call a dependency; the degradation policy decides what acceptable answer remains. An open breaker does not itself prove that a cached response is valid or that a static answer meets the contract.
A failed recovery probe must leave normal admission blocked. Closing a breaker before testing recovery can release traffic too early. Production implementations also need synchronization, probe scheduling, and bounded recovery concurrency; a serial example cannot establish those properties. Degraded responses should have their own count and age observations so that an HTTP 200 does not hide persistent loss of functionality.
Check it quickly
Save this as degrade_probe.py and run python3 degrade_probe.py. The standard-library fixture models one optional recommendation region. It performs no network or filesystem operations. Cache age and dependency failure are injected inputs, and the 120-second limit is an illustrative product decision. Probe calls are explicit and synchronous; no timer or concurrent request admission is implemented.
class DependencyDown(Exception):
pass
class PageService:
def __init__(self):
self.cached_recs = ["cached-pick"]
self.cached_age_s = 90 # Injected age, not a running clock.
self.max_age_s = 120
self.failures, self.threshold = 0, 3
self.open = False
self.live_calls = 0
def live_recs(self, fail):
self.live_calls += 1
if fail:
raise DependencyDown("recs down")
return ["fresh-pick"]
def fresh(self, data):
self.cached_recs, self.cached_age_s = list(data), 0
self.failures, self.open = 0, False
return {"mode": "fresh", "data": data}
def fallback(self, reason):
if self.cached_recs and 0 <= self.cached_age_s <= self.max_age_s:
return {"mode": "cached", "data": list(self.cached_recs),
"age_s": self.cached_age_s, "reason": reason}
return {"mode": "simplified", "data": ["popular-picks"], "reason": reason}
def get_recs(self, fail=False):
if self.open:
return self.fallback("breaker-open")
try:
return self.fresh(self.live_recs(fail))
except DependencyDown:
self.failures += 1
self.open = self.failures >= self.threshold
return self.fallback("dependency-failed")
def half_open_probe(self, fail=False):
# Explicit synchronous trial; not a concurrent admission controller.
assert self.open, "probe requires an open breaker"
try:
data = self.live_recs(fail)
except DependencyDown:
return self.fallback("probe-failed") # Leave the breaker open.
return self.fresh(data) # Close only after success.
svc = PageService()
assert svc.get_recs() == {"mode": "fresh", "data": ["fresh-pick"]}
assert svc.cached_recs == ["fresh-pick"] and svc.cached_age_s == 0
svc.cached_age_s = 120
cached = svc.get_recs(fail=True)
assert cached["mode"] == "cached" and cached["age_s"] == 120
assert cached["data"] == ["fresh-pick"]
svc.cached_age_s = 121
assert svc.get_recs(fail=True)["mode"] == "simplified"
svc.get_recs(fail=True)
assert svc.open
calls = svc.live_calls
assert svc.get_recs()["reason"] == "breaker-open" and svc.live_calls == calls
assert svc.half_open_probe(fail=True)["reason"] == "probe-failed"
assert svc.open and svc.live_calls == calls + 1
calls = svc.live_calls
svc.get_recs()
assert svc.live_calls == calls # A failed probe must not reopen normal admission.
assert svc.half_open_probe()["mode"] == "fresh" and not svc.open
assert svc.cached_age_s == 0
svc.cached_recs = []
assert svc.get_recs(fail=True)["mode"] == "simplified"
svc.cached_recs, svc.cached_age_s = ["unknown-age"], -1
assert svc.get_recs(fail=True)["mode"] == "simplified"
print("PASS: fresh refresh, bounded cache, empty/invalid cache, open gate, failed/successful probe")
A successful call refreshes the cache. The next failure serves that actual latest value when its injected age is exactly 120 seconds. At 121 seconds it uses a clearly simplified answer instead. Three consecutive failures open the call gate, and another ordinary request returns fallback without incrementing the live-call count. These checks distinguish acceptable stale data from merely available stale data.
The failed probe performs one live attempt but leaves the breaker open; a subsequent ordinary request still makes no live call. A successful probe then closes the gate and refreshes the cache. Separate empty-cache and invalid-negative-age controls also choose the simplified response. The final PASS line covers these deterministic decisions, not production availability.
Before deployment, exercise the chosen contract against real dependency failures and recovery under controlled load. Verify deadlines, cancellation, authorization scope, cache aging, and visible degradation metrics independently. The example’s static recommendation fallback is intentionally cheap; it does not demonstrate that an alternate remote service would remain healthy during a correlated outage.
Further reading
Sources
- AWS: Avoiding fallback in distributed systems, risks of infrequent fallback paths and amplified outages, checked 2026-10-15.
- Google SRE: graceful degradation, shedding to keep the useful parts, checked 2026-10-15.
- Martin Fowler: CircuitBreaker, open-state rejection and controlled recovery probes, checked 2026-10-15.
