Maintenance · XB Field Notes

A PodDisruptionBudget limits eviction-based disruption

Sat, 10 Apr 2027 00:00:00 +0000

A PodDisruptionBudget limits eviction-based disruption for a selected group of Pods. It can help preserve serving capacity or quorum during maintenance, but cannot guarantee either. Its protection depends on the caller using the Eviction API, the selector identifying the intended members, and Ready conditions meaning something useful for the application.

Choose the permitted disruption from the application’s failure model. A three-member majority system normally needs two functioning members, but Pod readiness does not itself prove membership, synchronized state, or connectivity between those members. The PDB complements application health and maintenance procedures; it does not replace them.

A PDB constrains a specific eviction path. Verify the selected members, readiness, policy and application health before treating an allowance as safe maintenance capacity.

The boundary is eviction

Node crashes, network partitions, and node-pressure eviction are not prevented by PDBs. Unavailable members can nevertheless reduce the remaining disruption allowance. Directly deleting a Pod or workload bypasses PDB enforcement, as do a workload controller’s normal rolling-update deletions. Voluntary does not automatically mean budget-protected.

A normal drain uses eviction requests and retries when protection blocks them. A caller that chooses direct deletion bypasses that protection. Review the maintenance tool’s behavior before relying on a budget. Deployment and StatefulSet rolling availability is controlled by their own update strategies, so coordinate rollouts and node maintenance instead of assuming one PDB serializes everything.

Read the arithmetic with its denominator

Set either minAvailable or maxUnavailable. For three healthy replicas, an integer minAvailable of two allows one healthy eviction under ordinary settled conditions. maxUnavailable of one expresses an allowance relative to intended scale. CurrentHealthy counts qualifying Ready Pods, excluding members being deleted or already recorded for disruption; Running alone is insufficient.

ExpectedPods is not always the controller’s desired replicas. In the reviewed Kubernetes 1.34.0 implementation, integer minAvailable uses the number of selected Pod objects. Percentage minAvailable and maxUnavailable derive expected scale from owning controllers. Therefore a mismatch with StatefulSet replicas can reflect policy type or reconciliation state, not only a selector typo.

The status allowance is broadly currentHealthy minus desiredHealthy, floored at zero, with additional safety checks and outstanding-eviction accounting. Inspect observedGeneration and conditions when interpreting it. Zero is a current decision, not a diagnosis. It can mean an intentionally strict policy, unavailable Pods, an in-progress eviction, or a controller synchronization problem.

Percentages round upward. Half of seven requires four healthy Pods, while maxUnavailable thirty percent on one desired replica permits losing that single replica. Percentage notation does not automatically make a small workload safe. Check the smallest scale your application can reach, including autoscaling and maintenance changes.

Health policy changes drain behavior

Under the default IfHealthyBudget policy, a Running but unready Pod can be evicted when currentHealthy is at least desiredHealthy. If the application is already below that threshold, such a Pod can block a drain while it has a chance to recover. The PDB sees its unhealthy state; it is not blind to the Pod.

AlwaysAllow permits eviction of those Running-but-unhealthy Pods regardless of that health threshold. It does not authorize unrestricted eviction of healthy Pods. Select the policy with recovery and maintenance needs in mind, especially when a broken readiness check could keep a replacement from ever becoming healthy.

For three healthy voting members, allowing one disruption is a common conservative choice. Recalculate quorum and availability when membership changes; an allowance of one is not suitable for every possible scale or membership protocol. Maintenance may wait for a replacement to become genuinely ready before another healthy eviction fits within the budget.

When it breaks

A minimum equal to all available replicas can deliberately block healthy evictions, rather than necessarily being a mistake. Document how maintenance should obtain room. Do not remove the budget simply because a drain is slow: first check replacement scheduling, readiness, existing disruption, and the application owner’s availability requirement.

A selector matching nothing protects nothing. Conversely, an empty policy/v1 selector matches every Pod in its namespace, which can broaden scope unexpectedly. Compare the complete selector, including matchExpressions, against live labels and owning controllers. Avoid treating a single status number as proof that the right application is protected.

Check it quickly

Inspect existing objects in the intended read-only context and namespace. These original commands were retained and were not rerun on a cluster during this review.

kubectl get pdb -A -o custom-columns=NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,EXPECTED:.status.expectedPods,HEALTHY:.status.currentHealthy,ALLOWED:.status.disruptionsAllowed
kubectl get pdb zk-pdb -o jsonpath='{.spec.selector.matchLabels}{"\n"}'
kubectl get statefulset zookeeper -o jsonpath='{.spec.replicas}{"\n"}'
kubectl get pods -l app=zookeeper --field-selector=status.phase=Running --no-headers | wc -l

The final command counts Running Pods, not Ready Pods. The selector command displays only matchLabels, so it omits any matchExpressions. Use those outputs as initial clues and inspect complete PDB and Pod objects before concluding that HEALTHY or EXPECTED is wrong. A zero allowance does not reveal its cause, and positive allowance does not prove application-level quorum survives an eviction.

Sources

Configure a PodDisruptionBudget, selectors, rounding and unhealthy-Pod policies, checked 2026-10-15.
Pod disruptions, eviction boundaries, involuntary failures and rolling updates, checked 2026-10-15.
Kubernetes 1.34.0 disruption controller, getExpectedPodCount and countHealthyPods distinguish integer minimums, controller scale and outstanding disruptions, checked 2026-10-15.

Vacuum keeps a growing database from bloating

Sun, 16 Aug 2026 00:00:00 +0000

In multi-version storage, deleting a row records the change without immediately freeing the space. Reclaiming that space is maintenance work, and when it is skipped the cost shows up slowly.

Deleted does not mean gone

An UPDATE or DELETE leaves old PostgreSQL row versions until they are no longer needed by any relevant snapshot. A long transaction can delay that point while other sessions keep changing rows. The transaction alone does not manufacture new bloat in an otherwise idle table. Reclaiming obsolete versions and making their space reusable is distinct from merely committing the delete.

Ordinary VACUUM usually leaves table size unchanged: it frees space inside existing files for future rows. Empty pages at the end can sometimes be truncated, but a general compaction needs a rewrite such as VACUUM FULL, with additional space and stronger locking. A stable file size after successful vacuum is therefore not evidence that cleanup failed. Distinguish live data, reusable internal space, and space returned to the operating system.

Reclaiming and statistics are separate jobs

Autovacuum has a launcher and multiple workers, not one worker that cleans everything on a fixed schedule. It decides whether tables need VACUUM or ANALYZE using activity estimates and age-related rules. ANALYZE collects planner distributions independently; plain VACUUM is not a substitute for refreshing those statistics. Insert-driven and wraparound-related work also mean dead-tuple count is not the only reason a table needs maintenance.

Cost-based delay throttles vacuum through pauses after estimated I/O work. The cost limit is not a maximum number of tuples one invocation can remove before it stops. Thresholds decide eligibility, worker availability affects scheduling, and storage capacity and throttling affect progress. Diagnose which stage is limiting the table rather than treating every slow cleanup as the same tuning problem.

When it breaks

Churn can outpace cleanup. Dead-tuple estimates and relation sizes then rise across several observation windows. Inspect completed runs, cancellations, worker availability and I/O alongside the write rate. Extra hardware can help an actual resource bottleneck, but restarting the database does not by itself make retained versions reusable. Neither a low threshold nor a high worker count guarantees a better result under the same I/O limit.

Alternatively, cleanup may run but leave particular old versions because a snapshot or retained xmin still needs them. Check transaction and replication state before intervening. An old transaction start time is a clue, not proof that it holds the relevant horizon. Ending an application transaction or removing a replication slot can disrupt work or require rebuilding a consumer; identify its owner and recovery requirements first.

Finally, do not diagnose bloat solely from one n_dead_tup value or a slow plan. These counts are estimates, and page layout, visibility, query selectivity and stale statistics can all change observed reads. Compare trends, vacuum reports and the actual plan. A full-size file with reusable space may be healthy, while a small count can still hide a retention or freezing problem.

Container levels are qualitative. Model work units, cadence and capacity are invented; they are neither PostgreSQL measurements nor vacuum cost-limit semantics. The model was not run.

Check it quickly

Save the following as vacuum_model.py and run it with python3 vacuum_model.py. It uses only the standard library to contrast a trigger, finite cleanup throughput, and an entirely pinned backlog. Work units and check intervals are invented. The base-plus-scale trigger borrows one idea from autovacuum; it omits PostgreSQL 18’s maximum threshold, insert triggers, freezing and scheduling. RECLAIM_CAPACITY is a toy throughput allowance, not vacuum_cost_limit.

"""Illustrative cleanup-backlog model, not a PostgreSQL vacuum simulator."""

BASE_THRESHOLD = 50
SCALE_FACTOR = 0.2
CHECK_INTERVAL = 2
RECLAIM_CAPACITY = 1600 # arbitrary work units per model cleanup opportunity
LIVE_TUPLES = 10_000
THRESHOLD = BASE_THRESHOLD + SCALE_FACTOR * LIVE_TUPLES


def needs_vacuum(dead):
 return dead > THRESHOLD


def run_workload(*, churn_per_tick, ticks, pinned=False):
 dead = 0
 runs = 0
 max_dead = 0
 for tick in range(1, ticks + 1):
 dead += churn_per_tick # each update leaves one dead version
 max_dead = max(max_dead, dead)
 if tick % CHECK_INTERVAL == 0 and needs_vacuum(dead):
 runs += 1
 if not pinned:
 dead = max(0, dead - RECLAIM_CAPACITY)
 return {"dead": dead, "runs": runs, "max_dead": max_dead}


def main():
 healthy = run_workload(churn_per_tick=400, ticks=20)
 assert healthy["runs"] >= 1, "healthy workload should trigger vacuum"
 assert healthy["dead"] < THRESHOLD, healthy
 assert healthy["max_dead"] < 3000, healthy

 overloaded = run_workload(churn_per_tick=1500, ticks=20)
 assert overloaded["runs"] >= 1, "overloaded workload still triggers vacuum"
 assert overloaded["dead"] > THRESHOLD, overloaded

 pinned = run_workload(churn_per_tick=400, ticks=20, pinned=True)
 assert pinned["runs"] >= healthy["runs"], (pinned, healthy)
 assert pinned["dead"] == 400 * 20, pinned
 assert pinned["dead"] > healthy["dead"], (pinned, healthy)

 print(f"healthy: runs={healthy['runs']} dead={healthy['dead']} max={healthy['max_dead']}")
 print(f"overloaded: runs={overloaded['runs']} dead={overloaded['dead']} max={overloaded['max_dead']}")
 print(f"pinned: runs={pinned['runs']} dead={pinned['dead']}")
 print("PASS: model threshold, limited throughput, and pinned backlog")


if __name__ == "__main__":
 main()

Keep Python assertions enabled; expect passing assertions and a PASS line. The healthy workload triggers several runs and ends below the threshold. The overloaded workload triggers on every check yet ends far above it, which is the falling-behind signature. The pinned workload runs more often but reclaims nothing. The original article reports both copies passing on Python 3.12.14; this source review did not rerun the model or recover that run’s raw log: healthy ends with 1,600 units, overloaded with 14,000, and pinned with 8,000. These are model outputs, not database measurements. Real vacuum does not stop after reclaiming this invented allowance, and a real old snapshot need not block every dead version. Use the model only to distinguish insufficient throughput from unavailable work, then collect database evidence for either diagnosis.

Sources

PostgreSQL: routine vacuum, why and when to vacuum, checked 2026-10-15.
PostgreSQL: maintenance, maintaining table health, checked 2026-10-15.
PostgreSQL: monitoring stats, reading activity numbers, checked 2026-10-15.
PostgreSQL 18 vacuum cost delay, pauses based on estimated I/O work, checked 2026-10-15.

A maintenance timer should leave its last result visible

Tue, 04 Aug 2026 00:00:00 +0000

Cron-like syntax is not the whole contract for a maintenance task. The command needs a clear service identity, a bounded runtime, and evidence that distinguishes “not scheduled” from “ran and failed”. A systemd timer and oneshot service keep those concerns separate.

A fired timer is only a scheduling event. Judge this run by its service result, exit status and fresh timestamp.

Define the work as a service

Use an explicit user and absolute command path:

# /etc/systemd/system/site-check.service
[Unit]
Description=Read-only public site check

[Service]
Type=oneshot
User=sitecheck
ExecStart=/usr/local/bin/check-public-site
TimeoutStartSec=2min

TimeoutStartSec limits the oneshot activation, which includes its command runtime; oneshot services otherwise have no start timeout by default. Timeout handling can then require a stop timeout and cannot instantly end uninterruptible kernel I/O, so the helper also needs bounded network requests. The helper should return non-zero when its contract fails and must not modify the service it checks.

Schedule the service

# /etc/systemd/system/site-check.timer
[Unit]
Description=Run the public site check every hour

[Timer]
OnCalendar=hourly
Persistent=true
RandomizedDelaySec=5min
Unit=site-check.service

[Install]
WantedBy=timers.target

Persistent=true triggers one catch-up activation when the timer is activated after missing at least one calendar event; it does not replay every missed hour. The catch-up is still subject to the randomized delay. RandomizedDelaySec spreads routine work across a window; it is not appropriate when the task must run at an exact instant.

Validate before enabling

After creating both unit files, validate them and reload the manager before starting the service manually. These are administrative operations on the intended test host, not read-only checks:

systemd-analyze verify /etc/systemd/system/site-check.service \
 /etc/systemd/system/site-check.timer
systemd-analyze calendar hourly
systemctl daemon-reload
systemctl start site-check.service
systemctl status site-check.service
journalctl -u site-check.service -n 50 --no-pager
systemctl show site-check.service -p Result -p ExecMainCode -p ExecMainStatus

Enable the timer only after the service succeeds with the intended user, environment, paths, and network access:

systemctl enable --now site-check.timer
systemctl list-timers site-check.timer

Alert on the service result

The timer firing is not success. Monitor the oneshot service’s exit status or journal result. Keep stdout concise and send detailed artifacts to a bounded path if they are needed for debugging.

A successful oneshot normally returns to inactive (dead); that alone is not a failure, and systemctl status can return nonzero for an inactive service. Correlate Result and exit status with a fresh journal timestamp so an old successful run is not mistaken for this one. Do not add RemainAfterExit=yes merely to keep it green: a recurring timer does not start a fresh instance while its service is already active. Calendar timing uses the host timezone unless specified, and normal timer accuracy adds scheduling slack beyond the random delay.

Sources

systemd.timer(5), calendar timers, persistence, and delay behavior, checked 2026-08-04.
systemd.time(7), calendar expression syntax, checked 2026-08-04.
systemd.service, oneshot service semantics, checked 2026-08-04.
systemd.service upstream manual (v257), Type=exec/oneshot, readiness, activation timeout, and restart semantics, checked 2026-10-15.
systemd.timer upstream manual (v257), persistence coalesces missed runs; active units, randomized delay, and accuracy, checked 2026-10-15.

Maintenance · XB Field Notes

A PodDisruptionBudget limits eviction-based disruption

The boundary is eviction

Read the arithmetic with its denominator

Health policy changes drain behavior

When it breaks

Check it quickly

Sources

Vacuum keeps a growing database from bloating

Deleted does not mean gone

Reclaiming and statistics are separate jobs

When it breaks

Check it quickly

Further reading

Sources

A maintenance timer should leave its last result visible

Define the work as a service

Schedule the service

Validate before enabling

Alert on the service result

Sources