Autoscaling · XB Field Notes

The Horizontal Pod Autoscaler depends on metric semantics

Wed, 22 Dec 2027 00:00:00 +0000

A HorizontalPodAutoscaler turns configured metrics into replica recommendations. Choosing the metric and its aggregation is therefore part of application design. Increasing replicas should reduce the measured pressure or increase useful throughput; otherwise the controller may spend more money without relieving the bottleneck. Set explicit minimum and maximum counts, but do not confuse a cost bound with a performance guarantee.

Arithmetic model from the source, not a live observation. Its historical Python-only pass is unchanged. The model omits real-controller tolerance, readiness, stabilization and rate policies; no cluster behavior or availability is established.

Separate measurement from control

HPA normally reconciles periodically, with a default controller sync period of fifteen seconds. For per-Pod average targets, its basic calculation is the ceiling of current replicas multiplied by current metric divided by target metric. CPU utilization is relative to resource requests, rather than the CPU limit or the entire node’s capacity.

Averaging across Pods is not the same as averaging a time series. Metrics Server reports CPU usage measured over a window; custom adapters define their own measurement and query semantics. HPA does not universally smooth every input over a configurable measurement window. Tolerance, readiness handling, stabilization, and rate policies are separate controller mechanisms layered around metric recommendations.

Check the unit before the target number. Per-Pod custom metrics use AverageValue; object and external metrics can use Value or AverageValue with different scaling meaning. A total queue depth and an average backlog per replica are not interchangeable. Resource utilization additionally needs the relevant resource requests. An unsuitable metric can return a perfectly valid number while still measuring the wrong bottleneck.

Pick a signal replicas can improve

CPU often works for CPU-bound services, but availability of the resource metrics API is a prerequisite, not a universal cluster property. Workers blocked on a database can have low CPU and high latency. More workers might make the database problem worse. Test whether the chosen target corresponds to sustainable work per replica before relying on autoscaling.

Request rate, in-flight work, and consumer lag each require explicit scope. For a per-Pod metric, expose the necessary per-Pod observations rather than collapsing everything into one workload series. For an object or external signal, aggregate to the intended workload or queue and exclude unrelated traffic. Avoid unnecessary per-user labels that increase adapter query cost and complicate selection.

Give replica ownership one home

HPA reads the target’s scale information and adjusts its replica count. A Deployment’s initial replicas value is not the HPA minimum; minReplicas is the configured floor. Reapplying a manifest that continually writes a fixed replica count can compete with HPA. Arrange configuration ownership deliberately, including what static count should apply if autoscaling is removed.

When it breaks

An unavailable custom metric does not automatically fall back to CPU. With multiple configured metrics, HPA chooses the largest usable recommendation. If one metric fails and the usable metrics recommend shrinking, it skips that scale-down. A valid recommendation above the current count can still permit scale-up. Thus explicitly adding CPU offers another input, not unconditional failover in both directions.

Distinguish a failed metric source from missing measurements for individual Pods. The latter also causes conservative adjustments in the resource algorithm. Inspect conditions and events rather than asserting that every missing sample must set ScalingActive=False. Successful readings, failed sources, and current bounds can coexist.

Stale adapter data can drive scaling after the relevant work has drained. Inspect timestamps, query scope, caching, and collection latency before retuning the target. A metric API error and a valid-but-irrelevant memory reading are different failures; choosing memory as a poor latency proxy does not itself make the reading Unknown.

Check it quickly

Save this as hpa_metric_sim.py and run python3 hpa_metric_sim.py. The standard-library model uses already-computed per-replica averages and starts inside its bounds. It demonstrates maximum selection and the failed-source scale-down guard. It is not an implementation of every HPA metric type.

import math
import sys


def desired_for(current, value, target):
 assert current > 0 and target > 0
 if value is None:
 return None
 assert value >= 0
 return math.ceil(current * value / target)


def decide(current, metrics, min_replicas, max_replicas):
 assert 0 < min_replicas <= current <= max_replicas
 recommendations = [desired_for(current, value, target)
 for _, value, target in metrics]
 available = [want for want in recommendations if want is not None]
 if not available:
 return current
 want = max(available)
 if None in recommendations and want < current:
 return current
 return max(min_replicas, min(max_replicas, want))


def check(actual, expected, message):
 assert actual == expected, (message, actual, expected)
 print(f"OK: {message} -> {actual}")


def main():
 current = 4
 cpu = ("cpu", 80, 50)
 queue = ("average_queue", 900, 300)
 missing_queue = ("average_queue", None, 300)
 cool_cpu = ("cpu", 10, 50)
 check(decide(current, [cpu, queue], 1, 20), 12, "maximum wins")
 check(decide(current, [cpu, missing_queue], 1, 20), 7, "missing allows scale-up")
 check(decide(current, [cool_cpu, missing_queue], 1, 20), 4, "missing blocks scale-down")
 check(decide(current, [("cpu", None, 50), missing_queue], 1, 20), 4, "all missing holds")
 check(decide(current, [cpu, queue], 1, 10), 10, "upper bound")
 check(decide(current, [cool_cpu, ("average_queue", 30, 300)], 1, 20), 1, "valid low metrics shrink")
 check(decide(current, [cool_cpu], 2, 20), 2, "lower bound")
 print("PASS: 7 model scenarios; no Kubernetes controller was exercised")
 print(sys.version.split()[0])


if __name__ == "__main__":
 main()

The seven assertions produce recommendations of twelve and seven for the hot inputs, hold four when a failed source accompanies a shrink request, and hold four when all sources fail. They also check both bounds and a valid shrink to one. Both language copies passed locally with Python 3.12.14 during this review.

No cluster controller, metrics adapter, tolerance, readiness adjustment, stabilization window, or scaling-rate policy is exercised. The model’s explicit missing-source guard is its learning target. Validate real timestamps, conditions, and scaling events separately before treating arithmetic success as evidence that production autoscaling works.

Sources

Kubernetes Horizontal Pod Autoscaling, documents algorithm and metric sources, checked 2026-10-15.
Resource Metrics Pipeline, documents metrics-server and the resource metrics API, checked 2026-10-15.
HorizontalPodAutoscaler walkthrough, Pods metrics use AverageValue; object/external types differ, checked 2026-10-15.

The horizontal autoscaler scales on demand

Thu, 04 Nov 2027 00:00:00 +0000

The HorizontalPodAutoscaler adjusts a scalable workload toward configured metric targets. Its output is a desired replica count, not a guarantee that additional Pods are already serving traffic. Scheduling, image startup, initialization, readiness, and available node capacity determine how quickly that recommendation becomes useful capacity. Design the control loop together with those delays.

Source arithmetic and list-state model only. Historical Python-only execution remains passed; it did not exercise tolerance, rate policies, per-Pod handling, Kubernetes reconciliation or useful ready capacity. No new run was performed.

Metrics produce recommendations

For a simple per-Pod average, desired replicas are the ceiling of current replicas multiplied by current metric divided by target metric. A ratio of two suggests doubling before additional controls apply. A ratio below one can suggest shrinking, but tolerance, missing measurements, readiness handling, bounds, and scaling behavior can prevent or limit the action.

CPU utilization uses resource requests as its denominator. Changing requests can change utilization without changing consumed CPU, so coordinating vertical and horizontal scaling matters. The metrics API must be available and the target must support the scale subresource. HPA changes a Deployment’s count; the Deployment controller manages its ReplicaSets and Pods.

The default reconciliation interval is fifteen seconds, but that is not an end-to-end response promise. Metrics have their own collection windows and delays. Distinguish average usage across Pods from temporal averaging in the measurement pipeline, and distinguish both from stabilization of recommendations. A stale but valid number can mislead an otherwise healthy controller.

Stabilize recommendations, not the last resize

Downscale stabilization retains recent replica recommendations and selects the highest within the configured window. The default downscale window is five minutes; autoscaling/v2 behavior can customize it. This is not a simple cooldown measured from the last replica change. A high recommendation that left the count unchanged can still postpone a later shrink.

Rate policies separately bound how many Pods or what percentage may change over a period. Scale-up can therefore be limited too; it is not necessarily immediate or unlimited. Set a realistic floor for startup latency and a maximum consistent with capacity and cost. Stabilization reduces churn but does not compensate for a target that tracks the wrong bottleneck.

Choose pressure that extra replicas relieve

CPU can be useful for compute-bound requests, while queue depth or work per replica may better describe workers. Neither latency nor an arbitrary resource graph automatically makes a good control input. If all replicas wait on a saturated shared database, multiplying them can increase contention instead of reducing delay.

Evaluate what happens after one replica is added: can work move to it, does per-replica pressure fall, and does downstream capacity remain sufficient? Long-lived connections, uneven partitions, and cache warmup can all delay the intended effect. A controller cannot redistribute an application’s work simply by changing a number.

When it breaks

Missing metrics are unknown demand, not zero demand. With multiple metrics, a valid scale-up recommendation can still be used even if another source fails; a failed source prevents a shrink suggested by the remaining sources. Missing samples for individual Pods also receive conservative treatment. Use conditions and events to distinguish these cases instead of assuming every missing sample stops every action.

A maxReplicas ceiling can leave queues growing while ScalingLimited reports that the calculated recommendation was constrained. A high minimum can hide low demand. Repeated fixed replicas writes from deployment automation can fight HPA. Also inspect CPU request changes and startup readiness before blaming normal rolling updates: HPA supports Deployment rollouts and does not simply count every initializing Pod as zero-load capacity for unrestricted shrink.

Check it quickly

Save this as hpa_loop.py and run python3 hpa_loop.py. It is a standard-library model of ratio arithmetic, bounds, and a highest-recent-recommendation downscale window. It deliberately omits tolerance, rate policies, individual Pod samples, and live Kubernetes reconciliation.

import math

TARGET = 50.0
MIN_REPLICAS = 2
MAX_REPLICAS = 10
WINDOW_S = 300


def desired(replicas, value):
 raw = math.ceil(replicas * value / TARGET)
 return max(MIN_REPLICAS, min(MAX_REPLICAS, raw))


def step(replicas, history, now, value):
 if value is None:
 print(f"t={now}s metric=missing hold={replicas}")
 return replicas
 recommendation = desired(replicas, value)
 history[:] = [(t, count) for t, count in history if now - t <= WINDOW_S]
 history.append((now, recommendation))
 if recommendation < replicas:
 result = min(replicas, max(count for _, count in history))
 else:
 result = recommendation
 print(f"t={now}s raw={recommendation} {replicas}->{result}")
 return result


def main():
 history = []
 replicas = step(3, history, 0, 100)
 assert replicas == 6
 replicas = step(replicas, history, 240, 50)
 assert replicas == 6
 replicas = step(replicas, history, 400, 15)
 assert replicas == 6 # A recent recommendation, not a recent scale event.
 replicas = step(replicas, history, 541, 15)
 assert replicas == 2
 replicas = step(replicas, history, 600, None)
 assert replicas == 2
 assert desired(2, 500) == 10
 assert desired(2, 5) == 2
 print("PASS: recommendation history holds 6 at t=400, then shrinks to 2")


if __name__ == "__main__":
 main()

At time zero the model grows from three to six. At 240 seconds it recommends six again without resizing. At 400 seconds a new recommendation of two is held at six because the recent recommendation remains in the window, even though the last actual resize was more than five minutes ago. At 541 seconds that high recommendation has expired and the model shrinks to two.

Both language copies passed with Python 3.12.14 in this review, including missing-input hold and both bounds. Those assertions establish the model’s stated behavior only. For cluster tuning, compare real HPA conditions, recommendations, replica transitions, Pod readiness, and application latency over time. No live controller or metrics pipeline was exercised here.

Sources

Kubernetes horizontal pod autoscaling, how HPA derives replica counts, checked 2026-10-15.
HorizontalPodAutoscaler walkthrough, worked example with multiple metrics and status conditions, checked 2026-10-15.

Autoscaling · XB Field Notes

The Horizontal Pod Autoscaler depends on metric semantics

Separate measurement from control

Pick a signal replicas can improve

Give replica ownership one home

When it breaks

Check it quickly

Further reading

Sources

The horizontal autoscaler scales on demand

Metrics produce recommendations

Stabilize recommendations, not the last resize

Choose pressure that extra replicas relieve

When it breaks

Check it quickly

Further reading

Sources