Nodes · XB Field Notes

Node affinity places workloads on matching nodes

Thu, 09 Dec 2027 00:00:00 +0000

Node affinity matches workload requirements to node labels. It can require NVMe storage or prefer a newer instance family, but it does not replace the scheduler’s other checks. Even without affinity, scheduling filters resource requests, taints and volume constraints before scoring feasible nodes. It is not simply a comparison of idle capacity.

GPU placement illustrates the distinction. A device-plugin GPU resource request accounts for available devices; an accelerator label can select a particular model. Labels alone allocate no GPU, and a toleration alone supplies neither the device nor a placement requirement.

Required affinity filters by labels. Preferred affinity contributes to ranking among feasible nodes; neither alone guarantees placement.

Match nodes by label

Labels such as accelerator=nvidia-a100 and storage=nvme describe properties that provisioning must establish and maintain. A replacement node does not automatically inherit arbitrary labels from its predecessor. Verify the new hardware before applying the old label; otherwise a formally matching rule can select an unsuitable machine.

Treat the label vocabulary as an API. Renaming a key affects every referring manifest, while stale values can misrepresent hardware. Validate manifests against the intended node inventory, including capacity planned but not yet online. Absence of a label may reflect provisioning or a temporarily missing pool, rather than a typo alone.

For isolation, label integrity matters. Kubernetes 1.34 documents the Node authorizer plus NodeRestriction admission plugin and protected node-restriction.kubernetes.io/ label keys. An ordinary self-reported compliance label is not an independent security boundary. Confirm who can change labels and workload placement requirements.

Prefer or require

requiredDuringSchedulingIgnoredDuringExecution filters out nodes that fail the rule. preferredDuringSchedulingIgnoredDuringExecution adds weighted preferences among feasible nodes; other scores can win. Neither guarantees spare capacity, a healthy application or resilience. A hard requirement may leave a Pod Pending, while a soft one permits a nonmatching node only if all hard constraints pass.

The names also specify when rules apply. A later label change does not evict an already-bound Pod under IgnoredDuringExecution, and changing a preference does not automatically redistribute existing replicas. Inspect new scheduling decisions separately from the current placement inventory.

When both nodeSelector and required node affinity exist, both must match. Alternative nodeSelectorTerms are ORed; matchExpressions inside one term are ANDed. Review that grouping before blaming capacity, because an accidental conjunction can eliminate the entire intended pool.

Anti-affinity splits replicas

Pod anti-affinity matches other Pods within a selected namespace scope and topology domain. kubernetes.io/hostname represents nodes; the standard zone key is topology.kubernetes.io/zone. All relevant nodes need consistent labels. Admission policy can restrict required anti-affinity topology keys, so a zone rule is not universally accepted.

With three mutually selected replicas, required hostname separation and only two otherwise eligible nodes, two replicas may run while the third remains Pending. That is different from preventing the whole workload from running. Preferred anti-affinity can allow co-location and does not promise nearly the same resilience. Choose the rule from the application’s failure tolerance, eligible domains and replacement capacity.

When it breaks

For a Pending Pod, inspect the complete scheduler event and admitted specification. A label mismatch is one possibility; insufficient requested resources, untolerated taints, volume topology or scheduler-profile constraints are others. Low observed CPU use does not establish request headroom. Fix the constraint actually reported rather than immediately weakening affinity.

For an unexpected bound Pod, compare its node’s labels with the rule that applied when it was scheduled. Check whether the rule was preferred, whether labels subsequently changed, and whether the Pod bypassed ordinary scheduling through nodeName. Do not infer that the scheduler migrated an existing GPU workload because the current inventory differs from intent.

Check it quickly

The retained commands below are an unexecuted, abbreviated starting point:

kubectl get nodes -L accelerator,workload-class --no-headers
kubectl describe pod api-0 | grep -A 8 "Node-Selectors\|Affinity"
kubectl get pods -o wide -l app=api | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head

The grep view can omit affinity details. The final pipeline includes unique Pod names and a header, so it does not aggregate replicas per node; fixed column positions are also fragile. Inspect the full admitted affinity and nodeSelector fields, actual nodeName values and selected peer Pods, then count topology domains explicitly. These commands do not prove placement or resilience. The diagram records an earlier graphics-only review. Live scheduling remains unverified.

Sources

Kubernetes node affinity docs, node affinity semantics, checked 2026-06-10.
Kubernetes taints and tolerations, documents repelling pods from specialized nodes, checked 2026-09-05.
Kubernetes 1.34 placement rules, label trust, grouping and scheduling-time affinity, checked 2026-10-18.
Kubernetes scheduler, filtering and scoring feasible nodes, checked 2026-10-18.
Schedule GPUs, device resource requests versus GPU model labels, checked 2026-10-18.

Taints repel pods that do not tolerate them

Thu, 25 Feb 2027 00:00:00 +0000

A taint repels Pods that do not tolerate its effect. A toleration removes that obstacle; it does not force placement onto specialized hardware. Dedicated-node designs usually combine a taint with a label and required node affinity, then check capacity and admission policy separately.

A toleration removes a matching taint obstacle. The effect determines the consequence; placement and authorization remain separate.

A sign for specialists

A NoSchedule taint blocks new ordinary scheduling when it is not tolerated; it leaves existing Pods alone. This can reserve GPU or other specialist nodes from general workloads. The stronger claim that every taint leaves running Pods untouched is false: NoExecute has eviction behavior, while PreferNoSchedule is only a scheduling preference.

Taints are not security isolation. Someone allowed to submit suitable tolerations may qualify for the pool, and directly assigning nodeName bypasses ordinary scheduler checks. Admission restrictions, authorization and the workload’s requested devices remain separate controls. A GPU toleration alone neither requests a GPU nor guarantees that one is available.

Effect decides the action

NoSchedule is a hard scheduling exclusion for untolerated Pods. PreferNoSchedule contributes a soft avoidance preference and cannot alone make a node ineligible. NoExecute can also trigger removal of already-bound Pods without a matching toleration. A matching NoExecute toleration may specify tolerationSeconds to limit how long the Pod remains after the taint appears.

The taint eviction controller, rather than the scheduler moving running processes, handles taint-based eviction. This is not equivalent to kubectl drain: disruption-budget handling and operational sequencing differ. Use the intended maintenance workflow and assess affected workloads before applying an eviction-producing taint.

Pair is a contract

For the Equal and Exists operators illustrated here, key, operator, value and effect determine matching. Equal checks the value; Exists ignores value for the selected key. An empty effect covers all effects. A permitted empty key requires Exists and broadens matching across keys. Validate the API form rather than letting malformed input quietly become a wildcard.

After matching, every remaining hard taint must be tolerated for this scheduling filter to pass. Passing it is only one requirement: resource fit, affinity, volumes and other scheduler plugins can still reject the node. Required affinity to the dedicated-node label supplies the placement restriction that a toleration deliberately lacks.

This model follows the Kubernetes 1.34 Equal/Exists subset, with omitted string values normalized to empty strings. It is not full API validation, defaulting or admission. Later feature-gated operators require their own version-aware checks.

When it breaks

The most expensive failure is the over-broad toleration that pollutes the specialist pool. A toleration with operator: Exists and no value, or one copied into a shared chart, matches every value of that key, so ordinary Deployments suddenly qualify for GPU or spot nodes. The reserved pool fills with cheap pods while the specialists it was built for stay Pending. This is the mirror image of the DaemonSet failure: there a missing toleration leaves special nodes uncovered, here an extra toleration leaves them unprotected. Audit broad tolerations; they are not RBAC authorization.

The second failure is applying NoExecute when you meant NoSchedule. NoSchedule steers only new pods, while NoExecute evicts running pods without a matching toleration, optionally after tolerationSeconds expires. A maintenance taint with the wrong effect can unexpectedly evict production Pods. Unlike standard IgnoredDuringExecution node affinity, the taint effect carries destructive power, so rehearse on one node and watch kubectl get pods -o wide before repeating across the pool.

The third failure is spelling drift between taint and toleration. dedicated=gpu:NoSchedule never meets dedicated=GPU:NoSchedule, and a toleration naming PreferNoSchedule does not cover a NoSchedule taint. Specialists sit Pending with had taint that the pod didn't tolerate while the pool looks healthy. Compare both sides with kubectl describe pod against kubectl get node <name> -o jsonpath='{.spec.taints}', and keep the canonical triple in one documented place.

Check it quickly

Save the following as taint_toleration_sim.py and run python3 taint_toleration_sim.py. It uses only the standard library, touches no cluster, and models the rules above: key plus operator plus value plus effect, with soft taints never blocking.

def tolerates(taint, tol):
 """One toleration matches one taint on key, value, operator, and effect."""
 operator = tol.get("operator", "Equal")
 if operator not in {"Equal", "Exists"}:
 raise ValueError("model supports only Equal and Exists")
 if not tol.get("key") and operator != "Exists":
 raise ValueError("empty key requires Exists")
 if operator == "Exists" and tol.get("value"):
 raise ValueError("Exists requires an empty value")
 if tol.get("key") and tol["key"] != taint["key"]:
 return False
 if tol.get("operator", "Equal") == "Equal":
 if tol.get("value", "") != taint.get("value", ""):
 return False
 effect = tol.get("effect", "")
 if effect and effect != taint.get("effect"):
 return False
 return True


def schedulable(taints, tolerations):
 """A pod schedules only if every hard taint is tolerated."""
 for taint in taints:
 if taint["effect"] == "PreferNoSchedule":
 continue
 if not any(tolerates(taint, tol) for tol in tolerations):
 return False
 return True


def check(actual, expected, message):
 if actual != expected:
 raise AssertionError(f"{message}: got {actual}, want {expected}")
 print(f"OK: {message} -> {actual}")


def main():
 gpu = {"key": "dedicated", "value": "gpu", "effect": "NoSchedule"}
 exact = {"key": "dedicated", "value": "gpu", "effect": "NoSchedule"}
 check(schedulable([gpu], [exact]), True, "exact match schedules")
 check(schedulable([gpu], []), False, "missing toleration is repelled")
 wide = {"key": "dedicated", "operator": "Exists", "effect": "NoSchedule"}
 other = {"key": "dedicated", "value": "spot", "effect": "NoSchedule"}
 check(schedulable([other], [wide]), True, "Exists toleration matches any value")
 check(schedulable([gpu], [{"key": "dedicated", "value": "spot", "effect": "NoSchedule"}]), False, "wrong value is repelled")
 evict = {"key": "maintenance", "value": "drain", "effect": "NoExecute"}
 any_effect = {"key": "maintenance", "value": "drain"}
 check(schedulable([evict], [any_effect]), True, "empty effect matches any effect")
 check(schedulable([evict], []), False, "untolerated NoExecute fails the scheduling filter")
 soft = {"key": "cost", "value": "cheap", "effect": "PreferNoSchedule"}
 check(schedulable([soft], []), True, "PreferNoSchedule never blocks")
 print("PASS: 7 scenarios; allow and repel verified")


if __name__ == "__main__":
 main()

The seven scenarios show that an exact toleration passes this model’s taint filter while a missing one repels, an Exists toleration matches any value but a wrong value is refused, an empty effect covers NoExecute, an untolerated NoExecute fails this filter; no actual eviction is simulated, and PreferNoSchedule never blocks. Expected output ends with PASS: 7 scenarios; allow and repel verified.

Both language fixtures were rerun with Python 3.12.14 on Linux. Additional checks rejected malformed tolerations and exercised empty-key and effect-mismatch controls. The script checks matching logic only, not capacity, affinity, or live taints. Before changing production, compare pod events against the node’s actual taint list.

Sources

Kubernetes taints and tolerations, documents effects and scheduling, checked 2026-10-15.
Kubernetes node affinity, documents matching pods to node labels, checked 2026-10-15.
Kubernetes v1.34.0 core API types, optional string values and Equal/Exists toleration operators, checked 2026-10-18.

Nodes · XB Field Notes

Node affinity places workloads on matching nodes

Match nodes by label

Prefer or require

Anti-affinity splits replicas

When it breaks

Check it quickly

Sources

Taints repel pods that do not tolerate them

A sign for specialists

Effect decides the action

Pair is a contract

When it breaks

Check it quickly

Further reading

Sources