Systems · XB Field Notes

Backpressure works when bounded admission changes the producer’s behavior

Thu, 12 Aug 2027 00:00:00 +0000

Backpressure connects a consumer’s limited capacity to the behavior of its producers. A full channel may make a producer wait, reject new work, or apply a deliberately chosen shedding policy. The important property is that the limit changes what upstream components do. A growing queue or a dashboard showing lag is evidence of imbalance, but neither is automatically a mechanism that controls admission.

When it breaks

An unbounded queue can absorb a temporary burst while hiding sustained overload. If work arrives faster than it finishes, outstanding work and waiting time continue growing. Moving the same backlog into pending threads, retry tasks, or another unbounded buffer does not remove that pressure. The capacity boundary must cover the place where work is actually retained, not merely one convenient queue object.

A bounded queue also does not define a complete overload policy by itself. Blocking forever can consume all producer threads or leave shutdown stuck. Immediate rejection can lose work if callers treat it as success. Retrying every rejection immediately can increase load. Decide whether work should wait within a deadline, be rejected for later retry, or be discarded under an explicit application contract.

Queue capacity and processing completion are separate. Removing an item can free a queue slot while the consumer is still working on that item. A capacity of two waiting items may therefore coexist with additional in-progress work. Likewise, a producer’s successful enqueue says only that the queue accepted the item; it does not establish execution, durable storage, or completion of its downstream side effects.

Removing a frees admission capacity before processing is acknowledged. A successful put, an empty queue and a completed join represent different boundaries; queue accounting is not durable delivery.

Check it quickly

Save this as check-backpressure.py and run python3 check-backpressure.py. It uses only Python’s standard library and was executed with Python 3.14.4. Threads coordinate through explicit events and bounded waits, so the example does not depend on a remote broker, database, or an artificially slow external service. All queues and work items exist only inside the process:

from queue import Queue,Full
from threading import Event,Thread
import json

unbounded=Queue()
for item in range(1000):unbounded.put_nowait(item)
assert unbounded.qsize()==1000
channel=Queue(maxsize=2)
channel.put('a');channel.put('b')
try:
 channel.put('deadline-rejected',timeout=.02)
 raise AssertionError('a full bounded queue accepted another item')
except Full:rejected=True
assert channel.qsize()==2
entered,delivered=Event(),Event()
def produce():
 entered.set();channel.put('c',timeout=2);delivered.set()
producer=Thread(target=produce,daemon=True);producer.start();assert entered.wait(1)
assert not delivered.wait(.05) and channel.qsize()==2
first=channel.get(timeout=1);assert first=='a'
assert delivered.wait(1);producer.join(timeout=1);assert not producer.is_alive()
assert channel.qsize()==2
rest=[channel.get(timeout=1),channel.get(timeout=1)];assert rest==['b','c']
assert channel.empty()
completed=Event()
def join():channel.join();completed.set()
waiter=Thread(target=join,daemon=True);waiter.start();assert not completed.wait(.05)
for item in [first,*rest]:channel.task_done()
assert completed.wait(1);waiter.join(timeout=1);assert not waiter.is_alive()
print(json.dumps({'unbounded_backlog':unbounded.qsize(),'bounded_capacity':2,'deadline_rejected':rejected,
 'blocked_producer_resumed_after_get':delivered.is_set(),'delivered_items':[first,*rest],
 'empty_queue_did_not_mean_processing_complete':True,'join_finished_after_task_done':completed.is_set()}))

The first queue has the default unbounded capacity. With no consumer running, a thousand put_nowait calls all succeed and leave a backlog of a thousand items. This is a small behavioral demonstration, not a memory-exhaustion benchmark. It shows that successful admission alone cannot establish downstream progress or that the arrival rate is sustainable.

The second queue has capacity two and starts with items a and b. Another put uses a short timeout and raises Full, leaving the queued items unchanged. The failed item is not secretly enqueued for later execution. This gives the caller a definite admission outcome on which to base retry or rejection handling instead of presenting an unlimited wait as successful delivery.

A producer thread then tries to put c. The test confirms that it remains blocked while both slots are occupied. Removing a opens one slot, the producer completes its put, and b and c remain queued. The program verifies that all three accepted items are retrieved in order. This demonstrates pressure reaching the producer and being released by a concrete capacity change.

After retrieving the remaining items, the queue is empty, but a separate thread waiting in join still has not completed. Only after task_done is called for all three accepted items does join return. Dequeueing freed capacity earlier; task acknowledgment marks the tracked work complete later. The test separates these events so an empty queue cannot be mistaken for completed business processing.

Preserve the admission contract across the pipeline

Use the queue operation’s actual result rather than a check-then-act decision based on qsize or full. Those observations are approximate when other threads can change the queue. The fixture’s events and paused consumers make its assertions controlled observations; a production caller should handle Full or a successful put atomically through the provided operation.

Choose a bound appropriate to retained resources. An item-count limit does not cap memory when items can be arbitrarily large, and a waiting-queue limit does not include every active worker or external request. Include processing concurrency and upstream retry behavior when evaluating overload. A bounded component surrounded by unlimited pending work can still participate in an unbounded system.

Finally, define what completion means. Python task_done updates queue accounting; it is not a durable transaction or proof that an external side effect succeeded. A worker failure, process crash, or retry after an ambiguous response needs the application’s corresponding delivery and idempotency contract. Backpressure controls admission and pace; correctness still depends on handling accepted, rejected, in-progress, and completed work explicitly.

Sources

Python queue documentation, bounded capacity, Full, blocking puts and task completion, checked 2026-09-12.
Google SRE: Handling Overload, client throttling, load shedding, and resource-based overload limits, checked 2026-09-12.

OOM investigation starts with the allocation scope

Mon, 26 Oct 2026 00:00:00 +0000

Linux can invoke an out-of-memory killer when an allocation cannot be satisfied within its relevant constraints after appropriate reclaim attempts. It is a last-resort recovery mechanism, not an application priority scheduler. Some allocations fail without a kill, and configuration can instead lead to panic behavior. Start an investigation by identifying the allocation scope and recorded event, rather than declaring that the machine must have exhausted every byte of RAM.

Scope comes before the score

A global memory shortage and a memory-control-group shortage are different situations. A cgroup can hit its own limit while the host still has available memory. Its OOM handling cannot simply solve that shortage by selecting an arbitrary process outside the constrained group. Other constraints, such as allowed memory nodes, can also matter. A dashboard of host-wide free memory is therefore insufficient evidence for or against a particular allocation failure.

Within the applicable selection rules, memory use and an adjustment influence victim choice. The kernel does not know which process is commercially valuable, idle-but-disposable, or essential to the business. A large database can be selected even when smaller workers contributed to pressure. oom_score is an observation exposed through procfs, not a promise that sorting every visible process predicts the next victim in every possible OOM domain.

Interpret adjustments without promising immunity

oom_score_adj ranges from -1000 to 1000. Positive values increase susceptibility and negative values reduce it; -1000 excludes a task from the kernel OOM selection addressed by that control. A modest negative adjustment does not guarantee survival. It is also not protection against explicit signals, a service manager, or a separate userspace OOM policy. Numeric adjustments cannot create missing capacity.

Protecting every task is not a workable capacity plan. If no eligible victim exists, the kernel cannot be assumed to escape a cgroup boundary and kill an unrelated supervisor. Group-kill behavior and panic settings require separate analysis. Adjustments can be inherited by children, while persistent service settings belong to the supervisor’s configuration. Changes require deliberate authorization and testing; the exercise below makes none.

When it breaks

A restart loop can hide an ongoing leak. If the same workload grows until it fails repeatedly, adding memory may merely lengthen the interval. Correlate allocation rate, resident and charged memory, workload size, reclaim, and swap activity. A process’s RSS alone does not describe all cgroup charges or all shared-memory relationships. Lowering a limit can contain growth but may make the application fail sooner.

Exit status 137 is compatible with SIGKILL; it is not exclusive evidence of OOM. Check the runtime’s termination reason, cgroup memory-event counters, and kernel records available to the operator. Distinguish a failed allocation from a killed process, and kernel OOM from userspace pressure-driven termination. A missing kernel log can reflect unavailable visibility rather than proof that no event occurred.

The score itself changes as memory and scope change. Processes can exit between procfs reads, and container PID visibility may hide much of the host. Never treat a successful read as an atomic system snapshot. Preserve the event time and affected cgroup when diagnosing an actual failure, then compare observations from the same scope.

OOM scope and event context come before score ranking; the PID ordering here is an invented presentation fixture.

Check it quickly

Save this as oom_observe.py and run python3 oom_observe.py with Python 3.9 or newer. It reads only its own process’s two procfs score files, allocates no pressure workload, and changes no limits or policy. Two deterministic assertions test a presentation sort on explicitly invented rows.

"""Read-only observations and a sorting test, not an OOM predictor."""
from pathlib import Path
import os

def snapshot(pid):
 base = Path('/proc') / str(pid)
 try:
 return (int(pid), int((base/'oom_score').read_text()),
 int((base/'oom_score_adj').read_text()))
 except (OSError, ValueError):
 return None

def ranked(rows):
 return sorted(rows, key=lambda row: (-row[1], row[0]))

fixtures = [(42, 700, 0), (17, 100, -500), (99, 700, 200)]
assert [row[0] for row in ranked(fixtures)] == [42, 99, 17]
assert ranked([]) == []
row = snapshot(os.getpid())
if row is None:
 print('SKIP: this process has no readable Linux OOM score files')
else:
 pid, score, adjustment = row
 assert -1000 <= adjustment <= 1000
 print(f'pid={pid} oom_score={score} oom_score_adj={adjustment}')
 print('Observed one process; no victim selection or memory pressure tested')
print('PASS: deterministic sorting checks; live observation is reported separately')

The fixture orders equal scores by numeric PID and puts the lower score last. It does not calculate kernel badness from the invented adjustment values. On Linux, the separate live branch prints the current process’s score and adjustment; without readable files it reports SKIP while still running the sorting assertions. A PASS line therefore does not imply live OOM handling was exercised.

Both exact examples passed the sorting assertions and read their own procfs files in this review’s Linux environment. No victim was selected or killed. Use this small observation to learn the interface, then investigate real events using scope-specific evidence. Changing the ranking or giving an important process a negative adjustment is never sufficient proof that the service will remain available under memory pressure.

Sources

Linux memory concepts, allocation, reclaim and OOM boundaries, checked 2026-10-15.
proc_pid_oom_score, reported process score, checked 2026-10-15.
proc_pid_oom_score_adj, adjustment range and constrained allocation context, checked 2026-10-15.
Linux cgroup v2, memory.max, OOM groups and protection scope, checked 2026-10-15.

Clock synchronization bounds time disagreement

Wed, 09 Sep 2026 00:00:00 +0000

Clock synchronization reduces disagreement between machines’ civil-time readings. It supports certificate validity checks, timestamp interpretation, and many expiration policies, but it does not establish a perfect global clock. A host can emit well-formatted UTC while measuring the wrong instant. Applications still need to distinguish elapsed durations, absolute deadlines, and causal ordering rather than making them all depend on raw wall-clock comparisons.

Four timestamps estimate two quantities

An NTP exchange records client send time T1, server receive time T2, server transmit time T3, and client receive time T4. The usual offset estimate is ((T2-T1)+(T3-T4))/2; a positive result means the server appears ahead. The round-trip delay estimate is (T4-T1)-(T3-T2), subtracting time spent processing at the server. These are estimates from observations, not direct measurements of every one-way link delay.

If forward and reverse paths have unequal delay, the offset estimate inherits an error. A small round-trip measurement can be useful without proving symmetry. A synchronization daemon uses repeated samples, selection, filtering, and a clock model rather than treating one packet as unquestionable truth. Source reachability, uncertainty, and source trust all matter; a low stratum number alone is not an accuracy certificate.

Timestamp representation also differs from measurement accuracy. A fractional field can encode very fine increments while the underlying clock or network observation remains much less certain. Printing additional decimal places does not improve that evidence.

Clock control is another stage. A daemon can gradually slew corrections, and configured conditions can permit stepping the clock. Do not promise that NTP never produces a wall-clock discontinuity or always reaches sub-millisecond accuracy. The network, oscillator, platform, and configuration determine achievable behavior. Choose monitoring bounds from application needs and measured conditions instead of adopting one universal offset threshold.

When it breaks

A wall clock can move backward or forward, making naive elapsed-time subtraction negative or misleading. Use a suitable monotonic clock for a local duration or timeout. Its reference point is not a portable timestamp for another host or a later boot. Certificate dates and business deadlines still need civil-time semantics, so monotonic time does not replace all timestamp handling.

Malformed, unrelated, unsynchronized, or policy-denied NTP responses must not be accepted as ordinary samples. Matching a response’s origin timestamp to the request helps reject unrelated packets, but it is not cryptographic authentication. Production clients need the protocol’s fuller validation and source-selection behavior. Synchronization also does not turn a timestamp-based lease into a safe distributed lock or guarantee that close events sort in causal order.

A four-timestamp exchange estimates offset; the two synthetic paths have equal round trips but different offset error.

Check it quickly

Use a fresh disposable Debian 12 Docker container with Python 3 and chrony installed; the package setup is apt-get update followed by apt-get install -y python3 chrony. Save the code as ntp-exchange.py and run it as root with python3 ntp-exchange.py. The original report used chrony 4.3 and Python 3.11.2; these package commands do not pin those versions. Run with assertions enabled, without -O or PYTHONOPTIMIZE. Drop SYS_TIME capability and avoid host networking when creating the container.

from pathlib import Path
from tempfile import TemporaryDirectory
import json
import os
import socket
import struct
import subprocess
import time

assert os.geteuid() == 0 and Path('/.dockerenv').exists(), 'use a disposable root Linux container'
NTP_EPOCH = 2208988800
ERA = 2**32


def stamp(unix_time):
 value = unix_time + NTP_EPOCH
 whole = int(value)
 return struct.pack('!II', whole % ERA, int((value - whole) * ERA))


def seconds(data, reference):
 whole, fraction = struct.unpack('!II', data)
 era = round((reference + NTP_EPOCH - whole) / ERA)
 return whole + era * ERA - NTP_EPOCH + fraction / ERA


def estimate(t1, t2, t3, t4):
 return ((t2 - t1) + (t3 - t4)) / 2, (t4 - t1) - (t3 - t2)


def validate(data, sent):
 if len(data) < 48:
 raise ValueError('short packet')
 if data[0] >> 6 == 3:
 raise ValueError('unsynchronised source')
 if (data[0] & 7) != 4 or ((data[0] >> 3) & 7) != 4:
 raise ValueError('unexpected mode or version')
 if not 1 <= data[1] <= 15:
 raise ValueError('unusable stratum')
 if data[24:32] != sent:
 raise ValueError('origin timestamp mismatch')


with TemporaryDirectory(prefix='ntp-exchange-') as directory:
 root = Path(directory)
 with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as reservation:
 reservation.bind(('127.0.0.1', 0))
 port = reservation.getsockname()[1]
 config = root / 'chrony.conf'
 config.write_text(f'pidfile {root}/pid\ncmdport 0\nbindcmdaddress {root}/command.sock\n'
 f'port {port}\nbindaddress 127.0.0.1\nlocal stratum 8\nallow 127.0.0.1\n')
 process = subprocess.Popen(['chronyd', '-x', '-n', '-u', 'root', '-f', str(config)],
 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 try:
 deadline = time.monotonic() + 5
 while not (root/'command.sock').exists():
 assert process.poll() is None
 if time.monotonic() > deadline:
 raise AssertionError('chronyd did not start')
 time.sleep(0.05)
 with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as client:
 client.settimeout(5)
 client.connect(('127.0.0.1', port))
 request = bytearray(48)
 request[0] = 0x23
 t1, start = time.time(), time.monotonic()
 sent = stamp(t1)
 request[40:48] = sent
 client.send(request)
 data = client.recv(512)
 t4, finish = time.time(), time.monotonic()
 validate(data, sent)
 t2, t3 = seconds(data[32:40], t1), seconds(data[40:48], t1)
 offset, delay = estimate(t1, t2, t3, t4)
 assert data[1] == 8 and finish >= start
 bad_origin = bytearray(data); bad_origin[24:32] = bytes(8)
 alarm = bytearray(data); alarm[0] |= 0xC0
 denied = bytearray(data); denied[1] = 0
 rejected = []
 for candidate in (data[:40], bad_origin, alarm, denied):
 try:
 validate(candidate, sent)
 except ValueError as error:
 rejected.append(str(error))
 else:
 raise AssertionError('invalid packet accepted')
 symmetric = estimate(100, 105.020, 105.022, 100.042)
 asymmetric = estimate(100, 105.010, 105.012, 100.042)
 assert abs(symmetric[0] - 5) < 1e-9 and abs(symmetric[1] - 0.040) < 1e-9
 assert abs(asymmetric[0] - 4.990) < 1e-9 and abs(asymmetric[1] - 0.040) < 1e-9
 print(json.dumps({'actual_loopback_sample': {'stratum': data[1],
 'offset_seconds': offset, 'delay_seconds': delay,
 'monotonic_elapsed_seconds': finish-start},
 'rejected_packets': rejected,
 'synthetic_symmetric_estimate': list(symmetric),
 'synthetic_asymmetric_estimate': list(asymmetric),
 'known_synthetic_offset': 5,
 'system_clock_adjustment_disabled': True,
 'external_UTC_accuracy_tested': False}))
 finally:
 process.terminate()
 process.communicate(timeout=5)

The fixture starts chronyd with -x, which disables system-clock adjustment, and a loopback-only source using local stratum 8. It sends a real version-four client packet and checks the returned mode, stratum, origin timestamp, and alarm state. Expect a stratum-eight sample with measured offset and delay values that vary between runs. Four altered or truncated packets are rejected. Both sockets and the daemon are closed afterward.

Two separate synthetic calculations hold the true server lead at five seconds. With twenty milliseconds in each direction, the estimate is five seconds and the round-trip network delay forty milliseconds. With ten milliseconds outbound and thirty inbound, the same forty-millisecond delay accompanies an estimated lead of 4.990 seconds. The ten-millisecond error demonstrates the symmetry assumption; those values are not measurements of the loopback path.

The original report states that all assertions passed; this review did not rerun the daemon or socket exchange, and its original raw runtime log was not available. The local source shares the test host’s clock and has no external UTC reference. This experiment neither disciplines that clock nor proves external accuracy, NTS authentication, or full NTP client conformance. Keep the distinction between a valid observed exchange, a bounded estimate, and trustworthy operational time visible in diagnostics.

Sources

RFC 5905: Network Time Protocol Version 4, timestamp exchange, offset estimation and delay, checked 2026-09-13.
chronyd 4.3 manual, clock control and the no-adjustment option, checked 2026-09-13.
Python: time, wall-clock and monotonic-clock behavior, checked 2026-09-13.
Python 3.11: assert statement, optimization removes assertion checks, checked 2026-10-18.