Queues · XB Field Notes

Backpressure works when bounded admission changes the producer’s behavior

Thu, 12 Aug 2027 00:00:00 +0000

Backpressure connects a consumer’s limited capacity to the behavior of its producers. A full channel may make a producer wait, reject new work, or apply a deliberately chosen shedding policy. The important property is that the limit changes what upstream components do. A growing queue or a dashboard showing lag is evidence of imbalance, but neither is automatically a mechanism that controls admission.

When it breaks

An unbounded queue can absorb a temporary burst while hiding sustained overload. If work arrives faster than it finishes, outstanding work and waiting time continue growing. Moving the same backlog into pending threads, retry tasks, or another unbounded buffer does not remove that pressure. The capacity boundary must cover the place where work is actually retained, not merely one convenient queue object.

A bounded queue also does not define a complete overload policy by itself. Blocking forever can consume all producer threads or leave shutdown stuck. Immediate rejection can lose work if callers treat it as success. Retrying every rejection immediately can increase load. Decide whether work should wait within a deadline, be rejected for later retry, or be discarded under an explicit application contract.

Queue capacity and processing completion are separate. Removing an item can free a queue slot while the consumer is still working on that item. A capacity of two waiting items may therefore coexist with additional in-progress work. Likewise, a producer’s successful enqueue says only that the queue accepted the item; it does not establish execution, durable storage, or completion of its downstream side effects.

Removing a frees admission capacity before processing is acknowledged. A successful put, an empty queue and a completed join represent different boundaries; queue accounting is not durable delivery.

Check it quickly

Save this as check-backpressure.py and run python3 check-backpressure.py. It uses only Python’s standard library and was executed with Python 3.14.4. Threads coordinate through explicit events and bounded waits, so the example does not depend on a remote broker, database, or an artificially slow external service. All queues and work items exist only inside the process:

from queue import Queue,Full
from threading import Event,Thread
import json

unbounded=Queue()
for item in range(1000):unbounded.put_nowait(item)
assert unbounded.qsize()==1000
channel=Queue(maxsize=2)
channel.put('a');channel.put('b')
try:
 channel.put('deadline-rejected',timeout=.02)
 raise AssertionError('a full bounded queue accepted another item')
except Full:rejected=True
assert channel.qsize()==2
entered,delivered=Event(),Event()
def produce():
 entered.set();channel.put('c',timeout=2);delivered.set()
producer=Thread(target=produce,daemon=True);producer.start();assert entered.wait(1)
assert not delivered.wait(.05) and channel.qsize()==2
first=channel.get(timeout=1);assert first=='a'
assert delivered.wait(1);producer.join(timeout=1);assert not producer.is_alive()
assert channel.qsize()==2
rest=[channel.get(timeout=1),channel.get(timeout=1)];assert rest==['b','c']
assert channel.empty()
completed=Event()
def join():channel.join();completed.set()
waiter=Thread(target=join,daemon=True);waiter.start();assert not completed.wait(.05)
for item in [first,*rest]:channel.task_done()
assert completed.wait(1);waiter.join(timeout=1);assert not waiter.is_alive()
print(json.dumps({'unbounded_backlog':unbounded.qsize(),'bounded_capacity':2,'deadline_rejected':rejected,
 'blocked_producer_resumed_after_get':delivered.is_set(),'delivered_items':[first,*rest],
 'empty_queue_did_not_mean_processing_complete':True,'join_finished_after_task_done':completed.is_set()}))

The first queue has the default unbounded capacity. With no consumer running, a thousand put_nowait calls all succeed and leave a backlog of a thousand items. This is a small behavioral demonstration, not a memory-exhaustion benchmark. It shows that successful admission alone cannot establish downstream progress or that the arrival rate is sustainable.

The second queue has capacity two and starts with items a and b. Another put uses a short timeout and raises Full, leaving the queued items unchanged. The failed item is not secretly enqueued for later execution. This gives the caller a definite admission outcome on which to base retry or rejection handling instead of presenting an unlimited wait as successful delivery.

A producer thread then tries to put c. The test confirms that it remains blocked while both slots are occupied. Removing a opens one slot, the producer completes its put, and b and c remain queued. The program verifies that all three accepted items are retrieved in order. This demonstrates pressure reaching the producer and being released by a concrete capacity change.

After retrieving the remaining items, the queue is empty, but a separate thread waiting in join still has not completed. Only after task_done is called for all three accepted items does join return. Dequeueing freed capacity earlier; task acknowledgment marks the tracked work complete later. The test separates these events so an empty queue cannot be mistaken for completed business processing.

Preserve the admission contract across the pipeline

Use the queue operation’s actual result rather than a check-then-act decision based on qsize or full. Those observations are approximate when other threads can change the queue. The fixture’s events and paused consumers make its assertions controlled observations; a production caller should handle Full or a successful put atomically through the provided operation.

Choose a bound appropriate to retained resources. An item-count limit does not cap memory when items can be arbitrarily large, and a waiting-queue limit does not include every active worker or external request. Include processing concurrency and upstream retry behavior when evaluating overload. A bounded component surrounded by unlimited pending work can still participate in an unbounded system.

Finally, define what completion means. Python task_done updates queue accounting; it is not a durable transaction or proof that an external side effect succeeded. A worker failure, process crash, or retry after an ambiguous response needs the application’s corresponding delivery and idempotency contract. Backpressure controls admission and pace; correctness still depends on handling accepted, rejected, in-progress, and completed work explicitly.

Sources

Python queue documentation, bounded capacity, Full, blocking puts and task completion, checked 2026-09-12.
Google SRE: Handling Overload, client throttling, load shedding, and resource-based overload limits, checked 2026-09-12.

Queues define delivery; consumers must protect effects

Fri, 05 Feb 2027 00:00:00 +0000

A queue’s delivery promise describes a bounded part of a workflow. Getting a message to a consumer is different from committing its business effect. At-least-once delivery allows redelivery; at-most-once delivery permits loss rather than retrying uncertain delivery. Neither label alone guarantees that an order, notification, or database update completes exactly once under every failure.

Locate publication, acknowledgment, and commitment

First establish when the producer knows that the broker accepted a message, how long the broker retains it, and which failures its durability policy covers. A message never successfully published, expired by retention, or deliberately deleted is not rescued by calling the consumer at-least-once. Recovery assumptions belong beside the delivery label.

On the consumer side, acknowledging before the effect commits creates a loss window: the broker can stop retrying work the application has not finished. Committing first leaves another window: the effect succeeds, but acknowledgment is lost and the work returns. Redelivery is then compatible with the delivery contract. It does not mean the original operation failed.

Visibility timeouts and acknowledgment deadlines govern availability for another attempt. They are not proof that the first worker stopped. A slow worker and a redelivery can overlap, so extending a deadline and protecting the effect from duplicates solve different problems. Record both delivery attempts and committed business outcomes.

Make repeated effects harmless within a real boundary

For effects inside one transactional database, store a unique operation identity and its effect in the same transaction. The uniqueness check must arbitrate concurrent attempts; a separate read followed by an unprotected write is insufficient. Persist enough payload information to reject reuse of the same identity for a different operation rather than silently accepting conflicting instructions.

A database transaction cannot automatically include an external email, payment API, or another independent store. Those destinations need their own idempotency or coordination contract. A local marker written before the remote call can lose the effect; written afterward, it can leave a duplicate window. Do not present a local deduplication table as an end-to-end exactly-once guarantee.

Some services provide explicitly scoped exactly-once delivery features. Google Pub/Sub, for example, documents regional and subscription-type restrictions and distinguishes delivery duplicates from publish-side duplicates. Read that contract instead of claiming the feature is impossible or universally solves downstream effects. Stable business identities remain useful when multiple published messages represent the same intended operation.

When it breaks

A repeatedly failing message can block ordered progress, while a short deadline can cause overlapping attempts for otherwise valid work. Redelivery counts suggest where to investigate; they do not uniquely distinguish poison data, slow dependencies, lost acknowledgments, or stalled workers. Correlate identity, payload, exception, processing duration, and acknowledgment evidence before changing retry policy.

Bound retries and define how isolated failures are reviewed or replayed. Dead-lettering moves unresolved work; it does not complete it. Deduplication retention must cover the replay horizon you intend to tolerate. Removing the identity record can allow an old request to take effect again, while retaining identifiers indefinitely has storage and privacy costs.

Historical exact SQLite sink evidence covers rollback, reopen and duplicate handling. The total is a synthetic fixture value; broker behavior and external effects remain separate.

Check it quickly

Save as delivery_guarantee_probe.py and use Python 3 with assertions enabled, without -O or PYTHONOPTIMIZE: assertions contain required calls. Retained exact-fence records report Python 3.12.14 and SQLite 3.53.1 passing; this review did not rerun them. A temporary SQLite database stores identities and a harmless numeric total. This exercises a transactional sink; no account is charged or external message sent.

from contextlib import closing
from pathlib import Path
from tempfile import TemporaryDirectory
import sqlite3


def apply(db, message_id, amount, interrupt=False):
 db.execute('BEGIN IMMEDIATE')
 try:
 prior = db.execute('SELECT amount FROM inbox WHERE id=?', (message_id,)).fetchone()
 if prior is not None:
 if prior[0] != amount:
 raise ValueError('same identity, different payload')
 db.commit()
 return False
 db.execute('INSERT INTO inbox VALUES (?, ?)', (message_id, amount))
 db.execute('UPDATE totals SET amount=amount+? WHERE id=1', (amount,))
 if interrupt:
 raise RuntimeError('controlled failure before commit')
 db.commit()
 return True
 except BaseException:
 db.rollback()
 raise


with TemporaryDirectory(prefix='queue-sink-') as directory:
 path = Path(directory) / 'sink.db'
 with closing(sqlite3.connect(path, isolation_level=None)) as db:
 db.execute('CREATE TABLE inbox(id TEXT PRIMARY KEY, amount INTEGER NOT NULL)')
 db.execute('CREATE TABLE totals(id INTEGER PRIMARY KEY, amount INTEGER NOT NULL)')
 db.execute('INSERT INTO totals VALUES (1, 0)')
 try:
 apply(db, 'work-1', 100, interrupt=True)
 except RuntimeError:
 pass
 else:
 raise AssertionError('controlled failure did not occur')
 assert db.execute('SELECT count(*) FROM inbox').fetchone() == (0,)
 assert db.execute('SELECT amount FROM totals').fetchone() == (0,)
 assert apply(db, 'work-1', 100)
 assert apply(db, 'work-2', 50)
 # Close and reopen the real database rather than retaining an in-memory set.
 with closing(sqlite3.connect(path, isolation_level=None)) as db:
 assert not apply(db, 'work-1', 100)
 try:
 apply(db, 'work-1', 999)
 except ValueError:
 pass
 else:
 raise AssertionError('conflicting payload reused the same identity')
 assert db.execute('SELECT count(*) FROM inbox').fetchone() == (2,)
 assert db.execute('SELECT amount FROM totals').fetchone() == (150,)
 assert not db.in_transaction
 print('rollback kept marker and effect together; reopened retry changed nothing; total=150')

A controlled exception after both writes rolls back both the marker and total. Retrying then commits two operations. After closing and reopening the database, replaying the first identity changes nothing, while reusing it with a different amount raises an error. The final state contains two identities and a total of 150, outside any open transaction.

The experiment verifies rollback, committed-state persistence across connection reopening, and duplicate handling for this SQLite sink. It does not kill the process, simulate power failure, test concurrent consumers, or exercise broker publication and acknowledgment. Test those boundaries separately with the intended service before making a broader delivery claim.

Sources

Amazon SQS standard queues, at-least-once delivery and duplicate handling, checked 2026-10-15.
Amazon SQS visibility timeout, retry eligibility and duplicate limits, checked 2026-10-15.
Google Pub/Sub exactly-once delivery, scoped guarantees and publish-side duplicates, checked 2026-10-15.
SQLite transactions, explicit transactions, commit and rollback, checked 2026-10-15.
Python assert, optimization removes assertion evaluation, checked 2026-10-18.

Queues · XB Field Notes

Backpressure works when bounded admission changes the producer’s behavior

When it breaks

Check it quickly

Preserve the admission contract across the pipeline

Sources

Queues define delivery; consumers must protect effects

Locate publication, acknowledgment, and commitment

Make repeated effects harmless within a real boundary

When it breaks

Check it quickly

Further reading

Sources