Database · XB Field Notes

Postgres jsonb stores JSON for querying

Sat, 04 Dec 2027 00:00:00 +0000

PostgreSQL offers two JSON storage types with different representation costs. json validates and retains the input text, while jsonb converts it into a decomposed binary value. Both support extracting fields and paths. Choosing between them is primarily a decision about preserving representation, processing documents repeatedly, and supporting the operators your queries need.

JSON text is reflowed for display; object layout is schematic. Arrays retain order and repeats; preserve original request bytes separately.

Preserve text or preserve a value

A json value retains insignificant whitespace, object key order, and repeated object keys. Its processing functions still need to parse that text, and when duplicate keys occur they treat the last value as operative. Retaining duplicates therefore does not mean every consumer will interpret them the same way. Avoid using duplicate keys as a meaningful part of an application contract.

A jsonb value discards insignificant whitespace and object key order, retaining only the last occurrence of each object key. This statement concerns objects: arrays retain their element order and repeated elements. Array containment has its own matching rules, so containment results must not be mistaken for evidence that the stored array was reordered or deduplicated.

The binary format belongs to PostgreSQL; it is not BSON and is not a portable signing format. The textual rendering of jsonb is not a promise of a particular external canonicalization algorithm. If a signature covers original request bytes, preserve those bytes separately, for example in bytea, and verify against that representation. A json column preserves JSON input text, but client encoding conversion and protocol processing mean it is not a general archive of the original network bytes.

Extraction is available in both types

Both types support -> for a JSON value, ->> for text, and #> or #>> for a nested path. Extracting a field from json does not require first casting the whole document to jsonb. The performance distinction is reparsing stored text versus working with the decomposed representation, not an absence of extraction operators.

Jsonb additionally supports structural containment, key existence, and index strategies such as GIN. A scalar expression extracted from either type can also be a candidate for an appropriate expression index. Index selection still depends on the exact expression, operator, and workload; selecting jsonb alone does not make every nested condition fast. Frequently joined, constrained, or ordered attributes may belong in ordinary typed columns.

Input and processing costs also matter. Jsonb does extra work during conversion and can use more space for some documents. It accepts almost, but not exactly, the same inputs as json: PostgreSQL numeric limits and stricter Unicode handling can reject values accepted as json text. Test representative payloads before converting an existing column, especially unusually large numbers and escaped characters.

When it breaks

Changing json to jsonb can destroy representation that an exporter expected to replay. Changing types also does not enforce an application schema: a valid JSON string, number, or array is still JSON even when the application expects an object. Add the actual structural and business constraints rather than assuming that JSON validation checks required fields or their meaning.

A subtler failure is treating all parsers as interchangeable around duplicate keys. The database’s last-value behavior does not establish what an upstream signer or downstream parser used. Reject ambiguous input at the appropriate boundary when agreement matters. For query results, distinguish missing paths, JSON null, and SQL NULL rather than translating all three into an unexplained empty string.

Check it quickly

Save this as json-types.sql and run psql -X -v ON_ERROR_STOP=1 -f json-types.sql in a disposable PostgreSQL 18 database. It uses one temporary table inside a transaction and rolls everything back.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE sample (raw json, parsed jsonb);
INSERT INTO sample VALUES ('{ "b": 2, "a": 1, "a": 3 }','{ "b": 2, "a": 1, "a": 3 }');
SELECT raw::text, parsed::text, raw->>'a' AS json_a, parsed->>'a' AS jsonb_a FROM sample;
SELECT '{"user":{"name":"Ada"}}'::json #>> '{user,name}' AS json_path;
DO $$ BEGIN
 IF (SELECT raw::text FROM sample) IS DISTINCT FROM '{ "b": 2, "a": 1, "a": 3 }' THEN
 RAISE EXCEPTION 'json text was changed';
 END IF;
 IF (SELECT parsed FROM sample) IS DISTINCT FROM '{"a":3,"b":2}'::jsonb THEN
 RAISE EXCEPTION 'unexpected jsonb value';
 END IF;
 IF (SELECT raw->>'a' FROM sample) IS DISTINCT FROM '3' THEN
 RAISE EXCEPTION 'json extraction did not use the last duplicate key';
 END IF;
 IF '[2,1]'::jsonb = '[1,2]'::jsonb THEN
 RAISE EXCEPTION 'jsonb array order was ignored';
 END IF;
 IF '[2,1,2]'::jsonb = '[2,1]'::jsonb THEN
 RAISE EXCEPTION 'jsonb duplicate array elements were ignored';
 END IF;
END $$;
DO $$ BEGIN
 PERFORM '{broken'::json;
 RAISE EXCEPTION 'invalid JSON accepted';
EXCEPTION WHEN invalid_text_representation THEN RAISE NOTICE 'invalid JSON rejected';
END $$;
ROLLBACK;

The historical PostgreSQL 18.6 single-user check, without psql coverage, returned the original spaced text for raw, a reformatted object containing a=3 and b=2 for parsed, and 3 from both extraction expressions. The nested path on json returned Ada directly. Assertions cover retained text, the jsonb object value, json extraction, and array order and repetitions. The displayed jsonb extraction and nested path need inspection; the handled invalid-input error prints an “invalid JSON rejected” notice.

This exercise establishes representation and extraction behavior, not a speed or storage benchmark. Compare actual document sizes, write rates, and query plans separately before migrating a production column. Keep original data available until conversion failures and any intentional loss of representation have been accounted for.

Sources

PostgreSQL JSON types, representation, input restrictions, and indexing, checked 2026-10-15.
PostgreSQL JSON functions and operators, extraction operators for both types and null behavior, checked 2026-10-15.

A leaked connection drains the pool

Fri, 05 Nov 2027 00:00:00 +0000

A pool leak is an ownership failure: application work finishes, but its borrowed connection remains checked out. The database may still answer other clients normally while this application has no leases left. Treat release as part of the operation’s contract, including exceptions and early returns, rather than a cleanup courtesy after success.

A lease has an owner and an end

A SQLAlchemy Connection represents a borrow from its Engine’s pool. Closing that Connection normally returns the underlying DBAPI connection; it does not necessarily disconnect it. Returning the lease and committing a transaction are separate decisions. A pool can therefore be healthy while business changes were rolled back, or exhausted while its physical database connections remain alive.

Use the smallest ownership scope consistent with the transaction’s atomicity. with engine.begin() commits a successful block, rolls back a failed block, and releases its connection. with engine.connect() scopes the connection, but does not promise to commit writes: an outstanding transaction is rolled back when the connection closes. Do not replace one with the other without checking the intended persistence behavior.

Keep unrelated network waits and user interaction outside a database transaction. That does not justify splitting changes that must commit together. If a caller owns the connection, a helper should not secretly close it; if the helper borrows it, its interface must make any transfer of ownership explicit. Counting opens without identifying the responsible owner misses the real boundary.

When it breaks

A forgotten release reduces usable capacity until another checkout waits or times out. With a finite QueuePool and acquisition timeout, new callers need not hang forever: they receive a pool timeout before executing SQL. Raising the limit can delay recurrence while increasing pressure on the database; it does not repair the missing release.

Exhaustion alone does not prove a leak. Legitimate concurrency, a long transaction, a database lock, or an application deadlock can hold every lease. Flat database CPU is not decisive, because an application can hold a connection while waiting elsewhere. Correlate checked-out counts, checkout wait, lease age, transaction state, and the work that actually remains active.

A useful leak reproduction shows that completed work still owns leases and that returning those exact leases restores capacity. Do not depend on garbage collection to establish timely release. Likewise, Engine.dispose() handles idle connections in its pool, not forced reclamation of connections still checked out. Restarting a process may hide the ownership bug without correcting its next occurrence.

Pool capacity and persisted rows have separate lifecycles. Retained exact outputs support this local example; the current dependency workflow remains held.

Check it quickly

In an empty macOS or Linux directory, run python3 -m venv .venv and .venv/bin/python -m pip install SQLAlchemy==2.0.52. Save this as pool_probe.py, then run .venv/bin/python pool_probe.py. The original article reports Python 3.14.4 with SQLite 3.53.0. Retained exact-fence outputs record Python 3.12.14, SQLAlchemy 2.0.52 and SQLite 3.53.1; this review did not rerun installation or the pool exercise. Keep Python assertions enabled. The temporary database is local and automatically removed.

from pathlib import Path
from tempfile import TemporaryDirectory

import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy.exc import TimeoutError as PoolTimeout
from sqlalchemy.pool import QueuePool


class WorkFailed(Exception):
 pass


assert sqlalchemy.__version__ == "2.0.52"
with TemporaryDirectory(prefix="pool-leak-") as directory:
 engine = create_engine(
 "sqlite:///" + str(Path(directory) / "probe.db"),
 poolclass=QueuePool, pool_size=2, max_overflow=0, pool_timeout=0.05,
 )
 borrowed = []
 try:
 with engine.begin() as connection:
 connection.exec_driver_sql("CREATE TABLE ledger(id INTEGER PRIMARY KEY)")
 for _ in range(2):
 borrowed.append(engine.connect()) # Deliberately retain both leases.
 assert engine.pool.checkedout() == 2
 try:
 unexpected = engine.connect()
 except PoolTimeout:
 pass
 else:
 unexpected.close()
 raise AssertionError("retained leases must exhaust the two-slot pool")
 assert engine.pool.checkedout() == 2
 print("two retained leases exhaust the pool; the next checkout times out")

 borrowed[0].close()
 with engine.begin() as connection:
 connection.exec_driver_sql("INSERT INTO ledger VALUES (1)")
 assert engine.pool.checkedout() == 1
 print("returning one lease restores capacity without enlarging the pool")

 try:
 with engine.begin() as connection:
 connection.exec_driver_sql("INSERT INTO ledger VALUES (2)")
 raise WorkFailed("injected after the insert")
 except WorkFailed:
 pass
 assert engine.pool.checkedout() == 1
 with engine.connect() as connection:
 assert connection.exec_driver_sql("SELECT id FROM ledger").scalars().all() == [1]
 connection.exec_driver_sql("INSERT INTO ledger VALUES (3)")
 # No commit: this connection context rolls back when it closes.
 with engine.connect() as connection:
 assert connection.exec_driver_sql("SELECT id FROM ledger").scalars().all() == [1]
 print("an exception rolls back row 2 and returns its lease; closing without commit also removes row 3")

 def early_return():
 with engine.connect() as connection:
 return connection.exec_driver_sql("SELECT 1").scalar_one()

 assert early_return() == 1 and engine.pool.checkedout() == 1
 with engine.begin() as connection:
 connection.exec_driver_sql("INSERT INTO ledger VALUES (4)")
 with engine.connect() as connection:
 rows = connection.exec_driver_sql("SELECT id FROM ledger ORDER BY id").scalars().all()
 assert rows == [1, 4] and engine.pool.checkedout() == 1
 borrowed[1].close()
 assert engine.pool.checkedout() == 0 and engine.pool.checkedin() == 2
 print("early return releases its lease; successful transaction commits row 4; final rows", rows)
 finally:
 for connection in borrowed:
 connection.close()
 engine.dispose()

The list deliberately holds strong references to both borrowed connections, making the simulated leak independent of garbage collection. A third checkout raises PoolTimeout. Returning one retained lease lets a transaction insert row 1 without enlarging the pool. One deliberate leak remains, so the expected checked-out count between properly scoped operations is one, not zero.

The injected WorkFailed exception occurs after inserting row 2. Exiting the transaction context rolls that insert back and returns the lease. A separate connection context inserts row 3 without committing; closing it also removes that row. An early return still exits its connection context. Finally, a successful transaction commits row 4, and the ordered result must be exactly [1, 4].

Closing the last intentionally retained lease brings checked-out count to zero while two physical connections remain idle for reuse. The final cleanup returns all retained handles before disposing the Engine. These assertions test local pool accounting and transaction outcomes, not production lock contention, cancellation, or distributed side effects. Returning a connection does not undo an HTTP call already sent elsewhere. Repeat the ownership test around the actual failure path before changing pool capacity.

Sources

SQLAlchemy connection pooling, checkouts, return and finite QueuePool limits, checked 2026-09-09.
SQLAlchemy Engine and Connection, transaction contexts, close and disposal semantics, checked 2026-09-09.
SQLAlchemy pool exhaustion errors, leaks, long transactions, concurrency and deadlocks, checked 2026-09-09.

LSM trees trade foreground writes for later merging

Tue, 02 Nov 2027 00:00:00 +0000

An LSM tree moves work from scattered updates toward buffered writes and later merging. That is a design trade, not a promise that every disk operation becomes sequential or that random writes always beat a B-tree. The useful question is whether foreground writes, background compaction, reads, and retained versions fit the same resource budget.

Buffering changes when work happens

RocksDB normally inserts writes into a memtable and can record them in a write-ahead log. WAL enablement and synchronization are separate durability choices; merely using an LSM layout does not promise that every acknowledged write survives power loss. A full memtable becomes an immutable sorted table. Flushes create files in level zero, while application writes continue through other available buffers.

With leveled compaction, level-zero files may overlap in key range. Deeper levels normally form one sorted run per level, divided among multiple nonoverlapping SST files. This is not one file per level. File boundaries help a lookup select candidates; compaction merges overlapping ranges into replacement files. Dynamic level sizing can also leave intermediate levels empty, so a fixed picture of every level being populated is misleading.

Reads still pay for versions

A key may have older values in several runs. A lookup must choose the visible version, and a deletion marker must stop an older value from reappearing. Bloom filters can reject candidate files that definitely lack the key; a positive answer still requires further checking. They can help successful overall lookups by skipping unrelated files before reaching the file containing the answer. They are not exclusively useful for absent keys.

Compaction rewrites data to reduce this search and space burden. Leveled and universal policies commonly trade write, read, and space amplification differently, but workload, compression, caching, and configuration determine the result. Compare like accounting: SST bytes written, WAL bytes, device writes, and logical application bytes are different quantities. A ratio without its numerator and denominator cannot size storage bandwidth reliably.

When it breaks

When flush or compaction work cannot keep up, immutable buffers and level-zero files can accumulate, reads can become more expensive, and the engine may slow or stop foreground writes. A nonzero historical stall counter alone does not diagnose today’s bottleneck. Inspect recent stall time, pending work, file counts, storage latency, and application throughput together. Raising a threshold may defer pressure while increasing the eventual recovery cost.

Deletes do not instantly reclaim every older value. A tombstone cannot safely disappear while a lower run can still expose the value it hides, or a retained snapshot requires relevant history. Reclaiming space depends on the compaction coverage and version-retention rules. Dividing physical bytes by zero live bytes also produces no finite space-amplification ratio; report absolute retained bytes for an entirely deleted dataset.

Merge versions; keep deletion safe. Source-based mechanism illustration. Model values and source-reported observations are not a new runtime test.

Check it quickly

Save this as lsm_model.py and run Python 3. The standard-library program is an explicit in-memory model of three flushed runs and complete-history compaction. It does not execute RocksDB, create SST files, perform I/O, simulate its scheduler, or measure latency. None of the printed counts should be described as production benchmark results.

from pprint import pprint

# A semantic model: sorted in-memory runs, not RocksDB or a disk benchmark.
runs = [] # newest run first
writes = 0
sequence = 0

def flush(operations):
 global writes, sequence
 memtable = {}
 for key, value in operations:
 sequence += 1
 memtable[key] = (sequence, value)
 run = dict(sorted(memtable.items()))
 runs.insert(0, run)
 writes += len(run)

def get(key, tables):
 for run in tables:
 if key in run:
 return run[key][1] # None is a tombstone: stop searching.
 return None

flush([("a", "one"), ("b", "old")])
flush([("a", "two"), ("c", "three")])
flush([("b", None)])
assert [get(k, runs) for k in "abc"] == ["two", None, "three"]
assert writes == 5 and sequence == 5
# Dropping a tombstone from only the newest run resurrects old b.
unsafe = [{k: v for k, v in runs[0].items() if v[1] is not None}] + runs[1:]
assert get("b", unsafe) == "old"
# Merge ALL history, with no retained snapshots, before dropping tombstones.
latest = {}
for run in reversed(runs):
 latest.update(run)
merged = {k: v for k, v in sorted(latest.items()) if v[1] is not None}
writes += len(merged)
assert [get(k, [merged]) for k in "abc"] == ["two", None, "three"]
assert (len(merged), writes, sequence) == (2, 7, 5)
pprint({"live_keys": list(merged), "logical_operations": sequence,
 "table_records_written": writes, "record_write_ratio": writes / sequence,
 "unsafe_partial_merge_resurrects": get("b", unsafe)})

The first flush stores a and b; the second replaces a and adds c; the third deletes b. Reading across all runs returns the newer a, no b, and c. Deliberately dropping only the newest tombstone resurrects the older b, demonstrating why a locally empty-looking deletion record cannot simply be discarded.

The safe model merge examines every run and assumes no snapshots. It retains two values and writes two more table records, giving seven record writes for five logical operations. This is a record-count ratio of 1.4, not byte-based write amplification: variable value sizes, the WAL, compression, metadata, and intermediate real compactions are absent.

For an actual engine, reproduce representative inserts, updates, deletes, and reads in an isolated database, record options and hardware, then inspect its measured counters. An SQLite timing comparison cannot establish RocksDB behavior, and a backlog bandwidth difference cannot directly predict disk-fill time without a model of retention, reclamation, throttling, and available capacity.

Sources

RocksDB overview, memtables, optional WAL, persistence and compaction, checked 2026-10-15.
RocksDB leveled compaction, sorted runs, files, level sizing and compaction pressure, checked 2026-10-15.
RocksDB Bloom filters, candidate filtering and false positives, checked 2026-10-15.
RocksDB snapshots, version retention during compaction, checked 2026-10-15.

Checkpoints establish a recovery starting point

Sun, 17 Oct 2027 00:00:00 +0000

A checkpoint records progress that lets a database recover without treating its entire log history as outstanding work. The exact operation depends on the engine. PostgreSQL synchronizes the data changes required for a checkpoint and establishes a redo starting point. SQLite in WAL mode transfers eligible committed page frames from its WAL into the main database file. Neither operation is simply another name for transaction commit.

When it breaks

Checkpoint requests and complete checkpoints are different observations. SQLite readers can hold snapshots that prevent newer WAL frames from being copied into the database file or prevent the log from being reset. A passive checkpoint can return successfully after doing only the work currently possible. Reading only its success flag hides the remaining frame count and the reader that still prevents completion.

Log size also needs context. Reusing an existing file can preserve its allocated size while useful checkpoint progress advances. PostgreSQL can retain older WAL for archives or replication slots even after a checkpoint. Removing log files manually to make a graph smaller can destroy recovery material. Check the engine’s progress markers and retention obligations before equating filesystem size with unfinished recovery work.

An interval is not a hard recovery-time bound. Replay speed, storage throughput, workload shape, and the actual crash point affect recovery duration. More frequent checkpoints may reduce later replay while increasing normal write traffic. Moving WAL onto another device does not move all the data-page writes there or make the checkpoint’s cost disappear.

SQLite completion depends on releasing the old reader; the update committed before the final checkpoint. Historical local evidence only.

Check it quickly

Save the following as checkpoint_lab.py and run python3 checkpoint_lab.py with assertions enabled. It uses Python’s standard sqlite3 module and private temporary files. The recorded run used Python 3.14.4 with SQLite 3.53.0. Check your runtime’s SQLite version when comparing results, since the Python and database engine versions are distinct.

import sqlite3
import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory(prefix="xbcatm-checkpoint-") as directory:
 path = Path(directory) / "test.db"
 writer = sqlite3.connect(path, isolation_level=None, timeout=0.1)
 reader = sqlite3.connect(path, isolation_level=None, timeout=0.1)
 try:
 assert writer.execute("PRAGMA journal_mode=WAL").fetchone() == ("wal",)
 writer.execute("PRAGMA synchronous=FULL")
 writer.execute("PRAGMA wal_autocheckpoint=0")
 writer.execute("CREATE TABLE marker (id INTEGER PRIMARY KEY, value INTEGER)")
 writer.execute("INSERT INTO marker VALUES (1,10)")
 assert writer.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone() == (0, 0, 0)
 reader.execute("BEGIN")
 assert reader.execute("SELECT value FROM marker").fetchone() == (10,)
 writer.execute("UPDATE marker SET value=20 WHERE id=1")
 assert writer.execute("SELECT value FROM marker").fetchone() == (20,)
 passive = writer.execute("PRAGMA wal_checkpoint(PASSIVE)").fetchone()
 blocked = writer.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()
 print("passive", passive)
 print("truncate_with_reader", blocked)
 assert passive[0] == 0 and passive[1] > passive[2] >= 0
 assert blocked[0] == 1 and blocked[1] > blocked[2]
 assert Path(str(path) + "-wal").stat().st_size > 0
 assert reader.execute("SELECT value FROM marker").fetchone() == (10,)
 reader.execute("COMMIT")
 completed = writer.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()
 print("truncate_after_reader", completed)
 assert completed == (0, 0, 0)
 assert Path(str(path) + "-wal").stat().st_size == 0
 assert reader.execute("SELECT value FROM marker").fetchone() == (20,)
 print("fresh_reader", 20)
 finally:
 reader.close()
 writer.close()

The initial truncate checkpoint moves value ten into the database file and resets the WAL. A reader then opens an explicit transaction and reads ten. Another connection commits value twenty with automatic checkpointing disabled. The writer can read twenty immediately, while the old reader still needs its original snapshot. This schedule makes the constraint visible without relying on concurrent thread timing.

In the recorded run, passive checkpoint output is (0, 1, 0): no blocking failure is reported, one frame exists, and none is checkpointed. The assertions deliberately check the relationship between frame counts instead of requiring that exact count on every engine build. The meaningful result is that a successful passive call left work behind.

The truncate request next returns (1, 1, 0) after the short busy wait. Its first field signals that the requested completion was blocked. The WAL remains nonempty, and the old reader still sees ten. The Python call returning a tuple is therefore not evidence that truncation succeeded; the tuple’s status must be inspected.

After the reader commits, another truncate checkpoint returns (0, 0, 0). The script checks the actual WAL length is zero and a fresh read sees twenty. The update was already committed before this final checkpoint. Both connections close and the temporary directory is removed even if an assertion fails.

Diagnose the work that remains

For SQLite, inspect long-lived read transactions and unfinished cursors when checkpoints cannot finish. Decide whether passive progress is sufficient or a blocking checkpoint belongs in a controlled maintenance window. Increasing the busy timeout merely waits longer for the same constraint; it does not release an application transaction. Forcing truncation after every write can add contention and defeat the concurrency benefit of WAL mode.

For PostgreSQL, use checkpoint and WAL-retention statistics appropriate to the deployed version. A completed checkpoint can coexist with retained log files and ongoing transactions. Compare foreground latency and a measured restart rehearsal when changing checkpoint frequency. This fixture verifies SQLite’s completion boundary; it does not measure a production recovery-time objective or claim identical checkpoint APIs across database engines.

Sources

SQLite write-ahead logging, reader snapshots, checkpoint progress, and WAL reuse, checked 2026-09-12.
SQLite checkpoint API, passive, restart, and truncate completion behavior, checked 2026-09-12.
SQLite checkpoint pragma, returned busy flag and frame counts, checked 2026-09-12.
PostgreSQL 18 WAL and checkpoints, checkpoint costs and independent WAL retention requirements, checked 2026-09-12.

Prepared statements separate values from SQL structure

Sat, 16 Oct 2027 00:00:00 +0000

Parameter binding gives a database value a designated position in a SQL statement. A name containing a quote or a semicolon remains a value instead of becoming another clause. Reusing a named prepared statement adds a separate optimization: the server can retain statement information across executions in one session. Safe binding does not require the application to create a named statement for every query.

Binding protects the SQL grammar boundary, not every language interpreted inside a value. A bound LIKE pattern still treats percent and underscore as wildcards. Decide whether an endpoint accepts a pattern or a literal substring and escape for that operator when needed. Regex, full-text search, and authorization each have their own semantics after safe binding.

When it breaks

In Psycopg, %s is a driver placeholder, not an invitation to use Python string formatting. Pass the values as the second argument to execute, leave the placeholder unquoted, and use a one-element tuple for one positional value. Interpolating a value first and then calling a parameter-capable API has already crossed the boundary the API was supposed to preserve.

A placeholder represents a value, not a table, column, or keyword. Dynamic identifiers require a deliberate structural choice. Use an allow-list to restrict the supported columns, then the driver’s identifier composition facility to quote that identifier correctly. Keep direction keywords under application control. Numeric LIMIT and OFFSET values can be bound, while still requiring product-specific range validation.

Preparation does not freeze the first execution plan forever. PostgreSQL distinguishes custom plans, built using current parameter values, from generic plans reused across values. Automatic selection considers planning overhead and estimated execution costs. Schema or statistics changes can also trigger replanning. A statement can remain parameterized while its access path changes; plan reuse and injection resistance are different properties.

Binding works with zero named statements. Named reuse and custom/generic planning are separate properties. The source fixture was not rerun; forced plan modes make no latency claim.

Check it quickly

Use Python with Psycopg 3 installed and set PG_DSN to a disposable PostgreSQL database. Supply connection configuration through your existing secure mechanism. The recorded run used Psycopg 3.3.5 and PostgreSQL 18.6. Save the following as a Python file and execute it with that environment variable set. The table is temporary and disappears when the connection closes:

import os
import psycopg
from psycopg import sql

with psycopg.connect(os.environ['PG_DSN'], autocommit=True,
 prepare_threshold=5) as conn:
 conn.execute('CREATE TEMP TABLE people (id integer PRIMARY KEY, label text NOT NULL)')
 conn.execute("INSERT INTO people VALUES (1,'alice'),(2,'bob')")
 query = 'SELECT id FROM people WHERE label=%s ORDER BY id LIMIT %s'
 hostile = "' OR true --"
 assert conn.execute(query, (hostile, 10), prepare=False).fetchall() == []
 count = conn.execute('SELECT count(*) FROM pg_prepared_statements').fetchone()[0]
 assert count == 0
 print('bound without named preparation:', count, 'named statements; no match')
 for value in ("O'Reilly", r'a\b', 'x; DROP TABLE people; --'):
 assert conn.execute('SELECT %s::text', (value,), prepare=False).fetchone() == (value,)
 for label, expected in (('alice', [(1,)]), ('bob', [(2,)])):
 assert conn.execute(query, (label, 10), prepare=True).fetchall() == expected
 assert conn.execute('SELECT count(*) FROM pg_prepared_statements').fetchone() == (1,)
 print('named preparation reused: alice=1, bob=2')

 def ordered(column, limit):
 if column not in {'id', 'label'}:
 raise ValueError('unsupported sort column')
 if type(limit) is not int or not 1 <= limit <= 100:
 raise ValueError('limit must be an integer from 1 through 100')
 statement = sql.SQL('SELECT id FROM people ORDER BY {} LIMIT %s').format(
 sql.Identifier(column))
 return conn.execute(statement, (limit,), prepare=False).fetchall()

 assert ordered('label', 1) == [(1,)]
 for column, limit in [('label; DROP TABLE people', 1), ('id', 0), ('id', 101)]:
 try:
 ordered(column, limit)
 except ValueError:
 pass
 else:
 raise AssertionError('invalid request accepted')
 print('identifier allow-list and bound LIMIT checked')

 conn.execute('PREPARE plan_lab(text) AS SELECT id FROM people WHERE label=$1')
 conn.execute('SET plan_cache_mode=force_custom_plan')
 assert conn.execute("EXECUTE plan_lab('alice')").fetchall() == [(1,)]
 counters = "SELECT generic_plans,custom_plans FROM pg_prepared_statements WHERE name='plan_lab'"
 assert conn.execute(counters).fetchone() == (0, 1)
 conn.execute('SET plan_cache_mode=force_generic_plan')
 assert conn.execute("EXECUTE plan_lab('bob')").fetchall() == [(2,)]
 assert conn.execute(counters).fetchone() == (1, 1)
 print('same prepared statement: custom=1, generic=1')
 conn.execute('RESET plan_cache_mode')
 conn.execute('DEALLOCATE plan_lab')

The first result reports zero named statements and no match for the hostile value. That call explicitly disables named preparation while retaining binding, so it demonstrates that the safety boundary exists before statement reuse. Quotes, backslashes, and semicolons then round-trip unchanged; the table remains available for subsequent queries.

Two calls with prepare=True reuse one named statement and return Alice’s identifier one and Bob’s identifier two. The configured threshold permits explicit preparation. Setting prepare_threshold=None instead disables named preparation altogether in the tested driver, including explicit requests. Do not confuse that switch with disabling value binding.

The ordering function returns one row for a supported column and bound limit. It rejects an unsupported column and limits outside one through one hundred before issuing SQL. Identifier quoting protects syntax; the allow-list defines which valid syntax the endpoint actually supports. Neither decision should be delegated to arbitrary request text.

Finally, the same SQL prepared statement executes once under forced custom planning and once under forced generic planning. Its counters change from zero generic and one custom plan to one of each. Both return the expected row. These forced settings make the distinction observable; they do not measure what automatic planning would choose for a realistic skewed workload or recommend forcing either mode globally.

Inspect reuse in the real deployment

PostgreSQL’s default automatic policy starts parameterized prepared statements with custom plans and later compares a generic alternative. Look at representative parameter distributions and actual execution plans before deciding whether reuse helps. This two-row exercise makes no latency or network-round-trip claim. Driver protocol batching, connection setup, and query cost affect those measurements independently.

Named statements belong to server sessions. Connection pools and transaction-pooling proxies must manage that lifetime correctly, using explicitly supported prepared-statement behavior or disabling named reuse where necessary. Binding remains the default even for a query executed only once. Continue to enforce tenant predicates, object permissions, and result-size limits: a safely parameterized query can still return data the caller was never meant to see.

The operational fixture is unchanged here and was not rerun in this review. Its expected outputs and earlier observations remain distinct from current execution evidence.

Sources

Psycopg parameter API, binding syntax and identifier composition, checked 2026-10-15.
Psycopg prepared statements, thresholds, explicit preparation, and session lifetime, checked 2026-10-15.
PostgreSQL 18 PREPARE, custom and generic plans and automatic selection, checked 2026-10-15.
PostgreSQL 18 pattern matching, LIKE wildcard and escape semantics, checked 2026-10-15.

Partial uniqueness protects the active subset

Fri, 15 Oct 2027 00:00:00 +0000

A partial index stores entries only for rows satisfying its predicate. This can reduce index size when the useful subset is small, but it can also enforce a rule that applies only to that subset. For example, a seat may have many historical reservations while allowing at most one active reservation. A partial unique index expresses that rule at the database boundary.

Membership follows the predicate

CREATE INDEX ... WHERE active includes rows for which active evaluates to true. False and null do not satisfy the predicate. Whether null should be allowed is a separate schema decision; the experiment makes active non-null so each reservation is explicitly active or inactive. The indexed key need not be the same column used by the predicate.

Membership changes with ordinary writes. An inactive row becoming active must enter the index and pass its checks; an active row becoming inactive leaves the active subset. This is not an immutable archive boundary. Predicate evaluation and changing membership still cost work, so a smaller index does not mean every update is free or automatically faster.

Uniqueness can apply to a subset

A unique index on seat WHERE active rejects two active rows for the same seat, while permitting any number of inactive historical rows. The index enforces that rule even if no SELECT ever chooses it. Query acceleration and integrity enforcement are different reasons to retain an index, so usage counters alone cannot justify deleting this one.

An application-level “check first, insert later” is weaker because another transaction can insert between the check and the write. A unique index coordinates conflicting writes in PostgreSQL; the loser must handle a uniqueness error or another chosen conflict policy. The fixture exercises rejection sequentially, while concurrency behavior remains part of the database’s constraint contract rather than a claim about an application booking workflow.

Plan queries with the same boundary

To use a partial index as an access path, the planner must recognize that the query’s conditions imply the index predicate. Merely querying the seat does not prove the caller wants only active rows. A query that explicitly asks for active reservations can qualify, although a tiny table may still be cheaper to scan. A logically similar expression is useful only if the planner can establish the implication.

Do not create hundreds of overlapping partial indexes to simulate partitioning. Planning and maintenance costs grow, and the optimizer does not automatically understand them as one coordinated partition scheme. Choose a predicate from a stable workload or integrity requirement, then verify the actual query and transition behavior.

When it breaks

Forgetting the predicate in a query can return old reservations alongside the current one, even though active uniqueness is enforced perfectly. Conversely, treating the index as globally unique can reject valid history in application logic. Document whether the rule concerns all rows or only the current subset, and test both populations.

A failed reactivation is another important boundary. Changing an old inactive reservation back to active can conflict with the current holder. The application must keep the old row inactive when that update fails and report the conflict, not silently replace the active booking. Bulk imports and status migrations need the same checks as ordinary insertion.

Predicate membership and uniqueness transitions are separate from planner selection. Historical sequential SQL evidence does not establish concurrent bookings.

Check it quickly

Save this block as partial-unique.sql and run psql -X -v ON_ERROR_STOP=1 -f partial-unique.sql in a disposable PostgreSQL 18 database. It creates a temporary reservation table, so no real seat inventory is touched.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE reservations (id integer PRIMARY KEY, seat text NOT NULL, active boolean NOT NULL);
CREATE UNIQUE INDEX one_active_seat ON reservations (seat) WHERE active;
INSERT INTO reservations VALUES (1,'A1',true),(2,'A1',false),(3,'A1',false);
DO $$ BEGIN
 INSERT INTO reservations VALUES (4,'A1',true);
 RAISE EXCEPTION 'two active reservations accepted';
EXCEPTION WHEN unique_violation THEN RAISE NOTICE 'active duplicate rejected';
END $$;
UPDATE reservations SET active=false WHERE id=1;
INSERT INTO reservations VALUES (4,'A1',true);
DO $$ BEGIN
 UPDATE reservations SET active=true WHERE id=2;
 RAISE EXCEPTION 'reactivation bypassed uniqueness';
EXCEPTION WHEN unique_violation THEN RAISE NOTICE 'conflicting reactivation rejected';
END $$;
SELECT active,count(*) FROM reservations GROUP BY active ORDER BY active;
SELECT id FROM reservations WHERE active AND seat='A1';
DO $$ BEGIN
 IF (SELECT id FROM reservations WHERE active AND seat='A1') IS DISTINCT FROM 4 THEN
 RAISE EXCEPTION 'wrong active reservation';
 END IF;
 IF (SELECT count(*) FROM reservations WHERE NOT active) <> 3 THEN
 RAISE EXCEPTION 'history was not preserved';
 END IF;
END $$;
ROLLBACK;

The first active duplicate is rejected while two historical duplicates are accepted. Deactivating reservation one releases the seat for reservation four. Reactivating reservation two then fails because four is active. The final counts are three inactive rows and one active row, whose identifier is four. Assertions check both the current holder and retained history.

Retained PostgreSQL 18.6 standalone SQL logs record these checks passing; this source review did not rerun them. That earlier adaptation omitted the psql directive and folded statement whitespace, so client invocation and concurrent bookings remain untested. The exception blocks catch only the expected uniqueness violation; an unexpected success raises a different error and fails the script. Rollback removes the fixture. This verifies subset membership and uniqueness transitions, not payment atomicity, distributed reservations, or the performance of a large historical table.

Sources

PostgreSQL partial indexes, predicate membership and partial uniqueness, checked 2026-10-15.
PostgreSQL CREATE INDEX, unique index enforcement and predicates, checked 2026-10-15.

Covering indexes can avoid heap visits

Thu, 14 Oct 2027 00:00:00 +0000

A covering index contains the values needed by a particular query. PostgreSQL can sometimes return those values through an index-only scan, avoiding heap visits for qualifying rows. The word “sometimes” matters: having the columns is a structural requirement, while visibility information determines whether execution can actually skip the heap. An Index Only Scan label alone does not prove zero table access.

Coverage supplies values; the heap page’s visibility bit governs whether an index-only scan must visit the heap. Writes and eligible vacuum can change that state. Heap visits are not necessarily physical disk reads.

Cover the actual projection

For a lookup by sku that returns price, a B-tree on sku with INCLUDE (price) stores price as payload. The sku key guides the search; the included price does not become another search key or establish an ordering by price. If the index is unique, uniqueness applies to its key columns, not to the included payload.

A later query change can invalidate the coverage assumption. Selecting a description column absent from the index requires another source for that value, usually the heap. Selecting every column casually can therefore turn a carefully covered lookup into a different access pattern. Start from the real predicate and projected columns, including expressions and visibility needs, rather than adding all table columns to a speculative index.

Not every access method can reconstruct complete values for index-only execution. B-tree supports it, while GIN generally stores extracted keys rather than the complete original value and does not support index-only scans. A matching index must support the needed operation as well as contain or reconstruct the requested data. Similar terminology across database engines does not imply identical implementation details.

The visibility map decides heap visits

PostgreSQL needs to know whether a candidate tuple is visible to the query’s MVCC snapshot. Index entries do not carry all the required visibility information. An index-only scan checks the heap page’s all-visible bit in the visibility map; when that bit is set, the value can be returned without fetching the heap tuple.

A write can clear the page’s all-visible status, making later scans visit the heap again even when their plan name remains unchanged. Vacuum can establish all-visible status when the page’s tuples meet the visibility requirements. Long-running transactions and ongoing changes affect that process. It is a page-level property, so an update to one row can influence the visibility check for another row on that page.

When it breaks

The common mistake is declaring success from EXPLAIN’s plan label. Read Heap Fetches and relevant buffers in EXPLAIN ANALYZE, and confirm the actual returned values. A write-heavy table may require frequent heap checks, reducing the read benefit while still paying for duplicated index payload. Repeating VACUUM after every application write is not a substitute for evaluating the workload.

Wide payload columns also consume space and cache, increase maintenance work, and can exceed index tuple limits. Updating an included column must maintain the index and can prevent a heap-only update that might otherwise have been possible. Include only values whose read benefit justifies their write and storage cost; a covering index is not a free materialized copy of the table.

Check it quickly

Save this as covering.sql and run psql -X -v ON_ERROR_STOP=1 -f covering.sql against a disposable PostgreSQL 18 database. Keep psql’s normal autocommit behavior: VACUUM cannot run inside an explicit transaction block. The temporary table remains confined to this session.

\set ON_ERROR_STOP on
CREATE TEMP TABLE products (sku integer, price integer);
INSERT INTO products SELECT g, g*10 FROM generate_series(1,10000) AS g;
CREATE INDEX products_price_lookup ON products (sku) INCLUDE (price);
VACUUM (ANALYZE) products;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT price FROM products WHERE sku=4321;
UPDATE products SET price=price+5 WHERE sku=4321;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT price FROM products WHERE sku=4321;
VACUUM (ANALYZE) products;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT price FROM products WHERE sku=4321;
DO $$ BEGIN
 IF (SELECT price FROM products WHERE sku=4321) IS DISTINCT FROM 43215 THEN
 RAISE EXCEPTION 'covering lookup returned an obsolete price';
 END IF;
END $$;
DROP TABLE products;

The first vacuum makes this quiescent fixture eligible for heap-free reads. The retained PostgreSQL 18.6 report used Index Only Scan in all three plans, with Heap Fetches zero, then two after the price update, then zero after vacuum. The assertion confirms the new price is 43215 rather than an obsolete indexed value. Temporary objects are removed explicitly and also disappear when the session ends.

The exact intermediate count can vary with implementation and page state; its significance is that heap access can occur under the same plan label. Heap Fetches counts heap visits, not necessarily physical disk reads: the pages may already be cached. This is a controlled visibility demonstration, not a guarantee that production vacuum will immediately make every page all-visible. Compare representative reads before and after real update activity when deciding whether coverage pays off.

Sources

PostgreSQL index-only scans, visibility maps and included payload, checked 2026-09-10.
PostgreSQL CREATE INDEX, INCLUDE semantics and index restrictions, checked 2026-09-10.

Query planners bet on cost, not certainty

Fri, 17 Sep 2027 00:00:00 +0000

A query planner chooses an execution path using estimated work. It does not run every candidate, time them, and keep the winner. PostgreSQL compares possible scans, joins, sorts, and other operations using statistics and a cost model. The chosen plan is therefore a prediction under current assumptions. Understanding a wrong prediction requires identifying which assumption failed, rather than treating a sequential scan or a nested loop as inherently suspicious.

Dependency statistics correct one correlation estimate after ANALYZE, yet miss an incompatible value pair. Compare predicted and observed work. Cost units are not milliseconds, and improved row estimates alone do not demonstrate faster execution.

Costs describe a model

PostgreSQL costs combine estimates of page access, tuple processing, operator work, and related activity. These units are not milliseconds. Startup cost matters when a consumer needs only a few rows, while total cost matters when the full result must be produced. Useful ordering can avoid a later sort, so the cheapest isolated scan is not necessarily the best input to the complete query.

The search also has limits. PostgreSQL considers alternative paths and join orders rather than executing them all. Large join problems can use the genetic query optimizer to control planning effort. Available indexes, legal transformations, configuration, and search choices constrain the candidates. Calling the selected plan mathematically optimal over every imaginable implementation overstates what the optimizer guarantees.

When it breaks

Fresh statistics can still produce a bad estimate. Ordinary statistics describe columns individually, so combining two predicates may assume independence even when the columns always move together. An underestimated intermediate result can make repeated lookups appear cheaper than they become, or distort later join choices. Repeating the same analysis without changing the statistical model cannot teach it a missing relationship.

Extended statistics address particular questions, not every estimation error. Functional dependencies help compatible equality conditions against constants, but do not establish the actual mapping between values. They are not general proofs for range filters or cross-column comparisons. An incompatible pair of constants can remain badly estimated even after a dependency is collected.

Check it quickly

Run this temporary-table experiment with PostgreSQL 18 and psql. Both columns contain the same one hundred distinct values, each repeated one hundred times. The final rollback removes the table and its statistics object.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE dimensions (a integer,b integer);
INSERT INTO dimensions SELECT g%100,g%100 FROM generate_series(1,10000) AS g;
ANALYZE dimensions;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT * FROM dimensions WHERE a=7 AND b=7;
CREATE STATISTICS pg_temp.dimensions_dependency (dependencies) ON a,b FROM dimensions;
ANALYZE dimensions;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT * FROM dimensions WHERE a=7 AND b=7;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT * FROM dimensions WHERE a=7 AND b=8;
DO $$ BEGIN
 IF (SELECT count(*) FROM dimensions WHERE a=7 AND b=7) <> 100 OR
 (SELECT count(*) FROM dimensions WHERE a=7 AND b=8) <> 0 THEN RAISE EXCEPTION 'wrong correlated population'; END IF;
END $$;
ROLLBACK;

In the retained PostgreSQL 18.6 report, before extended statistics the planner estimated one row for a=7 AND b=7, although the query returned one hundred. Each condition individually selects one percent; multiplying those fractions predicts one hundredth of a percent of ten thousand rows. The calculation is reasonable only under an independence assumption that this fixture deliberately violates.

Creating the dependencies object tells the next ANALYZE which relationship to collect. The second plan estimated one hundred rows and returned one hundred. Merely defining the object would not have populated it. The scan remained sequential because this example has no index and still examines the same small table. A better cardinality estimate is demonstrated; a faster application is not.

The final predicate asks for a=7 AND b=8. It returns zero rows, but the dependency estimate remains one hundred. This is the important negative case: knowing that one column determines another does not tell this statistics kind that seven must map specifically to seven. The assertions distinguish correct query results from inaccurate predictions; estimation errors do not change SQL semantics.

Choose the next experiment from the discrepancy

A multivariate common-value list can represent frequent value combinations when that is the missing information. Other workloads may need different statistics, an appropriate index, or a rewritten expression that exposes a usable condition. Add only the object justified by the observed predicates; unnecessary statistics consume analysis and planning resources too.

When comparing alternatives, keep the dataset, parameters, relevant settings, and result requirements consistent. Include rows processed, buffer activity, memory spills, and measured latency. A locally cheaper-looking node is not sufficient evidence if it creates more work upstream. Conversely, accurate row estimates do not rule out waiting, costly I/O, or an unsuitable cost model. Follow the specific difference between predicted and observed work before changing global planner settings.

Sources

PostgreSQL 18: Planner and optimizer, candidate access paths and join strategies, checked 2026-09-10.
PostgreSQL 18: Planner statistics, column distributions and extended-statistics limits, checked 2026-09-10.
PostgreSQL 18: Multivariate statistics examples, correlated predicates and dependency estimates, checked 2026-09-10.

Full table scans can win on small tables

Thu, 16 Sep 2027 00:00:00 +0000

An index trades storage and write work for a possible reduction in read work. A full scan can win when the table is small or a predicate selects much of it, but neither row count nor selectivity decides alone. Projection, ordering, caching, row width, and available access paths change the comparison. Start with the query’s actual result contract.

Understand the competing work

A B-tree range scan does not descend from the root for every matching index entry. After finding the range, it can advance through adjacent entries. A non-covering path may then need table lookups for missing columns; those lookups can dominate a broad query. A covering index can avoid them, so even a query matching most rows can benefit from an index.

Pages matter, but they are not the only currency. PostgreSQL’s default seq_page_cost and random_page_cost are relative planner estimates, not milliseconds or a charge mechanically applied to every returned row. Caching, correlation, CPU work, bitmap scans, and index-only scans matter too. SQLite stores ordinary table rows in a rowid B-tree; that is a different storage model from PostgreSQL’s heap. Do not transfer a measured ratio between them.

SQLite’s PRAGMA page_count reports the whole database file, including indexes and other objects. It is not the page count of one named table. Likewise, logical page traversal is not a guarantee of contiguous physical disk reads. Compare consistent object scopes and record page size, schema, payload width, and cache conditions alongside row count.

Keep earlier measurements in context

Earlier notes recorded SQLite 3.53 measurements with a 2 MB page cache. For a broad predicate, scan/index times were 0.04/0.05 ms at 1,000 rows, 1.79/6.61 ms at 50,000, and 44.37/531.18 ms at one million. A selective million-row query instead took 0.61 ms through the index versus 17.88 ms scanning. Equal-row narrow/wide fixtures were reported as 12/1,010 pages and 0.02/1.30 ms.

Those are historical observations, not universal thresholds or measurements repeated here. The earlier data generator and complete benchmark environment are unavailable in this review. The exercise below therefore establishes a separate reproducible comparison and prints its runtime version. It does not claim to reproduce the earlier timings, page counts, or automatic planner choices.

When it breaks

A growing tenant can invalidate an old selectivity estimate while every query still returns correct rows. Statistics are one suspect, not proof of the cause: changed parameters, index coverage, cache state, or competing workload can also explain a slower plan. Check estimated versus observed work before forcing a permanent access path.

Fresh statistics do not guarantee a scan for every broad range predicate. SQLite builds differ in optional statistics support, and plan text can change between versions. A SCAN entry can even describe a covering-index scan rather than a table scan. Read the entire plan and compare equivalent query results instead of turning one keyword into a pass/fail rule.

The tenant index lacks payload, so its range path includes table lookups. All three fixture paths must return equal results; no winner is asserted. Historical timings remain unreplicated.

Check it quickly

Save this as scans.py and run python3 scans.py. It uses Python’s SQLite binding, creates its own temporary database, and never opens an existing application database. All three paths must return 2,880,000 for the broad predicate and 3,200 for the selective one. Seven alternating-order warm samples report medians without asserting which path wins.

from pathlib import Path
from tempfile import TemporaryDirectory
from statistics import median
from time import perf_counter
import sqlite3

with TemporaryDirectory(prefix="scan-crossover-") as directory:
 db = sqlite3.connect(Path(directory) / "fixture.db")
 try:
 db.executescript("""CREATE TABLE events(id INTEGER PRIMARY KEY, tenant INTEGER, payload TEXT);
 CREATE INDEX events_tenant ON events(tenant);""")
 db.executemany("INSERT INTO events VALUES(?,?,?)",
 ((i, i % 1000, "x" * 64) for i in range(50000)))
 db.commit()
 db.execute("ANALYZE")
 print("SQLite", sqlite3.sqlite_version)
 print("database pages", db.execute("PRAGMA page_count").fetchone()[0],
 "page size", db.execute("PRAGMA page_size").fetchone()[0])
 for predicate, expected in [("tenant < 900", 2880000), ("tenant = 7", 3200)]:
 base = "SELECT sum(length(payload)) FROM events {} WHERE " + predicate
 paths = {"chosen": "", "scan": "NOT INDEXED", "index": "INDEXED BY events_tenant"}
 queries = {name: base.format(clause) for name, clause in paths.items()}
 samples = {name: [] for name in queries}
 for name, query in queries.items():
 assert db.execute(query).fetchone()[0] == expected
 print(predicate, name, db.execute("EXPLAIN QUERY PLAN " + query).fetchall())
 for repeat in range(7):
 names = list(queries) if repeat % 2 == 0 else list(reversed(queries))
 for name in names:
 start = perf_counter()
 result = db.execute(queries[name]).fetchone()[0]
 samples[name].append((perf_counter() - start) * 1000)
 assert result == expected
 print("warm median ms", {k: round(median(v), 3) for k, v in samples.items()})
 old = db.execute("SELECT stat FROM sqlite_stat1 WHERE idx='events_tenant'").fetchone()[0]
 db.execute("UPDATE events SET tenant=5")
 db.commit()
 stale = db.execute("SELECT stat FROM sqlite_stat1 WHERE idx='events_tenant'").fetchone()[0]
 assert stale == old
 db.execute("ANALYZE")
 fresh = db.execute("SELECT stat FROM sqlite_stat1 WHERE idx='events_tenant'").fetchone()[0]
 assert fresh.split()[:2] == ["50000", "50000"] and fresh != old
 print("statistics before/stale/fresh", old, stale, fresh)
 try:
 db.execute("SELECT * FROM events INDEXED BY absent_index")
 except sqlite3.OperationalError as error:
 assert "no such index" in str(error), error
 else:
 raise AssertionError("missing required index was silently ignored")
 print("missing INDEXED BY target rejected")
 finally:
 db.close()

The payload expression requires data absent from the tenant index, making this a non-covering comparison. INDEXED BY requires the named index; it is not merely a suggestion. The missing-index negative control confirms that requirement. NOT INDEXED disables secondary indexes here; elsewhere it does not prohibit a rowid lookup. Keep these controls in diagnosis, rather than copying them into application queries without justification.

After the data changes to one tenant, sqlite_stat1 remains stale until ANALYZE. The refreshed entry reports 50,000 rows sharing that key. This tests statistics maintenance, not a guaranteed plan flip. In production, follow the SQLite version’s recommended PRAGMA optimize workflow and schedule expensive analysis appropriately.

The fixture warms data and measures one process without competing writers. It cannot establish cold-storage latency, production concurrency, or the best long-term index budget. Retain representative distributions and query projections in performance tests, and repeat them as the table grows; an acceptable scan today can become expensive without any correctness failure.

Sources

SQLite query planning, range traversal, table lookups, and covering indexes, checked 2026-10-15.
SQLite EXPLAIN QUERY PLAN, SCAN/SEARCH meaning and unstable presentation, checked 2026-10-15.
SQLite PRAGMA page_count, database-wide page count, checked 2026-10-15.
SQLite ANALYZE, statistics and recommended optimize workflow, checked 2026-10-15.
SQLite INDEXED BY, required indexes and NOT INDEXED scope, checked 2026-10-15.
PostgreSQL planner cost constants, relative costs and cache assumptions, checked 2026-10-15.
PostgreSQL using EXPLAIN, row estimates, bitmap scans, and index-only scans, checked 2026-10-15.

Composite indexes match multi-column queries

Wed, 15 Sep 2027 00:00:00 +0000

A composite B-tree index orders tuples by its first key, then by the next key within each first-key group. This structure can make a tenant-specific time range cheap to locate. The useful rule is about how much index space a predicate narrows, not a prohibition against mentioning later columns or writing WHERE conditions in a different textual order.

A local excerpt of tenant-first key order, followed by a possible later-key search strategy. Repeated searches are cost-dependent; the diagram is not a plan trace, and output ordering still depends on the chosen path.

Order the keys, not the SQL clauses

For an index on (tenant, happened), equality on tenant and a range on happened describe a compact portion of the tree. PostgreSQL can search that range rather than inspecting all tenants. Writing the time condition before the tenant condition in SQL does not change the logical conjunction or the index’s key order. The schema’s column order and the query’s presentation order are different things.

Leading equality conditions and the first non-equality key generally determine the range that must be scanned. Conditions on later keys can still be checked within the index, saving heap visits even when they do not further shrink a simple contiguous range. Reversing key columns changes these opportunities; it does not make an index universally unusable.

Later keys can still help

PostgreSQL 18 can use B-tree skip scans when repeated searches over missing leading-key values are cheaper than reading most of the index. With only a few tenants, a predicate on happened alone can search within each tenant group. With many distinct leading values, that strategy may be too expensive, and the planner can prefer a sequential scan or another index.

Even without a profitable skip scan, a composite index can sometimes be scanned for a condition involving a later key. The question is cost, coverage, and selectivity, not whether such SQL is legal or whether the index must be ignored. Do not use the shorthand “leftmost prefix” as an absolute claim that contradicts the target engine’s actual access methods.

Include ordering and the workload

Fixing the tenant key can make the following happened key useful for ordered results. Across several tenants, however, the index groups by tenant first; it does not provide one globally time-sorted stream. A bitmap scan also loses index ordering and may need a separate sort. Look at the complete plan instead of inferring final order from an index declaration.

Equality-first and range-next is a useful starting point for one query shape, but there is no universal “most selective column first” recipe. Shared prefixes, joins, sort direction, LIMIT, parameter distributions, and write costs all affect the choice. Compare actual recurring queries, then keep the smallest useful set of indexes. Redundant wide indexes consume cache and maintenance work even when nobody benefits from them.

When it breaks

A team may build (tenant, happened) for per-tenant queries, then add a global timeline and assume it is equally efficient. The timeline’s ordering and selectivity differ. Another regression comes from testing one tiny tenant while another owns most rows: the same parameterized SQL can have very different costs. Statistics and representative values matter more than a screenshot of one favorable plan.

Incorrect rules also cause unnecessary index churn. Adding one index per suffix because “the existing index cannot be used” can duplicate useful storage. Conversely, assuming skip scan always rescues a missing prefix can hide a costly path. Measure the chosen plan, actual rows, buffers, and sort work before changing the schema.

Check it quickly

Save this as composite.sql and run psql -X -v ON_ERROR_STOP=1 -f composite.sql on a disposable PostgreSQL 18 database. The temporary fixture contains four tenant values and twenty thousand ordered event numbers.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE events (tenant integer, happened integer, body text);
INSERT INTO events SELECT g % 4, g, repeat('x',40) FROM generate_series(1,20000) AS g;
CREATE INDEX events_tenant_time ON events (tenant, happened);
ANALYZE events;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT happened FROM events WHERE tenant=2 AND happened BETWEEN 100 AND 120
ORDER BY happened;
SELECT array_agg(happened ORDER BY happened) AS matching_times
FROM events WHERE happened BETWEEN 100 AND 120 AND tenant=2;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT tenant FROM events WHERE happened=12346;
DO $$ BEGIN
 IF (SELECT array_agg(happened ORDER BY happened) FROM events
 WHERE happened BETWEEN 100 AND 120 AND tenant=2)
 IS DISTINCT FROM ARRAY[102,106,110,114,118] THEN
 RAISE EXCEPTION 'wrong composite range';
 END IF;
 IF (SELECT tenant FROM events WHERE happened=12346) IS DISTINCT FROM 2 THEN
 RAISE EXCEPTION 'suffix predicate lost the row';
 END IF;
END $$;
ROLLBACK;

The tenant-range query returns exactly 102,106,110,114,118, including when the WHERE conditions are written in the opposite order. The retained PostgreSQL 18.6 report used a bitmap index scan followed by a sort. The suffix-only lookup returned tenant two through an index-only scan with repeated index searches, demonstrating a usable path without tenant equality.

Plan names, buffer counts, and search counts are observations for this dataset, not guaranteed outputs across configurations. The assertions verify result semantics, and rollback removes all objects. Repeat the measurement with more leading-key values to assess whether the suffix path remains worthwhile for a real workload.

Sources

PostgreSQL multicolumn indexes, leading constraints and skip scans, checked 2026-09-10.
PostgreSQL indexes and ORDER BY, ordered scans and sort choices, checked 2026-09-10.

Read replicas serve demand copies

Mon, 12 Jul 2027 00:00:00 +0000

A read replica can serve queries while another database handles writes. That can move reporting, search, or other eligible reads away from the writer, but routing a connection is only part of the design. The application must specify which committed changes a result needs to include. This article uses a PostgreSQL physical hot standby, whose normal recovery mode permits reads but does not accept ordinary writes.

When it breaks

With asynchronous replication, a successful write can be followed by a stale replica read. A user changes an account and refreshes the page, yet the previous value reappears. The replica may be healthy and continuously receiving WAL; the required commit still has to be replayed. A freshness check based on receipt rather than replay tests the wrong boundary.

Even replay progress is insufficient when the query uses an older transaction snapshot. A repeatable-read transaction intentionally preserves its view as recovery advances. Waiting until a replica has applied a write does not refresh that transaction. A correct read-after-write path needs both adequate replay progress and a snapshot taken after the progress condition was satisfied.

The freshness promise must also be limited. Including this user’s earlier write does not guarantee that no newer transaction exists elsewhere. A balance used to authorize another debit still needs a transactional check on the write path. Replica read capacity does not replace the concurrency control that protects the business operation.

Replay coverage does not refresh an existing snapshot. The 100/80 values belong to the held PostgreSQL fixture, not a capacity benchmark.

Check it quickly

Run this script as a non-root Linux account with PostgreSQL 18 server and client binaries in PATH. The original article reports PostgreSQL 18.6; this source review did not rerun the fixture. It builds private primary and standby clusters connected through Unix sockets, with no TCP listener. Unix sockets are still listeners and require environment permission. A named pipe keeps one reader session open so the snapshot boundary is controlled without installing a driver.

#!/bin/sh
set -eu
umask 077
lab=$(mktemp -d /tmp/xbcatm-read-replica.XXXXXX)
mkdir "$lab/socket"
primary=$lab/primary
standby=$lab/standby
reader_pid=''
reader_open=0
cleanup() {
 if test "$reader_open" = 1; then exec 3>&-; fi
 if test -n "$reader_pid"; then wait "$reader_pid" || :; fi
 for cluster in "$standby" "$primary"; do
 if test -f "$cluster/postmaster.pid"; then
 pg_ctl -D "$cluster" -m fast -w stop >/dev/null
 fi
 done
}
trap cleanup EXIT
initdb -D "$primary" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject >/dev/null
cat >> "$primary/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$lab/socket'
port = 56432
wal_level = replica
wal_keep_size = '64MB'
CONF
pg_ctl -D "$primary" -l "$lab/primary.log" -w -t 15 start >/dev/null
export PGHOST=$lab/socket PGPORT=56432 PGDATABASE=postgres PGUSER=$(id -un)
export PGCONNECT_TIMEOUT=2
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
CREATE TABLE account (id integer PRIMARY KEY,balance integer);
INSERT INTO account VALUES (1,100);
SQL
pg_basebackup -D "$standby" -R -X stream -c fast -w
cat >> "$standby/postgresql.conf" <<CONF
port = 56433
hot_standby = on
CONF
pg_ctl -D "$standby" -l "$lab/standby.log" -w -t 15 start >/dev/null
mkfifo "$lab/reader.sql"
PGPORT=56433 PGAPPNAME=replica_snapshot psql -X -v ON_ERROR_STOP=1 -qAt <"$lab/reader.sql" >"$lab/reader.log" 2>&1 &
reader_pid=$!
exec 3>"$lab/reader.sql"
reader_open=1
printf '%s\n' 'BEGIN ISOLATION LEVEL REPEATABLE READ;' 'SELECT balance FROM account WHERE id=1;' >&3
attempt=0
while test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT EXISTS (SELECT FROM pg_stat_activity WHERE application_name='replica_snapshot' AND state='idle in transaction' AND backend_xmin IS NOT NULL)")" != t; do
 attempt=$((attempt+1))
 test "$attempt" -lt 100 || exit 1
 sleep 0.1
done
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_wal_replay_pause()' >/dev/null
attempt=0
while test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_get_wal_replay_pause_state()='paused'")" != t; do
 attempt=$((attempt+1))
 test "$attempt" -lt 100 || exit 1
 sleep 0.1
done
psql -X -v ON_ERROR_STOP=1 -qAt -c 'UPDATE account SET balance=80 WHERE id=1'
target=$(psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_current_wal_lsn()')
printf 'primary_balance '
psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT balance FROM account WHERE id=1'
printf 'paused_replica_balance '
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT balance FROM account WHERE id=1'
for attempt in 1 2 3 4 5; do
 test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_last_wal_replay_lsn()>='$target'::pg_lsn")" = f
 sleep 0.1
done
echo fresh_read_gate_timed_out
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_wal_replay_resume()' >/dev/null
attempt=0
while test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_last_wal_replay_lsn()>='$target'::pg_lsn")" != t; do
 attempt=$((attempt+1))
 test "$attempt" -lt 150 || exit 1
 sleep 0.1
done
test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT balance FROM account WHERE id=1')" = 80
echo fresh_snapshot_after_replay_80
printf '%s\n' 'SELECT balance FROM account WHERE id=1;' 'COMMIT;' >&3
exec 3>&-
reader_open=0
wait "$reader_pid"
reader_pid=''
test "$(cat "$lab/reader.log")" = "$(printf '100\n100')"
echo old_repeatable_read_snapshot_still_100
printf 'lab_directory %s\n' "$lab"

The fixture begins with a balance of one hundred. On the standby, a repeatable-read transaction performs an initial query and remains open. The script confirms its backend is idle inside a transaction with a snapshot, then pauses actual WAL replay. The primary commits a change to eighty and captures a post-commit WAL position as the read’s progress target.

Expected output shows primary_balance 80 and paused_replica_balance 100. Five replay checks separated by short waits remain false, producing fresh_read_gate_timed_out. That result refuses to certify freshness while the required change is unapplied. The exercise does not silently treat the stale result as acceptable or pretend those polling intervals form a precise end-to-end latency guarantee.

Replay then resumes. After its LSN reaches the target, a new standby query must read eighty. The original repeatable-read session queries again and still reads one hundred. Its saved output must contain one hundred twice, proving the old snapshot survived the change in replay progress. Both facts can hold simultaneously: the server has caught up, and that particular transaction still sees older data.

The script commits the held read transaction, closes its pipe, stops both servers, and prints the retained log directory. Its assertions inspect database values and actual replay state. No production routing or application connection pool is changed.

Make the routing promise explicit

A progress token must refer to the same database history as the destination replica. An LSN is not a globally unique business version across unrelated clusters or diverged timelines. Bind any production gating design to the relevant topology and handle failover explicitly. The fixture uses one fixed primary and its direct standby, so that identity is established by construction.

If freshness cannot be established within the request budget, choose an explicit application response or an authorized read from the writer with capacity available. Do not let a deadline quietly weaken the meaning of the result. Acquire a fresh snapshot after the gate succeeds, and keep transactions short enough that snapshot retention does not defeat later checks.

Long standby queries can also conflict with recovery. PostgreSQL may cancel them to let replay proceed; feedback settings trade such cancellations against retention pressure on the primary. Monitor replay delay, query cancellations, and transaction age together. Measure the read workload actually moved off the writer, alongside the replication and operational costs of maintaining the copy.

Sources

PostgreSQL 18 hot standby, read-only recovery, transaction snapshots, and recovery conflicts, checked 2026-09-12.
PostgreSQL 18 administrative functions, replay position and pause/resume semantics, checked 2026-09-10.
PostgreSQL 18 replication configuration, standby delay and feedback tradeoffs, checked 2026-09-12.

Sequential identifiers reveal allocation patterns

Sat, 29 May 2027 00:00:00 +0000

A sequential identifier makes nearby values easy to guess. Seeing order number four thousand gives an observer a useful place to probe, and collecting identifiers over time may reveal allocation activity. It does not establish that four thousand orders exist, that all numbers represent successful purchases, or that the largest identifier committed last. Those conclusions require information that a sequence does not contain.

When it breaks

PostgreSQL sequences allocate values outside ordinary transaction rollback. An aborted insert can consume a number permanently. Deleted rows, reserved cache ranges, explicit sequence changes, and multiple writers further separate the counter from the table’s current population. Using the largest identifier as a customer count produces misleading business reports even without malicious activity.

Allocation order also differs from commit order. One transaction can obtain a smaller number and wait while another obtains a larger number and commits immediately. A consumer that treats increasing identifiers as a complete committed event stream can advance past the smaller row before that row becomes visible. A numeric key is not a change-data-capture position or a complete ordering of business events.

Random identifiers reduce easy enumeration but do not grant access control. A UUID can appear in logs, shared links, or another response. Anyone who learns it must still pass the resource’s authorization check. UUIDv4 contains random bits; UUIDv7 starts with a millisecond timestamp field and is designed for time ordering. Its prefix is not random, and its timestamp is not proof of database commit time.

Allocation, commit order, population count and authorization are different facts. The UUID stays unchanged when the ownership policy changes access. Source fixture only; not rerun.

Check it quickly

Use PostgreSQL 18 with two administrator psql sessions, A and B, against a disposable database. Version 18 is needed here for uuidv7() and UUIDv7 timestamp extraction; extraction functions themselves already exist in version 17. Ensure the schema and two role names are unused. Begin with autocommit enabled and the usual Read Committed isolation. In B, create the fixture:

CREATE SCHEMA article_ids;
CREATE ROLE article_id_alice NOLOGIN;
CREATE ROLE article_id_bob NOLOGIN;
SET search_path=article_ids,public;
CREATE TABLE objects (
 id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
 public_id uuid NOT NULL DEFAULT gen_random_uuid() UNIQUE,
 owner text NOT NULL,
 label text NOT NULL
);

In A, allocate the first identifier without committing:

SET search_path=article_ids,public;
BEGIN;
INSERT INTO objects (owner,label) VALUES ('article_id_alice','allocated first') RETURNING id;

In B, insert the second row and inspect visibility:

INSERT INTO objects (owner,label) VALUES ('article_id_bob','committed first') RETURNING id;
SELECT array_agg(id ORDER BY id) FROM objects;

B returns identifier two and sees only {2}. A has already obtained one, but its uncommitted row is invisible to B. Now commit A:

COMMIT;

Continue in B, making a deliberate rollback gap and inspecting both UUID versions:

SELECT array_agg(id ORDER BY id) FROM objects;
BEGIN;
INSERT INTO objects (owner,label) VALUES ('article_id_alice','rolled back') RETURNING id;
ROLLBACK;
INSERT INTO objects (owner,label) VALUES ('article_id_alice','after gap') RETURNING id;
SELECT max(id),count(*) FROM objects;
SELECT bool_and(uuid_extract_version(public_id)=4) FROM objects;
WITH sample AS MATERIALIZED (SELECT uuidv7() AS value)
SELECT uuid_extract_version(value),uuid_extract_timestamp(value) IS NOT NULL FROM sample;

The outputs include {1,2}, the abandoned identifier three, the next identifier four, and 4|3: maximum four, only three rows. The final checks report true for version four and 7|t for version seven with an extractable timestamp. Exact UUID values change on each run and are not expected constants.

Still in B, save Bob’s UUID in a client variable and query as Alice before installing any row policy:

SELECT public_id AS bob_id FROM objects WHERE owner='article_id_bob' \gset
GRANT USAGE ON SCHEMA article_ids TO article_id_alice,article_id_bob;
GRANT SELECT ON objects TO article_id_alice,article_id_bob;
SET ROLE article_id_alice;
SELECT count(*) FROM objects WHERE public_id=:'bob_id'::uuid;
RESET ROLE;

Alice sees one matching row despite knowing only Bob’s random public identifier. Enable ownership filtering and repeat the identical lookup:

ALTER TABLE objects ENABLE ROW LEVEL SECURITY;
CREATE POLICY owner_only ON objects FOR SELECT USING (owner=current_user);
SET ROLE article_id_alice;
SELECT count(*) FROM objects WHERE public_id=:'bob_id'::uuid;
SELECT array_agg(id ORDER BY id) FROM objects;
RESET ROLE;
SET ROLE article_id_bob;
SELECT count(*) FROM objects WHERE public_id=:'bob_id'::uuid;
RESET ROLE;

Now Alice receives zero matches and sees only identifiers one and four, while Bob receives one match for his own identifier. The changed result comes from the policy, not from changing UUID generation. These test roles are neither table owners nor superusers; privileged roles can bypass row security. Clean up in B after A has committed:

DROP SCHEMA article_ids CASCADE;
DROP ROLE article_id_alice;
DROP ROLE article_id_bob;

Choose identifiers and permissions separately

An internal sequence plus a public random identifier can support efficient joins and less predictable URLs. It also adds another unique index and another identifier to maintain. Evaluate that cost against the actual enumeration and disclosure requirements. A time-ordered UUID may improve insertion locality in some workloads, but exposes time information and does not automatically improve every database index.

Authorize every lookup using the authenticated principal and requested object, including alternative lookup routes. The demonstration maps principals to database roles for clarity; an application must establish its own trusted identity context rather than accept an owner name supplied by the requester. Measure totals with the appropriate business query, and use a documented event or replication position when completeness across concurrent commits matters.

Sources

PostgreSQL 18 CREATE SEQUENCE, gaps and sequence allocation semantics, checked 2026-09-10.
RFC 9562, UUIDv4 randomness and UUIDv7 timestamp layout, checked 2026-09-12.
PostgreSQL 18 UUID functions, generation and extraction used in the exercise, checked 2026-09-12.
PostgreSQL 18 row security, ownership policies and privileged bypass, checked 2026-09-12.
PostgreSQL 17 UUID functions, pre-existing extraction functions and the version-specific UUIDv7 boundary, checked 2026-10-15.

Foreign keys keep references honest

Tue, 25 May 2027 00:00:00 +0000

A foreign key makes a relationship enforceable during ordinary database writes. A task can reference an existing project, and the database can reject an unknown project identifier even when an import or a new application code path forgets to check it. The constraint defines referential integrity; it does not decide whether the selected project is the correct one for the business operation.

State the relationship precisely

A referencing key points to eligible unique columns in the parent table, commonly its primary key. Several child rows may reference the same parent, so a foreign key alone describes neither one-to-one cardinality nor a requirement that every parent have children. Add the separate uniqueness or other rules that the actual relationship requires instead of inferring them from the reference.

Nullability is an independent decision. A single-column foreign key normally permits SQL NULL, representing no reference. Add NOT NULL when every child must have a parent. For composite keys, PostgreSQL’s default MATCH SIMPLE allows a row to avoid matching when any referencing component is null; MATCH FULL instead requires all components to be null or all to match. A partially filled composite reference is therefore not automatically invalid under the default.

Scope belongs in the key too. If identifiers are only unique within a tenant, reference a suitable tenant-and-identifier key rather than assuming a matching identifier in another tenant is acceptable. Even a correctly scoped foreign key is not an authorization check: an existing project can still be the wrong project for the current user or workflow.

Parent deletion needs a policy

NO ACTION is PostgreSQL’s default delete action. With the normal immediate checking mode, deleting a referenced parent fails unless the resulting statement satisfies the constraint. A deferrable NO ACTION constraint can postpone the check; RESTRICT does not permit that deferral of the delete restriction. The two names should not be treated as unconditional synonyms.

CASCADE removes referencing child rows when the parent is deleted. It fits a component that has no independent lifecycle, but its reach can extend through several relationships. SET NULL retains the child while clearing its reference, which requires compatible nullability and other constraints. SET DEFAULT also must leave a valid reference; choosing a default does not waive referential integrity.

ON UPDATE separately describes what happens when referenced key values change. An update cascade copies the new key into referencing rows; it is not a request to overwrite unrelated child attributes. Choose actions according to ownership and retention requirements. Replacing a restrictive rule with cascade merely to silence an error can turn a useful integrity failure into unintended data deletion.

When it breaks

An application-side existence query can race with another transaction deleting the parent. The database constraint coordinates the actual write, so keep it as the authoritative relationship check rather than relying only on the earlier read. Privileged operations that disable enforcement, or migrations that have not validated old rows, are separate states and cannot be assumed to satisfy a fully enforced, validated constraint.

The write cost is also workload-dependent. Parent deletion can require finding many children, taking locks, and executing cascades. The referenced key must already have an eligible primary/unique constraint or suitable unique index; adding a foreign key does not create an index on the child reference columns. Review those access paths and the size of the deletion graph. A declared relationship does not guarantee join elimination or make every join inexpensive.

The first parent deletion fails atomically. Removing the blocking task allows CASCADE and SET NULL to apply their distinct lifecycles. Earlier source outcomes; no new database run.

Check it quickly

Save this as foreign-key-actions.sql and run psql -X -v ON_ERROR_STOP=1 -f foreign-key-actions.sql in a disposable PostgreSQL 18 database. The fixture uses temporary tables for tasks, project-owned files, and retained history, with different delete actions. Rollback removes all test objects and changes.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE projects (id integer PRIMARY KEY);
CREATE TEMP TABLE tasks (id integer PRIMARY KEY,project_id integer REFERENCES projects ON DELETE NO ACTION);
CREATE TEMP TABLE files (id integer PRIMARY KEY,project_id integer REFERENCES projects ON DELETE CASCADE);
CREATE TEMP TABLE audit_history (id integer PRIMARY KEY,project_id integer REFERENCES projects ON DELETE SET NULL);
INSERT INTO projects VALUES (1),(2);
INSERT INTO tasks VALUES (10,1),(11,NULL);
INSERT INTO files VALUES (20,1);
INSERT INTO audit_history VALUES (30,1);
DO $$ BEGIN
 INSERT INTO tasks VALUES (12,999);
 RAISE EXCEPTION 'dangling reference accepted';
EXCEPTION WHEN foreign_key_violation THEN RAISE NOTICE 'missing project rejected';
END $$;
DO $$ BEGIN
 DELETE FROM projects WHERE id=1;
 RAISE EXCEPTION 'referenced project deleted';
EXCEPTION WHEN foreign_key_violation THEN RAISE NOTICE 'NO ACTION blocked parent deletion';
END $$;
DO $$ BEGIN
 IF (SELECT count(*) FROM files) <> 1 OR
 (SELECT project_id FROM audit_history WHERE id=30) IS DISTINCT FROM 1 THEN
 RAISE EXCEPTION 'failed deletion left partial side effects';
 END IF;
END $$;
DELETE FROM tasks WHERE id=10;
DELETE FROM projects WHERE id=1;
SELECT * FROM tasks;
SELECT * FROM audit_history;
SELECT count(*) AS remaining_files FROM files;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM projects) IS DISTINCT FROM ARRAY[2] THEN
 RAISE EXCEPTION 'wrong surviving project';
 END IF;
 IF (SELECT count(*) FROM files) <> 0 OR
 (SELECT count(*) FROM audit_history WHERE id=30 AND project_id IS NULL) <> 1 OR
 (SELECT count(*) FROM tasks WHERE id=11 AND project_id IS NULL) <> 1 THEN
 RAISE EXCEPTION 'wrong delete action or null handling';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 rejected the task pointing to project 999, while accepting task 11 with a null reference. The first deletion of project one failed because task 10 still referenced it. That failed statement did not partially remove the other dependent data.

After explicitly removing task 10, the parent deletion succeeded. Its file disappeared through CASCADE, history row 30 remained with a null project, and the unrelated null-reference task remained. Project two survived without any children. Assertions verify these distinct outcomes, showing that “references stay valid” and “children must be deleted” are different contracts. Use the same lifecycle examples when reviewing schema changes with application owners.

Sources

PostgreSQL constraints, referential integrity, nulls, actions, and indexing, checked 2026-09-10.
PostgreSQL CREATE TABLE, foreign-key matching and deferrability syntax, checked 2026-09-10.

EXPLAIN plans reveal the route of a query

Fri, 21 May 2027 00:00:00 +0000

EXPLAIN shows the execution plan selected for a statement. It helps locate expensive work and mistaken estimates, but a slow request can also involve locks, network transfer, or application behavior outside the plan. Use plan evidence to test a specific hypothesis rather than assuming that every performance problem requires a different scan or another index.

Read the tree without double-counting

In PostgreSQL, leaves commonly scan relations while upper nodes join, sort, aggregate, or modify rows. Upper nodes do real work too. Their costs include child costs, so adding all displayed costs double-counts work. The rows estimate is the number a node emits, not necessarily how many rows it examines. Filters can explain a large difference between input and output without proving a planning mistake.

EXPLAIN alone reports estimates. EXPLAIN ANALYZE actually executes the statement and adds observations. Actual rows and times are averages per execution when loops exceeds one; interpret them together with loops. Compare estimates with corresponding observed output, not with unrelated table size. Buffer counts include child activity and are not counts of unique pages, so they also require careful interpretation.

Statistics influence a choice

Row counts, distributions, correlations, query structure, and cost settings influence planning. ANALYZE refreshes statistics, but a large estimate error does not uniquely diagnose stale statistics. Sampling, correlated predicates, missing extended statistics, and parameter selectivity can also matter. Refresh the relevant statistics when appropriate, then check whether the mismatch actually improved.

The planner does not simply watch the latest observed cache-hit ratio and automatically select a new route. Settings such as effective_cache_size express assumptions about caching; actual warm-cache conditions can change execution time even with the same plan. Record data size, parameters, relevant settings, and measurement conditions when comparing results. A sequential scan can be the best choice for a large fraction of a table.

When it breaks

Prepared statements need version-specific interpretation. PostgreSQL’s automatic choice can compare custom plans with a generic plan; it does not simply cache the first parameter’s plan for all later values. Use EXPLAIN EXECUTE with representative values and inspect whether parameters appear as constants or placeholders. Different database products have different caching rules, so avoid importing one product’s explanation into another.

Functions, casts, and collations can prevent a particular index from matching the predicate, but do not universally force a full scan. An expression index may support lower(column), and some cross-type operators remain indexable. PostgreSQL can reject an unsupported varchar-to-integer comparison instead of quietly converting it. Inspect the actual operator, types, and index condition before deciding on a rewrite.

EXPLAIN ANALYZE is not harmless inspection. Writes, triggers, locks, and called functions can have effects. A transaction rollback does not undo every possible effect, such as consumed sequence values or external actions. Use controlled data and an appropriate test environment; even a SELECT may invoke side-effecting functions. Budget the load and avoid treating a diagnostic command as automatically safe for production.

Confirm an improvement, not merely a different shape

Capture comparable before-and-after results. A changed plan can be slower, and the same plan shape can become faster through less work, different data, or improved execution conditions. Inspect elapsed time, output correctness, row estimates, buffers, loops, and spills together. Cost units are not milliseconds, and a low estimated cost is not an operational latency guarantee.

All bars use the same relative-cost scale. The reversal comes from a toy formula and supplied selectivity, not measured database execution. Model not rerun.

Check it quickly

Save the following as plan_route_check.py and run python3 plan_route_check.py. It uses only the standard library to model two candidate paths under a deliberately wrong selectivity estimate. It neither invokes PostgreSQL nor reproduces its optimizer. The fixture’s stale input is known by construction; the warning asks for investigation rather than diagnosing every real estimate error as stale statistics.

#!/usr/bin/env python3
"""Toy planner model: estimated cost picks a route, actual rows judge it.

Stdlib only. It simulates stale statistics: the planner believes 500 rows
match, while 600,000 really do, and checks that the estimate-error alarm fires.
"""

from __future__ import annotations


def seq_scan_cost(pages: float, rows: float) -> float:
 return pages * 1.0 + rows * 0.01


def index_scan_cost(selectivity_est: float, pages: float, rows: float) -> float:
 heap_pages = selectivity_est * pages
 return 4.0 + heap_pages * 4.0 + selectivity_est * rows * 0.01


def main() -> None:
 pages, rows = 8_000.0, 1_000_000.0
 stale_selectivity = 0.0005 # planner belief: 500 rows
 true_selectivity = 0.60 # reality: 600,000 rows
 est_rows = stale_selectivity * rows
 actual_rows = true_selectivity * rows

 seq = seq_scan_cost(pages, rows)
 idx_est = index_scan_cost(stale_selectivity, pages, rows)
 idx_true = index_scan_cost(true_selectivity, pages, rows)
 chosen = "index scan" if idx_est < seq else "seq scan"
 print(f"estimated rows: {est_rows:.0f}; actual rows: {actual_rows:.0f}")
 print(f"estimated cost: seq={seq:.1f} index={idx_est:.1f} -> planner picks {chosen}")
 print(f"true cost at actual selectivity: seq={seq:.1f} index={idx_true:.1f}")

 error = abs(actual_rows - est_rows) / actual_rows
 print(f"estimate error: {error:.1%}")
 warned = False
 if error > 0.50:
 warned = True
 print("WARNING: row estimate off by >50% - inspect statistics, predicates, and parameter selectivity.")
 if chosen == "index scan" and idx_true > seq:
 warned = True
 print("WARNING: chosen route loses at actual selectivity - check EXPLAIN ANALYZE.")
 assert warned, "expected a stale-estimate warning for this fixture"
 assert est_rows < actual_rows / 10, "fixture must keep the estimate an order of magnitude low"
 print("PASS: planner picked the stale route and the estimate-error alarm fired")


if __name__ == "__main__":
 main()

The model chooses an index scan at estimated cost 25 versus 18,000 for the sequential path. At the supplied actual selectivity, its index cost becomes 25,204. It prints an estimate error of 99.9% and a PASS line after the warnings. Both costs are outputs of these simple formulas, not measured database execution times.

This confirms the arithmetic and the deliberately constructed counterexample. It does not validate a particular PostgreSQL plan, justify adding an index, or prove that ANALYZE repairs a production query. Repeat the investigation with actual plans and controlled execution evidence before changing a real workload.

Sources

PostgreSQL: EXPLAIN, generating the plan, checked 2026-10-15.
PostgreSQL: using EXPLAIN, interpreting the output, checked 2026-10-15.
PostgreSQL: planner/optimizer, how routes are chosen, checked 2026-10-15.
MySQL: EXPLAIN statement, plan format and output columns, checked 2026-10-15.
PostgreSQL PREPARE, generic and custom plan selection, checked 2026-10-15.
PostgreSQL expression indexes, indexed expressions including lower(column), checked 2026-10-15.

Query parameters preserve the boundary between values and SQL

Mon, 17 May 2027 00:00:00 +0000

A parameterized query sends the database instruction and its values through distinct API arguments. The database interprets the instruction as SQL and uses each parameter where a value belongs. This boundary works in embedded engines such as SQLite as well as client-server databases; it does not depend on a network exchange or on an application explicitly managing named server statements.

When it breaks

String concatenation removes that boundary. A quote supplied as part of a name can end the intended literal, and following text can introduce a condition or comment. Blocking semicolons does not fix the problem: an attacker may change the meaning of one SELECT without adding a second statement. Similarly, an API that refuses multiple statements still needs proper parameter binding.

Binding prevents a value from becoming SQL grammar, but the chosen operator still interprets the value according to its documented rules. With LIKE, percent matches a sequence of characters and underscore matches one character. A bound search string can therefore match more records than a literal-text search intended. That is a pattern-semantics issue, not evidence that parameter binding failed.

Other mistakes produce incorrect results without injection. A comma-separated string bound to one IN placeholder remains one scalar value; it is not expanded into a list. A bound column name remains a text value rather than a column reference. Type checks, allowed structural choices, and application permissions remain necessary alongside binding.

Binding protects SQL grammar while LIKE still interprets wildcards. Lists, identifiers and page-size policy keep separate contracts. Local SQLite assertions, not a production guarantee.

Check it quickly

Run this complete Python script with the standard sqlite3 module. It creates only an in-memory fixture and closes it after the assertions. The recorded environment was Python 3.12.14 with SQLite 3.53.1. The deliberately vulnerable query is included solely to compare behavior against three invented users:

import sqlite3

with sqlite3.connect(':memory:') as db:
 db.execute('CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT NOT NULL)')
 db.executemany('INSERT INTO users VALUES (?, ?)', [(1, 'Alice'), (2, 'A_1'), (3, 'Ab1')])
 payload = "' OR 1=1 --"
 # Deliberately vulnerable comparison, only against this in-memory fixture.
 unsafe = "SELECT id FROM users WHERE name='" + payload + "' ORDER BY id"
 assert sorted(db.execute(unsafe).fetchall()) == [(1,), (2,), (3,)]
 assert db.execute('SELECT id FROM users WHERE name=?', (payload,)).fetchall() == []
 print('concatenated: 3 rows; bound: 0 rows')
 for value in ("O'Reilly", r'a\b', 'x; DROP TABLE users; --'):
 assert db.execute('SELECT ?', (value,)).fetchone() == (value,)
 assert db.execute('SELECT count(*) FROM users').fetchone() == (3,)
 print('quote, backslash, semicolon: literal round trips')
 assert db.execute('SELECT id FROM users WHERE name LIKE ? ORDER BY id', ('A_1',)).fetchall() == [(2,), (3,)]
 literal = 'A_1'
 escaped = literal.replace('!', '!!').replace('%', '!%').replace('_', '!_')
 assert db.execute("SELECT id FROM users WHERE name LIKE ? ESCAPE '!' ORDER BY id", (escaped,)).fetchall() == [(2,)]
 assert db.execute('SELECT id FROM users WHERE name=?', (literal,)).fetchall() == [(2,)]
 print('bound LIKE: 2 rows; escaped LIKE and equality: 1 row')
 assert db.execute('SELECT id FROM users WHERE id IN (?)', ('1,3',)).fetchall() == []
 assert db.execute('SELECT id FROM users WHERE id IN (?,?) ORDER BY id', (1, 3)).fetchall() == [(1,), (3,)]
 assert db.execute('SELECT ? FROM users LIMIT 1', ('name',)).fetchone() == ('name',)
 assert db.execute('SELECT id FROM users ORDER BY id LIMIT ? OFFSET ?', (1, 1)).fetchall() == [(2,)]
 assert db.execute('SELECT id FROM users ORDER BY id LIMIT ?', (-1,)).fetchall() == [(1,), (2,), (3,)]
 assert db.execute('SELECT id FROM users ORDER BY id LIMIT ?', (0,)).fetchall() == []
 print('IN needs separate values; bound column is a literal; LIMIT/OFFSET bind')
 print('negative LIMIT removes the cap; zero LIMIT returns no rows')
db.close()

The concatenated query returns all three identifiers, while the bound version returns no matches for the same input. The injected comment also removes the appended ordering clause, so the assertion sorts the vulnerable result before comparing identifiers. The test does not depend on an unordered query coincidentally returning insertion order.

The next assertions round-trip a quote, a backslash, and a semicolon-containing string exactly. The three-row count remains unchanged. These are actual calls to the SQLite driver, not a string-escaping simulation or a check that a query happens to contain a question mark. Save the SQL text and parameter assignment together when reviewing a real call site.

The bound LIKE pattern A_1 matches both A_1 and Ab1. Escaping the underscore using an explicit exclamation-mark escape reduces the result to the literal name. Escaping the escape character first is necessary when arbitrary search text can contain it. For exact matching, equality is simpler and also returns only the literal name here.

A single bound value containing 1,3 returns no identifiers. Two placeholders with two integers return identifiers one and three. The next SELECT returns the literal text name, demonstrating why a value placeholder cannot choose a projected column. Finally, bound numeric limit and offset values return identifier two. No numeric string interpolation is required for pagination. Binding negative one as SQLite LIMIT removes the cap; zero returns no rows. These additional assertions show why safe SQL grammar still needs an application page-size policy.

Review the complete data path

Choose placeholders according to the actual driver. Python’s SQLite interface accepts question marks and named parameters; Psycopg uses different binding notation. Pass the parameter collection separately rather than applying language-level formatting first. A value loaded from your own database is still a value when reused in another query; storing it once does not make later interpolation safe.

For variable-length lists, construct only the required placeholder structure and bind each element, using the driver’s supported collection facilities where available. Define empty-list behavior and respect parameter-count limits. Dynamic identifiers and ordering keywords need an allowed set of query structures; filtering punctuation out of arbitrary SQL text is not a reliable substitute.

Set appropriate page-size bounds and enforce object authorization even when every value is bound. Parameterization protects the SQL grammar boundary, not the entire business operation. Tests should check ordinary results as well as adversarial strings and operator-specific edge cases, so a secure rewrite does not silently broaden searches, change ordering, or expose another user’s rows.

Sources

SQLite SELECT, negative and zero LIMIT behavior, checked 2026-10-15.
Python sqlite3, driver placeholders and binding API, checked 2026-09-12.
SQLite expressions, parameters, LIKE escaping, and IN semantics, checked 2026-09-12.
OWASP SQL injection prevention, parameterized values and allowed structural choices, checked 2026-09-12.

Transactions make atomic commits

Wed, 12 May 2027 00:00:00 +0000

A transaction lets related database changes commit together or be discarded together. Debiting an account, crediting another, and recording a ledger entry belong to the same decision when the operation requires all three. PostgreSQL provides that boundary through BEGIN, COMMIT, and ROLLBACK. The application must still place the correct statements inside it: atomic execution does not discover a forgotten credit or invent a missing business check.

The failed second transfer must return to the committed balances and ledger count. A missing COMMIT reply is a separate uncertainty requiring outcome reconciliation. This conceptual diagram adds no runtime evidence.

Decide what belongs to one unit

Without an explicit transaction block, ordinary psql autocommit runs each statement in its own transaction. A successful debit can therefore commit before a later ledger insert fails. Explicitly grouping the writes changes the failure boundary. Other transactions do not see the group’s uncommitted intermediate table changes, and rollback discards its transactional changes rather than relying on compensating updates issued afterward.

Atomicity is distinct from isolation and durability. A committed unit can still embody an incorrect concurrent decision if its isolation or locking protocol is insufficient. The persistence guarantees of an acknowledged commit also depend on the database’s durability configuration. None of those questions is answered solely by finding a BEGIN in the code.

Every PostgreSQL statement already executes in a transaction, including reads. An explicit read transaction can have a purpose, such as selecting a repeatable snapshot or using transaction-local settings. The relevant decision is the required boundary, not a rule that all read transactions are useless overhead.

When it breaks

In PostgreSQL, an unhandled SQL error normally leaves an explicit transaction block aborted. Further ordinary statements fail until the client rolls back, or rolls back to an applicable earlier savepoint. Disabling a client’s stop-on-error behavior does not restore the server transaction. Returning that connection to a pool without cleaning up can make the next request inherit the failure state.

A server-reported rejection and a lost commit acknowledgement are different evidence. If a connection fails while the client awaits COMMIT, the client may not know whether the database committed before communication was lost. Blindly repeating a payment can duplicate it. A durable operation identifier and outcome reconciliation address that ambiguity; ordinary rollback semantics do not prove what happened on an unreachable server.

Check it quickly

Run the following in a fresh psql -X -v ON_ERROR_STOP=1 session on PostgreSQL 18 using a disposable database. Temporary tables keep the account identifiers local to the session. Error stopping is disabled only for two deliberate failures so their SQLSTATE values can be observed, then enabled again.

\set ON_ERROR_STOP on
CREATE TEMP TABLE accounts (id integer PRIMARY KEY,balance integer CHECK(balance>=0));
CREATE TEMP TABLE ledger (id integer PRIMARY KEY,amount integer CHECK(amount>0));
INSERT INTO accounts VALUES (1,100),(2,100);
BEGIN;
UPDATE accounts SET balance=balance-20 WHERE id=1;
UPDATE accounts SET balance=balance+20 WHERE id=2;
INSERT INTO ledger VALUES (1,20);
COMMIT;
SELECT id,balance FROM accounts ORDER BY id;
BEGIN;
UPDATE accounts SET balance=balance-30 WHERE id=1;
\set ON_ERROR_STOP off
INSERT INTO ledger VALUES (1,30);
\echo duplicate_error :SQLSTATE
SELECT balance FROM accounts WHERE id=1;
\echo aborted_error :SQLSTATE
\set ON_ERROR_STOP on
ROLLBACK;
SELECT id,balance FROM accounts ORDER BY id;
SELECT count(*) AS ledger_rows FROM ledger;
DO $$ BEGIN
 IF (SELECT array_agg(balance ORDER BY id) FROM accounts) IS DISTINCT FROM ARRAY[80,120]
 OR (SELECT count(*) FROM ledger) <> 1
 OR (SELECT amount FROM ledger WHERE id=1) IS DISTINCT FROM 20 THEN
 RAISE EXCEPTION 'partial transfer survived';
 END IF;
END $$;
DROP TABLE ledger,accounts;

The first transaction commits balances of 80 and 120 and one ledger row. The second transaction debits another thirty, then attempts to reuse ledger identifier one. The retained PostgreSQL 18.6 report records 23505 for that uniqueness violation; this review did not rerun the client workflow. The following ordinary read reports 25P02, showing that the transaction is aborted even though the client continues sending commands.

After rollback, the balances are still 80 and 120, with one ledger row. The second debit did not survive. The assertion checks the relevant balances and ledger state; the two printed SQLSTATE values still require inspection. Seeing an error alone does not prove rollback worked. The final drop removes the temporary fixture explicitly; closing the session would also remove its temporary tables.

This is a controlled atomicity experiment, not a complete money-transfer service. A real operation must verify that intended accounts exist, affected-row counts match expectations, amounts obey the business contract, and concurrent decisions are protected. Otherwise a transaction can atomically commit the wrong set of changes.

Keep recovery inside the contract

Savepoints allow selected work to be undone while preserving earlier transactional changes. They are useful when that partial outcome is intentional; swallowing a required ledger failure and committing the debit would violate this operation’s contract. Exceptions caught inside database procedural blocks can also establish a smaller rollback scope, so examine the actual handling path.

Sequence advancement and external actions such as sending mail are not undone by rolling back table writes. Keep irreversible external effects out of an attempt that may be retried, or coordinate them through a durable protocol. Short, explicit transactions make the commit decision easier to reason about, while checked results and a clear recovery path establish that the decision still means what the application intends.

Sources

PostgreSQL 18: Transactions, atomic groups and rollback boundaries, checked 2026-09-10.
PostgreSQL 18: Transaction isolation, visibility and nontransactional sequence behavior, checked 2026-09-10.
PostgreSQL 18: SAVEPOINT, intentional partial rollback within a transaction, checked 2026-09-10.

Connection pools amortize slow setup

Thu, 28 Jan 2027 00:00:00 +0000

A connection pool amortizes setup across several uses of a physical connection. It does not make setup disappear, and creating a pool does not necessarily open anything. Distinguish the application’s short-lived lease from the underlying database connection: closing the former normally returns the latter for reuse.

Reuse is a lifecycle, not a promise of free work

A remote database connection may require TCP, TLS, authentication, and session initialization. Reusing it avoids repeating those steps on every request, while SQL execution still costs time. Expiry, disconnection, invalidation, or process replacement can require setup again. Keep an SQLAlchemy Engine long-lived within its owning process; creating a fresh Engine for every request defeats that reuse.

SQLAlchemy pools grow lazily. A QueuePool configured with pool_size=2 starts empty, not with two warm connections. Two sequential leases can reuse one physical connection. Concurrent demand can grow this example to two retained connections plus one overflow connection. With finite max_overflow=1, three simultaneous leases exhaust its capacity; returning them leaves at most two idle connections.

These meanings belong to this pool implementation, not every library. In QueuePool, pool_size=0 removes the size limit rather than disabling pooling, and max_overflow=-1 permits unlimited overflow. Use explicit finite values when the pool is meant to enforce a connection budget.

When it breaks

A pool can be exhausted by legitimate concurrent work, slow transactions, or leases that are never returned. Pool acquisition time is therefore distinct from query time. pool_timeout bounds waiting for an available slot, not physical connection establishment or the SQL statement itself; its small floating-point value is not a hard real-time deadline.

Returning a connection also does not erase every trace of its previous borrower. SQLAlchemy’s normal reset removes transactional state through rollback, coordinated with Connection.close(). Committed temporary tables and other session state may survive. Do not infer tenant isolation from a returned lease; explicitly define any session setup and cleanup required by the backend and application.

A disconnected connection presents a different boundary. Explicit invalidation discards its underlying DBAPI connection. Later use obtains a replacement from the pool, possibly another idle connection rather than a newly opened one. If a transaction was active when invalidated, it must be ended before reconnecting. This is not transparent recovery of a transaction that may already have had effects.

The SQLite/SQLAlchemy exercise checks lifecycle and finite capacity using library events. It does not measure network setup savings or throughput.

Check it quickly

Use an empty directory on macOS or Linux. Run python3 -m venv .venv, then .venv/bin/python -m pip install SQLAlchemy==2.0.52. Save the following as pool_probe.py and run .venv/bin/python pool_probe.py. This was checked with Python 3.14.4 and SQLite 3.53.0. It creates and removes its own temporary database and does not contact a database server.

from pathlib import Path
import sqlite3
from tempfile import TemporaryDirectory

import sqlalchemy
from sqlalchemy import create_engine, event
from sqlalchemy.exc import TimeoutError as PoolTimeout
from sqlalchemy.pool import QueuePool


assert sqlalchemy.__version__ == "2.0.52"
with TemporaryDirectory(prefix="pool-reuse-") as directory:
 engine = create_engine(
 "sqlite:///" + str(Path(directory) / "probe.db"),
 poolclass=QueuePool, pool_size=2, max_overflow=1, pool_timeout=0.05,
 )
 events = {"opened": 0, "closed": 0}

 @event.listens_for(engine, "connect")
 def opened(dbapi_connection, entry):
 events["opened"] += 1
 entry.info["serial"] = events["opened"]

 @event.listens_for(engine, "close")
 def closed(dbapi_connection, entry):
 events["closed"] += 1

 try:
 assert engine.pool.checkedin() == 0 and events["opened"] == 0
 with engine.begin() as connection:
 serial = connection.info["serial"]
 connection.exec_driver_sql("CREATE TEMP TABLE marker(value INTEGER)")
 connection.exec_driver_sql("INSERT INTO marker VALUES (42)")
 with engine.connect() as connection:
 assert connection.info["serial"] == serial
 assert connection.exec_driver_sql("SELECT value FROM marker").scalar_one() == 42
 assert events["opened"] == 1 and engine.pool.checkedout() == 0
 print("lazy start: 0 physical connections; two sequential leases reuse 1; committed temp state persists")

 held = []
 try:
 for _ in range(3):
 held.append(engine.connect())
 assert len({connection.info["serial"] for connection in held}) == 3
 assert engine.pool.checkedout() == 3 and events["opened"] == 3
 try:
 unexpected = engine.connect()
 except PoolTimeout:
 pass
 else:
 unexpected.close()
 raise AssertionError("the fourth concurrent lease must time out")
 assert events["opened"] == 3
 finally:
 for connection in held:
 connection.close()
 assert engine.pool.checkedout() == 0 and engine.pool.checkedin() == 2
 assert events["closed"] == 1
 print("2 persistent slots + 1 overflow allow 3 leases; the fourth times out; only 2 remain idle")

 with engine.connect() as connection, engine.connect() as occupied:
 previous = connection.info["serial"]
 occupied_serial = occupied.info["serial"]
 connection.invalidate() # No transaction has started on this lease.
 assert connection.exec_driver_sql("SELECT 1").scalar_one() == 1
 assert connection.info["serial"] != previous
 assert occupied.info["serial"] == occupied_serial
 assert connection.exec_driver_sql(
 "SELECT count(*) FROM sqlite_temp_master WHERE name = 'marker'"
 ).scalar_one() == 0
 assert events["opened"] == 4
 print("with the other slot occupied, invalidation forces a new physical connection without the old temp table")
 finally:
 engine.dispose()
 assert events == {"opened": 4, "closed": 4}
print("all 4 connect events have matching close events; SQLite", sqlite3.sqlite_version)

The first output proves lazy creation and reuse using connect events and a serial stored with the DBAPI connection. The committed marker remains visible on the next lease: transaction reset is not a complete session reset. Next, three retained leases reach the configured cap; the fourth raises PoolTimeout. Returning all three leaves two idle connections and one close event.

For the invalidation check, the other idle connection is deliberately occupied. That makes the next SQL require a newly created physical connection, whose temporary schema lacks the old marker. Without that controlled occupancy, invalidation can simply borrow the other idle connection. Final disposal follows return of every lease, and four connect events match four close events. These are observed library events, not a guarantee about failed closes on an arbitrary backend.

This experiment demonstrates lifecycle and capacity, not TCP/TLS savings or throughput. For performance decisions, measure setup frequency, acquisition wait, lease duration, and query latency on the actual backend. A larger pool is justified only within its shared connection budget; it cannot by itself repair leaked leases or slow queries.

Sources

SQLAlchemy connection pooling, lazy creation, QueuePool limits and reset-on-return, checked 2026-09-09.
SQLAlchemy Engine and Connection, Engine lifetime, connection information and invalidation, checked 2026-09-09.
SQLAlchemy pool events, connect and close event semantics, checked 2026-09-09.
Python sqlite3, the local database runtime used in the exercise, checked 2026-09-09.

Migrations are versioned schema changes

Sun, 24 Jan 2027 00:00:00 +0000

The schema of a database is not a one-time design; it evolves with the software that uses it. Instead of editing the schema ad hoc, migrations apply a named, ordered set of changes that any environment can replay.

Schema as code

Treat the schema as source code: each migration is one small, named, ordered step that moves the database from one state to the next, and the sequence itself lives in the repository. A fresh environment replays the steps from an empty database to the current version, which is exactly how tests, CI, and new servers all converge on the same shape without ad-hoc ALTER TABLE edits on a live box. The schema stops being an undocumented fact of a single instance and becomes a reviewable artifact with history, blame, and a pull request.

The order is the contract

With the same starting schema, compatible engine, required extensions, and deterministic steps, ordered replay can reproduce the intended schema. Order alone does not neutralize environmental differences or data-dependent changes. That predictability is the contract: it is what lets a test runner build a disposable database, a deploy spin up a staging clone, and an operator upgrade production all with the same script. The corollary is inviolable: never rewrite or reorder a migration that anyone has already applied, because the runner uses its recorded identity to decide what has run; that identity need not include a content checksum. Editing history can create divergent environments that require inspection and an explicit forward repair.

The danger is the irreversible

A committed migration that drops a column or table destroys stored data; recreating its structure alone cannot recover the values. Give it a plan: back up before it applies, split the drop into its own later migration so you can revert the schema change first, and only run it toward a state you can live with. The real discipline is ‘forward only, with a way back’ — never a migration that silently locks you out. Always practice a destructive migration against a staging schema before it touches production, so the irreversible step is boring by the time it runs for real.

When it breaks

An irreversible migration is the most expensive way this breaks. On PostgreSQL, dropping a column avoids a table rewrite but still needs an ACCESS EXCLUSIVE lock. Many type changes and volatile defaults require rewrites; qualifying binary-compatible changes can avoid them. Avoiding a rewrite does not remove the lock or its waiting time. A large rewrite may wait or run for a long time. PostgreSQL can roll back transactional DDL on failure; partially applied migrations instead depend on earlier commits, nontransactional steps, and the runner’s boundaries. A down migration does not undo data loss either: dropping the email column and re-adding it returns an empty column, not the deleted addresses. Split destructive steps into their own later migration, back up first, and prefer expand-then-contract — add, backfill, remove — so deploys stay boring.

The other two classic failures are concurrent runners and edited history. If two deploy replicas both execute the migrator with no lock, they can apply the same version twice or interleave versions. Production runners therefore use a single migrator or a lock such as a PostgreSQL advisory lock, and golang-migrate marks the database dirty after a failed run. Rewriting a migration that is already applied — or merging two branches that each added version 7 — forks history: environments can diverge. Check what the runner actually records: golang-migrate’s PostgreSQL ledger tracks version and dirty state, not a checksum for every file. Reconcile actual state before a forward repair. Tools such as strong_migrations can flag risky operations during migration execution; run those checks in development before deployment.

The version/name ledger makes the local chain repeatable, but cannot detect arbitrary code edits or restore deleted values. Historical execution passed only for the in-memory model; real locks, concurrent runners and backup restores were not tested. No example was rerun for this drawing.

Check it quickly

Save the following as migration_chain_check.py and run python3 migration_chain_check.py with Python 3.10 or newer. It uses only the standard library and models three migrations over an in-memory schema and an applied ledger, so no database is needed. Its identity records only version and name, not executable content; it detects renaming, not arbitrary code edits.

"""Simulate a versioned migration chain with an applied ledger (stdlib only)."""

# Each migration is (version, name, up, down). down=None means irreversible.
# The `applied` dict emulates a schema_migrations table: version -> a name-based identity (not a content digest).

def identity(version, name):
 return "%d:%s" % (version, name)


def make_migrations():
 db = {"users": []}

 def up1():
 db["users"] = []

 def down1():
 del db["users"]

 def up2():
 for user in db["users"]:
 user["email"] = None
 db["has_email_column"] = True

 def down2():
 # Irreversible in effect: dropping the column destroys its data.
 for user in db["users"]:
 del user["email"]
 del db["has_email_column"]

 def up3():
 db["index_on_email"] = True

 def down3():
 del db["index_on_email"]

 migrations = [
 (1, "create_users", up1, down1),
 (2, "add_email_column", up2, down2),
 (3, "add_email_index", up3, down3),
 ]
 return db, migrations


def apply_single(db, applied, migrations, version):
 by_version = {m[0]: m for m in migrations}
 assert version in by_version, "unknown migration version: %r" % (version,)
 wanted = by_version[version]
 if version in applied:
 if applied[version] != identity(version, wanted[1]):
 raise RuntimeError("version drift: v%d was edited after apply" % version)
 return "skipped" # idempotent re-run
 missing = [m[0] for m in migrations if m[0] < version and m[0] not in applied]
 if missing:
 raise RuntimeError("out of order: v%d needs missing %s first" % (version, missing))
 wanted[2]()
 applied[version] = identity(version, wanted[1])
 return "applied"


def migrate_up(db, applied, migrations):
 results = []
 for m in migrations:
 results.append((m[0], apply_single(db, applied, migrations, m[0])))
 return results


def migrate_down(db, applied, migrations, target):
 by_version = {m[0]: m for m in migrations}
 while applied and max(applied) > target:
 version = max(applied)
 migration = by_version[version]
 assert migration[3] is not None, "v%d is irreversible" % version
 migration[3]()
 del applied[version]
 return sorted(applied)


# 1. Fresh database replays the whole chain in order.
db, migrations = make_migrations()
applied = {}
assert migrate_up(db, applied, migrations) == [(1, "applied"), (2, "applied"), (3, "applied")]
assert sorted(applied) == [1, 2, 3] and db["has_email_column"] is True
print("fresh replay applies v1..v3 in order:", sorted(applied))

# 2. Re-running is idempotent: the applied ledger makes it a no-op.
assert migrate_up(db, applied, migrations) == [(1, "skipped"), (2, "skipped"), (3, "skipped")]
assert sorted(applied) == [1, 2, 3]
print("second run is a no-op; applied ledger still:", sorted(applied))

# 3. Out-of-order apply is rejected instead of silently forking history.
db2, migrations2 = make_migrations()
applied2 = {}
try:
 apply_single(db2, applied2, migrations2, 3)
except RuntimeError as error:
 assert "out of order" in str(error), error
 print("out-of-order v3 rejected:", error)
else:
 raise AssertionError("applying v3 before v1..v2 must fail")
assert applied2 == {} and "index_on_email" not in db2

# 4. Rollback runs downs in reverse and only to the requested target.
assert migrate_down(db, applied, migrations, 2) == [1, 2]
assert "index_on_email" not in db and sorted(applied) == [1, 2]
assert migrate_up(db, applied, migrations) == [(1, "skipped"), (2, "skipped"), (3, "applied")]
print("rollback to v2 drops the index; re-up restores it:", sorted(applied))

# 5. A down migration destroys data: emails do not come back on re-up.
db["users"].append({"id": 1, "email": "a@example.com"})
assert migrate_down(db, applied, migrations, 1) == [1]
assert db["users"] == [{"id": 1}] # email column (and its values) is gone
assert migrate_up(db, applied, migrations)[1] == (2, "applied")
assert db["users"] == [{"id": 1, "email": None}] # re-up gives NULL, not the old address
print("down v2 drops stored emails; re-up yields None instead of the original")

# 6. Editing an already-applied migration is version drift, not a fix.
drifted = [(v, n + "_renamed" if v == 2 else n, u, d) for v, n, u, d in migrations]
try:
 migrate_up(db, applied, drifted)
except RuntimeError as error:
 assert "drift" in str(error), error
 print("edited history detected:", error)
else:
 raise AssertionError("renaming applied v2 must fail")

print("PASS: ordered up, idempotent ledger, rejected jump, ordered rollback, lossy down, drift check")

Six scenarios run in order: fresh replay applies v1 through v3, a second run is a no-op thanks to recorded identities, jumping to v3 is rejected, rollback to v2 removes only the index, rolling back v2 proves the down step is lossy as the address returns None, and renaming an applied migration trips the drift check. The final line prints PASS only if every assertion holds.

This fixture checks ordering, idempotency, rollback, and drift logic — not real database locks, concurrent migrator processes, or backup restores. The revised exact example passed on Python 3.12.14 on Linux. Before production, rehearse the real migration on a staging copy and confirm the down path.

Sources

golang-migrate/migrate, versioned up/down migrations with dirty-state tracking, checked 2026-10-15.
PostgreSQL: ALTER TABLE, locking and rewrite behavior of table changes, checked 2026-10-15.
strong_migrations, catches dangerous migrations such as unindexed or blocking operations, checked 2026-10-15.

A bounded retry gives a transaction another chance

Mon, 09 Nov 2026 00:00:00 +0000

A retry can recover a transaction rejected because of concurrent work. It cannot guarantee success, and it does not automatically make external effects idempotent. The retry unit should include every database read and application decision that determined the writes. Reusing an amount calculated from the first attempt’s snapshot can defeat the purpose of starting a fresh transaction.

Retry all reads and decisions after a confirmed concurrency rejection, within one budget. Stop on permanent errors or exhaustion. Reconcile unknown commit outcomes separately; rollback cannot undo external side effects.

When it breaks

PostgreSQL identifies serialization failures with 40001; deadlocks use 40P01. Either can justify another complete attempt after rollback, within a finite budget. A duplicate identifier producing 23505 may be permanent. Retrying all exceptions indiscriminately wastes capacity and hides bad input. Some uniqueness failures depend on earlier key-selection reads, but their policy requires knowledge of that particular operation.

Errors can appear during statements or commit. A server-declared rejection differs from losing the connection while awaiting commit confirmation: the latter may leave a committed operation whose acknowledgement never reached the caller. Replaying it blindly can duplicate work. A payment or message sent outside the database transaction also survives database rollback; a retry loop alone cannot undo or deduplicate it.

Check it quickly

Use Python 3 on Linux or macOS and PostgreSQL 18’s psql in your executable path. Set PGDATABASE, PGHOST, and any required connection settings to a disposable database where you can create a schema. Arrange authentication beforehand; the example disables interactive password prompts. Save the following program as retry-demo.py and run python3 retry-demo.py. Use one exclusive disposable database: an existing fixture schema makes setup fail. Setup precedes the cleanup block, so a later setup failure can leave newly created objects; inspect that failed run before retrying or cleaning up.

import os
import random
import re
import selectors
import subprocess
import time

PSQL = ['psql', '-X', '-w', '-qAt', '-v', 'ON_ERROR_STOP=1', '-v', 'VERBOSITY=sqlstate']
ENV = dict(os.environ, PGCONNECT_TIMEOUT='2')


def run(sql):
 return subprocess.run(PSQL, input=sql, text=True, capture_output=True,
 check=True, timeout=5, env=ENV).stdout.strip()


def read_value(worker):
 deadline = time.monotonic() + 5
 data = bytearray()
 with selectors.DefaultSelector() as ready:
 ready.register(worker.stdout, selectors.EVENT_READ)
 while b'\n' not in data:
 remaining = deadline - time.monotonic()
 if remaining <= 0 or not ready.select(remaining):
 raise TimeoutError('Initial read was not confirmed')
 chunk = os.read(worker.stdout.fileno(), 4096)
 if not chunk:
 raise RuntimeError('Connection ended before initial read')
 data.extend(chunk)
 return int(data.strip())


def attempt(mode, number):
 worker = subprocess.Popen(PSQL, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
 stderr=subprocess.PIPE, text=True, env=ENV)
 try:
 worker.stdin.write("SET statement_timeout='1s';\n"
 'BEGIN ISOLATION LEVEL REPEATABLE READ;\n'
 'SELECT n FROM retry_demo.counter WHERE id=1;\n')
 worker.stdin.flush()
 observed = read_value(worker)
 # A real second connection commits after this attempt's snapshot.
 if mode == 'always' or (mode == 'once' and number == 1):
 run('UPDATE retry_demo.counter SET n=n+1 WHERE id=1;')
 statement = ('INSERT INTO retry_demo.counter VALUES (1,99);' if mode == 'permanent'
 else f'UPDATE retry_demo.counter SET n={observed + 1} WHERE id=1;')
 finish = r'''
\set ON_ERROR_STOP off
__STATEMENT__
\if :ERROR
 \echo ATTEMPT :SQLSTATE
 ROLLBACK;
\else
 COMMIT;
 \if :ERROR
 \echo ATTEMPT :SQLSTATE
 \else
 \echo ATTEMPT 00000
 \endif
\endif
'''.replace('__STATEMENT__', statement)
 output, _ = worker.communicate(finish, timeout=5)
 match = re.fullmatch(r'ATTEMPT ([0-9A-Z]{5})', output.strip())
 if worker.returncode != 0 or match is None:
 raise RuntimeError('No confirmed transaction outcome; do not replay blindly')
 return match[1], observed
 finally:
 if worker.poll() is None:
 worker.kill()
 worker.communicate()


def increment(mode):
 observations = []
 for number in range(1, 4):
 state, observed = attempt(mode, number)
 observations.append(observed)
 if state not in {'40001', '40P01'} or number == 3:
 return {'attempts': number, 'state': state, 'reads': observations}
 time.sleep(random.uniform(0, 0.01 * 2 ** (number - 1)))


run('CREATE SCHEMA retry_demo; '
 'CREATE TABLE retry_demo.counter (id integer PRIMARY KEY,n integer); '
 'INSERT INTO retry_demo.counter VALUES (1,0);')
try:
 cases = [
 ('once', {'attempts': 2, 'state': '00000', 'reads': [0, 1]}, 2),
 ('permanent', {'attempts': 1, 'state': '23505', 'reads': [0]}, 0),
 ('always', {'attempts': 3, 'state': '40001', 'reads': [0, 1, 2]}, 3),
 ]
 for mode, expected, expected_value in cases:
 run('UPDATE retry_demo.counter SET n=0 WHERE id=1;')
 result = increment(mode)
 value = int(run('SELECT n FROM retry_demo.counter WHERE id=1;'))
 assert result == expected and value == expected_value, (mode, result, value)
 print(mode, result, 'stored', value)
finally:
 run('DROP SCHEMA retry_demo CASCADE;')

The program uses two actual database connections. Each attempt starts Repeatable Read, reads the counter, and calculates a new absolute value. A competing connection can commit between that read and update, forcing the original snapshot’s update to fail. This deliberately exposes whether the retry reads again rather than merely resending the original calculated value.

For the once case, expect two attempts, final state 00000, observations zero then one, and stored value two. One increment belongs to the competing connection and one to the successful retry. The rejected first attempt contributes nothing. If the second attempt reused the old calculated value, it would not preserve that result.

For permanent, expect one attempt, state 23505, observation zero, and stored value zero. The existing primary key makes the requested insert invalid. The classifier stops immediately instead of spending the whole retry allowance on unchanged input. No successful application increment occurred.

For always, expect three attempts, state 40001, observations zero, one, and two, and stored value three. All three stored increments belong to the competing connection; the application never committed its operation. This is deliberate exhaustion, not eventual success. The assertions check both the control-flow report and durable data, because the database value alone would not identify which participant succeeded.

Keep error handling attached to the decision

The example asks psql for its structured SQLSTATE and emits a dedicated result marker. It does not search localized error-message wording. Expected statement errors are followed by rollback; commit errors are checked separately. Missing confirmation, process failure, and transport timeouts escape the retry classifier instead of being guessed into a concurrency category. The initial read and process completion have separate time bounds. These three cases do not generate a deadlock or inject a lost commit reply, and the harness does not implement a shared deadline or caller cancellation.

In an application, use the database driver’s transaction and structured error APIs rather than copying this process harness. Include connection acquisition, every attempt, and backoff in the request’s total deadline. Record attempts and the final error, and stop when the caller cancels. Jitter spreads retries but does not repair permanent contention or guarantee fairness.

Give externally visible operations stable identifiers and a durable way to discover their outcome. Reconcile an uncertain commit before reissuing a non-idempotent action. Keep irreversible effects outside a retryable callback unless their delivery protocol explicitly tolerates repetition. These boundaries are what make recovery safe; a small loop and a sleep only implement the scheduling part.

Sources

PostgreSQL 18 serialization failure handling, retry classifications and recomputing the whole transaction, checked 2026-09-10.
PostgreSQL 18 transaction isolation, snapshot conflicts used by the experiment, checked 2026-09-10.
PostgreSQL 18 psql, SQLSTATE, ERROR, conditional execution, and client error handling, checked 2026-09-10.

Zero-downtime migrations change schema while old code runs

Tue, 03 Nov 2026 00:00:00 +0000

A zero-downtime migration is a compatibility and availability goal, not a guarantee attached to small statements. Old and new application versions may overlap while data is changing. Plan the schema, writer behavior, backfill, reader cutover, and recovery together. Dropping information is not automatically reversible, even when the preceding steps were additive.

Lock acquisition, row conflicts, resource usage, and application assumptions all matter. A metadata-only change can wait behind another transaction; a backfill can exhaust I/O or replication capacity without holding an exclusive table lock.

Expand, backfill, contract

Expand introduces a representation that existing readers and writers can tolerate. Deploy compatible writers before relying on new data, backfill existing rows, verify the invariant, switch readers, and contract only after old binaries, background jobs, and rollback requirements no longer need the old shape. Adding a nullable column is often useful, but still needs a lock and compatibility checks.

Dual writing must define failure and ordering behavior. Two columns updated in one database transaction can remain consistent; separate writes can partially succeed. Old writers that still update only the old column can invalidate an already backfilled value. Retire them or provide a tested synchronization mechanism before declaring convergence. A rollback is safe only while its required representation remains accurate.

Use restartable batches and a stable progress key. Prevent stale backfill data from overwriting a concurrent application update, for example through a transactionally checked version or a suitable conditional update. A null-only predicate is sufficient only for the intended invariant and writer behavior. Recheck skipped or concurrently changed rows and validate after writers converge. Batching limits individual transactions but can add total work; it is not a substitute for correct phase ordering.

Locks are one failure boundary

In PostgreSQL, ACCESS EXCLUSIVE conflicts with ordinary reads and writes. Many ALTER TABLE forms take it even when no table rewrite is needed. In contrast, an ordinary UPDATE takes ROW EXCLUSIVE plus row locks; it does not globally exclude every writer. Conflicts depend on affected rows and other operations, while long transactions retain locks until completion.

A queued strong lock can delay later conflicting requests. Identify actual blockers, transaction boundaries, and wait events rather than blaming whichever transaction is oldest. SQLite rollback-journal and WAL behavior are different engine contracts. The earlier article’s millisecond and page-count figures have no exact benchmark evidence established here and are not PostgreSQL predictions or measurements from this review.

Validate under a weaker lock

For PostgreSQL 17, add an appropriate CHECK ... NOT VALID only once current writers satisfy it: new inserts and updates are checked immediately, while old rows await validation. VALIDATE CONSTRAINT scans under SHARE UPDATE EXCLUSIVE, which permits ordinary DML but conflicts with some maintenance and schema operations. It is not lock-free.

A validated CHECK proving the column cannot be null can let SET NOT NULL skip its scan, provided that check is not dropped in the same command. This does not eliminate the stronger lock needed by the final alteration, and setting NOT NULL does not inherently rewrite the table. PostgreSQL 18 adds not-null constraint capabilities, so consult the deployed major version rather than assuming every release supports identical syntax.

When it breaks

A backfill finishes once, then a surviving old writer makes the new representation stale. Monitor mismatch counts and writer versions before cutting reads over. A destructive contract phase may require restore or forward repair; rehearse that recovery before removing data.

Ordinary CREATE INDEX blocks writes during construction. CREATE INDEX CONCURRENTLY permits normal DML, but performs extra scans and waits, cannot run inside a transaction block, and can leave an invalid index after failure. Such an index may still impose update overhead or enforce uniqueness. Inspect its exact state before an approved recovery operation.

Migration compatibility: mechanism and evidence boundaries.

Check it quickly

These retained commands are an operator checklist, not an isolated migration test. Use only an explicitly selected disposable PostgreSQL database where an owned events table exists and note does not. Confirm connection and search_path, suitable catalog visibility, DDL permission, and a separate total statement timeout. The fourth command changes schema; do not paste it into production.

psql -c "SELECT pid, state, now()-xact_start AS age, left(query,50) FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY xact_start;"
psql -c "SELECT pid, mode, granted, left(relation::regclass::text,40) FROM pg_locks WHERE NOT granted;"
psql -c "SELECT indexrelid::regclass, indisvalid FROM pg_index WHERE NOT indisvalid;"
psql -c "SET lock_timeout='3s'; ALTER TABLE events ADD COLUMN note text;"
psql -c "SELECT conname, convalidated FROM pg_constraint WHERE conrelid='events'::regclass;"

The activity query lists candidates, not proven blockers; pg_blocking_pids supplies blocking relationships. Ungranted locks do not identify every holder. lock_timeout bounds each acquisition wait, not execution time or total impact. convalidated alone does not show which condition was proven. No commands ran here; rollout compatibility, concurrent backfill, constraint transitions, and recovery still need an authorized, version-pinned fixture.

Sources

PostgreSQL: ALTER TABLE, SET NOT NULL, NOT VALID, and VALIDATE CONSTRAINT, checked 2026-09-16.
PostgreSQL: CREATE INDEX, building indexes concurrently and the invalid-index failure mode, checked 2026-09-16.
PostgreSQL: explicit locking, which commands take ACCESS EXCLUSIVE and SHARE, checked 2026-09-16.
Martin Fowler: parallel change, the expand/contract pattern under another name, checked 2026-09-16.
PostgreSQL 17: ALTER TABLE, version-specific CHECK validation and NOT NULL transitions, checked 2026-10-18.
PostgreSQL 17: client defaults, lock wait timeout versus statement duration, checked 2026-10-18.
PostgreSQL 17: system information functions, actual blocker relationships through pg_blocking_pids, checked 2026-10-18.
PostgreSQL 17: transaction isolation, concurrent updates and predicate rechecks, checked 2026-10-18.