Index · XB Field Notes

InnoDB covering indexes carry the query's columns

Fri, 03 Mar 2028 00:00:00 +0000

A covering index contains the columns needed to evaluate a particular query and return its requested values. In InnoDB, secondary-index records also carry the primary-key columns, so a query can be covered without explicitly listing the primary key again in the secondary-index definition. Coverage is a relationship between one query and an index, not a permanent property that applies to every query against the table.

When it breaks

Adding a projected column can remove coverage. An index that supplies an identifier and timestamp may not contain the large payload that a later application change begins selecting. The optimizer can still use that same index to locate matching rows, then obtain missing values from the clustered records. Index use and covering access are different facts.

Coverage also does not guarantee a selective access path. A wide scan can be fully covered while examining many entries. Column order, predicates, sorting requirements, and row estimates still matter. Do not interpret Using index as evidence that a query is fast, or confuse it with Using index condition, which describes index condition pushdown rather than complete projection coverage.

InnoDB visibility checks qualify the usual index-only explanation. Secondary records do not contain the same hidden transaction metadata as clustered records. Delete-marked secondary entries or pages changed by newer transactions can require clustered-record lookups to obtain the correct visible version. A covering plan is therefore not a universal promise of zero clustered access under concurrent MVCC activity.

Projection coverage and MVCC visibility are separate. Historical MySQL plans lack a recovered raw log; neither coverage nor pushdown proves speed.

Check it quickly

The unchanged fixture was not rerun in this source review. The retained review records say MySQL 8.4 was unavailable, and no raw MySQL run log was recovered in the retained validation files; the reported plans remain attributed historical observations.

Use a fresh MySQL 8.4 instance and a client session with permission to create a disposable schema. The recorded run used MySQL 8.4.11. Ensure xbcatm_index_lab is unused, stop on unexpected SQL errors, and keep the same session so its selected database remains active. Create ten thousand rows, with one hundred paid orders and a deliberately wider payload:

CREATE DATABASE xbcatm_index_lab;
USE xbcatm_index_lab;
CREATE TABLE orders (
 id INT PRIMARY KEY,
 status VARCHAR(10) NOT NULL,
 created_at DATETIME NOT NULL,
 payload VARCHAR(1000) NOT NULL,
 INDEX by_status_created (status,created_at)
) ENGINE=InnoDB;
INSERT INTO orders
WITH digits(n) AS (
 SELECT 0 UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3
 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6
 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9
)
SELECT number+1,IF(MOD(number+1,100)=0,'paid','open'),
 TIMESTAMP '2026-01-01 00:00:00' + INTERVAL (number+1) SECOND,
 REPEAT('x',640)
FROM (SELECT a.n+10*b.n+100*c.n+1000*d.n AS number
 FROM digits a CROSS JOIN digits b CROSS JOIN digits c CROSS JOIN digits d) AS numbered;
ANALYZE TABLE orders;
SELECT COUNT(*),SUM(status='paid') FROM orders;

Manually verify that the final counts are ten thousand and one hundred; these SELECT statements print results without automated assertions. The secondary index contains status and creation time explicitly; InnoDB supplies the primary identifier implicitly. Inspect and run a query that needs only those values:

EXPLAIN SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded tabular plan reports by_status_created and Using index. The analyzed plan identifies a covering index lookup and returns twenty rows. Result identifiers are one hundred, two hundred, and so on through two thousand, ordered by the timestamp and identifier. These are checked results on populated data, not a plan inferred from an empty schema.

Now request the payload as well:

EXPLAIN SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded plan still uses by_status_created, but loses Using index; the analyzed plan describes an index lookup without the covering qualification. This isolates the effect of changing the requested columns while keeping the filter, ordering, and index unchanged. The diagnostic execution does run the SELECT, so account for that when applying EXPLAIN ANALYZE to expensive real queries.

On success, remove the schema created for this exercise. MySQL DDL can commit implicitly, so an error or interruption does not roll the setup back; inspect and remove only this fixture schema afterward:

DROP DATABASE xbcatm_index_lab;

The timings from this tiny warm experiment are not a throughput or latency benchmark. Scheduling, caching, and instrumentation can outweigh the difference between two twenty-row reads. The verified conclusion concerns access shape and returned values, not a promised percentage reduction in physical I/O.

Pay for coverage deliberately

Appending columns to a secondary index consumes storage and buffer-pool capacity and increases maintenance when those indexed values change. Long primary keys enlarge secondary entries too. Cover the important narrow query rather than automatically copying a large payload into every index. Keep projected columns explicit so later application changes make their access requirements visible in review.

Compare representative plans, examined rows, foreground latency, and write cost before retaining a new index. Test with realistic selectivity and concurrent activity; a quiet fixture does not measure MVCC fallback frequency or replica maintenance cost. An index should earn its place through the workload it serves, not through the appearance of a desirable label in one explanation.

Sources

MySQL 8.4 clustered and secondary indexes, primary-key columns carried by secondary records, checked 2026-09-12.
MySQL 8.4 EXPLAIN output, index-use and covering-plan indicators, checked 2026-09-12.
MySQL 8.4 InnoDB multiversioning, clustered lookups required for secondary-index visibility checks, checked 2026-09-12.

Partial indexes and writes

Tue, 15 Feb 2028 00:00:00 +0000

A partial index filters rows, not columns. An index on id WHERE status = 'open' stores identifiers for open rows; status decides membership without becoming the search key. That distinction matters when estimating maintenance: a column need not appear in the stored key to affect an update’s cost.

For a table dominated by closed orders, excluding that history can save substantial index space. It does not make closed rows free to write, guarantee faster queries, or remove the table’s vacuum work. Evaluate the smaller structure alongside the writes that keep it correct.

Membership answers which rows belong. Write cost also depends on changed keys, predicate-column dependencies and same-page HOT eligibility.

Separate membership from update cost

An inserted row outside the predicate needs no entry in this particular index. Predicate evaluation, table storage, other indexes, and ordinary transaction work remain. Moving a row into the subset creates an entry. Moving it out changes visible membership; obsolete row versions and index entries are reclaimed later, rather than necessarily disappearing physically at that moment.

Keeping membership unchanged is not enough to skip maintenance. An open order whose indexed identifier changes needs a new index entry. Even changing an unrelated note can require new entries when PostgreSQL cannot use a heap-only tuple, or HOT, update.

For PostgreSQL 18 B-tree indexes, HOT requires unchanged values in every column referenced by the table’s index definitions, including expressions and partial-index predicates, plus space for the new tuple on the same heap page. Changing status from closed to archived blocks HOT even though neither value satisfies this index. Changing only an unindexed note can qualify. Lower table fillfactor leaves room but is no guarantee; other indexes can independently block the optimization.

This rule is version- and index-method-sensitive. PostgreSQL 16 introduced an exception for summarizing indexes such as BRIN; those summaries may still need updating during HOT. That exception does not apply to the B-tree example here. Read the documentation for the deployed version before extrapolating update behavior.

Eligibility is separate from usefulness

The planner must prove that query conditions imply the partial predicate. An open-only index cannot by itself answer a request including closed orders. A generic plan for status = $1 lacks that proof; a custom plan can know the supplied value. Parameters elsewhere in a query with literal status = 'open' do not erase that fact.

An eligible index still competes on cost. A sequential scan can be sensible for a small table or broad result. Inspect estimates, actual rows, and buffers before blaming predicate spelling. A test that always demands one index name confuses a performance hypothesis with a correctness requirement.

When it breaks

A growing backlog can make the open subset large; frequent status changes can also erase expected write savings. Compare representative reads and transitions, rather than extrapolating from index size alone. Many overlapping partial indexes add planning and maintenance work. Review the whole index set before adding another slice or removing an apparently unused index, especially one enforcing uniqueness.

Check it quickly

Run both blocks in one fresh psql -X -v ON_ERROR_STOP=1 session on a disposable PostgreSQL 18 database with track_counts enabled. Historical review records report SQL checks on PostgreSQL 18.6 in socket-free single-user mode; the client/server connection path was not tested. The temporary table contains 20,000 rows, with 200 open; explicit names avoid guessing PostgreSQL’s generated index names.

BEGIN;
CREATE TEMP TABLE partial_cost_demo (
 id integer NOT NULL,
 status text NOT NULL,
 note integer NOT NULL
) WITH (fillfactor=50);
INSERT INTO partial_cost_demo
SELECT g, CASE WHEN g % 100 = 0 THEN 'open' ELSE 'closed' END, 0
FROM generate_series(1,20000) AS g;
CREATE INDEX cost_all_id ON partial_cost_demo (id);
CREATE INDEX cost_open_id ON partial_cost_demo (id) WHERE status='open';
ANALYZE partial_cost_demo;
SELECT pg_relation_size('cost_all_id') AS full_bytes,
 pg_relation_size('cost_open_id') AS partial_bytes;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT id FROM partial_cost_demo WHERE status='open' ORDER BY id LIMIT 5;
SELECT id FROM partial_cost_demo WHERE status='open' ORDER BY id LIMIT 5;

Expect identifiers 100 through 500 in steps of 100. The partial index should occupy fewer bytes, without an exact one-percent size promise: page overhead matters. Record the chosen plan rather than asserting its shape. The full index supplies a same-key size comparison and remains a competing access path.

UPDATE partial_cost_demo SET note=1 WHERE id=100;
SELECT n_tup_upd, n_tup_hot_upd FROM pg_stat_xact_user_tables
WHERE relid='partial_cost_demo'::regclass;
UPDATE partial_cost_demo SET status='archived' WHERE id=101;
SELECT n_tup_upd, n_tup_hot_upd FROM pg_stat_xact_user_tables
WHERE relid='partial_cost_demo'::regclass;
SELECT count(*) FROM partial_cost_demo WHERE status='open';
ROLLBACK;

The transaction-local counters update continuously: total updates progress from one to two. The note update may increase the HOT count; the status change should not, despite unchanged predicate membership. The isolated run recorded counters of (1, 1), then (2, 1), and 200 open rows. Index sizes were 466,944 and 16,384 bytes, respectively; the chosen index-only scan still made five heap fetches. These are fixture observations, not universal expectations. Rollback removed the fixture. This separates HOT eligibility from subset size; temporary-table timings do not establish production WAL savings or sustained write throughput.

Sources

PostgreSQL 18 partial indexes, membership, implication, and excessive partial indexes, checked 2026-10-15.
PostgreSQL 18 HOT, update eligibility and page space, checked 2026-10-15.
PostgreSQL 18 HOT implementation notes, predicate-column dependencies and tuple chains, checked 2026-10-15.
PostgreSQL 16 release notes, BRIN exception introduced in version 16, checked 2026-10-15.
PostgreSQL 18 PREPARE, custom and generic plans, checked 2026-10-15.
PostgreSQL 18 statistics, transaction-local counters, checked 2026-10-15.
PostgreSQL 18 EXPLAIN, execution and buffer measurements, checked 2026-10-15.
PostgreSQL 18 size functions, main-fork relation size, checked 2026-10-15.

Hash indexes recheck collisions and maintain changed keys

Mon, 24 Jan 2028 00:00:00 +0000

PostgreSQL hash indexes narrow an equality search using a hash code, then verify the original value in the table. A hash match is a candidate, not proof of equality. Two different values can have the same code, and different codes can also map to the same bucket. Neither collisions nor shared buckets imply incorrect query results when the normal recheck is performed.

What lives in a bucket

An index tuple contains a four-byte hash value and a reference to the table tuple; it does not contain the full indexed value. The bucket mapping directs a lookup to relevant index pages. Buckets are shared storage, not one separately allocated bucket for every distinct key. An overflowing bucket can acquire additional pages that its scans must visit.

PostgreSQL can sort hash codes within an individual index page to accelerate local searching, but that does not provide ordering by the original column. A hash index cannot satisfy a key range or an ordered result. It supports single-column equality access, cannot enforce uniqueness, and does not support IS NULL. A B-tree is still a candidate for equality as well as the broader operations it supports.

The absence of original values makes hash scans lossy: matching index entries must be checked against table values. A collision can therefore appear as Rows Removed by Index Recheck in an execution plan. That is evidence that candidate verification worked, not an indication that the database returned both colliding values as equal.

When it breaks

Describing an indexed update as touching only one bucket hides important work. Changing the indexed key changes the tuple’s indexed value and may send its new entry to another bucket. PostgreSQL’s versioned rows and deferred index cleanup mean obsolete references do not necessarily disappear immediately. Growth can also split a bucket, and overflow pages add I/O.

Consequently, a workload with heavily repeated values or rapid growth can behave differently from a nearly unique, steady equality workload. Hash lookup is not an unconditional constant-time latency guarantee. Modern PostgreSQL hash indexes are crash recoverable, but durability does not eliminate storage maintenance or turn the structure into a uniqueness constraint.

Equal codes and shared buckets are different. Shapes are schematic; the named historical collision lacks a recovered raw log and was not rerun.

Check it quickly

Run this in a fresh psql session on a disposable PostgreSQL 18 database, stopping on any SQL or psql meta-command error. It creates two hundred thousand temporary text keys using C collation, discovers a collision with the server’s own hashtext, and tests lookup and update behavior. It does not assume hash values are portable identifiers. Rollback removes the fixture.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE hash_items (key text COLLATE "C");
INSERT INTO hash_items SELECT 'key-'||g FROM generate_series(1,200000) AS g;
CREATE TEMP TABLE collision_pair AS
SELECT min(key) AS a,max(key) AS b FROM hash_items
GROUP BY hashtext(key) HAVING count(*)>1 ORDER BY hashtext(key) LIMIT 1;
SELECT a AS collision_a,b AS collision_b FROM collision_pair \gset
SELECT a,b,hashtext(a),hashtext(b) FROM collision_pair;
CREATE INDEX hash_items_key ON hash_items USING hash(key);
ANALYZE hash_items;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT key FROM hash_items WHERE key=:'collision_a';
UPDATE hash_items SET key='changed-'||key WHERE key=:'collision_a';
SELECT count(*) AS old_key_rows FROM hash_items WHERE key=:'collision_a';
SELECT count(*) AS new_key_rows FROM hash_items WHERE key='changed-'||:'collision_a';
DO $$ BEGIN
 IF (SELECT hashtext(a)=hashtext(b) AND a<>b FROM collision_pair) IS DISTINCT FROM true THEN
 RAISE EXCEPTION 'fixture did not establish a collision';
 END IF;
 IF (SELECT count(*) FROM hash_items h JOIN collision_pair p ON h.key=p.a) <> 0 OR
 (SELECT count(*) FROM hash_items h JOIN collision_pair p ON h.key='changed-'||p.a) <> 1 OR
 (SELECT count(*) FROM hash_items h JOIN collision_pair p ON h.key=p.b) <> 1 THEN
 RAISE EXCEPTION 'hash lookup or indexed update returned wrong keys';
 END IF;
END $$;
ROLLBACK;

The historical article report, not rerun here and without a recovered raw collision log in the retained validation files, says that PostgreSQL 18.6 discovered key-129539 and key-7295, both with hash code -1960436116. The equality plan used hash_items_key, returned one row, and reported one row removed by index recheck. The two strings remained distinct despite their identical hash code. This is a concrete negative control against treating hashes as collision-free keys.

Updating the first key to its changed- form left zero rows under its old value and one under its new value. The other colliding key still had one row. The final assertions verify all three facts and verify that the original pair really collided. A successful update alone would not establish that unrelated colliding entries remained searchable.

The particular pair and plan are observations from this version. If a different environment does not discover a pair, the script must stop rather than claiming a collision was tested. Even when the pair changes, the required semantics stay the same: exact equality returns only matching original values, and updating one value does not change another merely because their hashes coincide.

Evaluate maintenance with lookup performance

Measure page accesses, equality latency, update behavior, and index size using representative distributions. VACUUM can remove dead references and recycle overflow pages, but it does not shrink the hash index’s bucket count or return its allocation to the operating system. Rebuilding is a separate maintenance operation with its own cost. A smaller hash entry can be useful, yet only a full workload comparison establishes whether that benefit exceeds recheck, overflow, and write overhead.

Sources

PostgreSQL 18: Hash indexes, collision rechecks and bucket maintenance, checked 2026-09-10.
PostgreSQL 18: Index types, supported predicates for B-tree and hash indexes, checked 2026-09-10.

Postgres JSONB pairs with GIN for flexible queries

Wed, 28 Jul 2027 00:00:00 +0000

A PostgreSQL JSONB column can hold documents with different shapes, but efficient querying still requires a precise contract between the predicate and the index. A GIN index on the whole document supports particular JSONB operators. It does not automatically accelerate every expression that extracts a field from that document.

Match the operator to the index

The containment predicate payload @> '{"status":"pending"}' asks whether the document contains that key and JSON string value. With the default jsonb_ops operator class, a whole-column GIN index can support containment and top-level key existence, together with supported JSON path searches. The index stores searchable components; a bitmap heap scan may still visit the table and recheck candidate rows.

By contrast, payload->>'status' = 'pending' compares extracted text. A GIN index on payload is not an index on that text expression. A B-tree expression index on (payload->>'status') matches the scalar equality query. If many queries sort, join, or constrain status, an ordinary typed column may express the contract more clearly and avoid repeatedly deriving it from flexible documents.

Do not rewrite every text predicate to containment without checking types. The text extraction of a JSON number and a JSON string can produce the same text, while containment distinguishes their JSON types. A rewrite is equivalent only under the relevant data assumptions. Parameterize values through the database driver instead of constructing JSON or SQL by concatenating untrusted input.

An operator class is also a capability choice. jsonb_path_ops supports containment and supported path-match operators, but not the key-existence operators supplied by jsonb_ops. It can be smaller and more selective for suitable data, yet it is not a universally better replacement. Likewise, an index on a nested expression must match the expression actually queried; indexing the complete document does not imply every extracted subtree has its own index.

Missing and null are different questions

An absent key, a key whose value is JSON null, and a key containing the string “null” are distinct states. The existence operator detects the latter two as present. Extracting JSON with -> returns SQL NULL for a missing key, but a JSON null value for an explicit null. Extracting text with ->> maps both missing and JSON null to SQL NULL; the string remains text.

This affects filters, uniqueness assumptions, and application defaults. Use a key-existence check when presence matters and inspect JSON types when a field must be a string or number. SQL equality with NULL does not mean “is absent.” Also remember that the ? operator searches the top level; it does not recursively find a key nested anywhere in the document.

When it breaks

A common performance failure is installing GIN while leaving an incompatible extraction predicate unchanged. Another is assuming that an eligible index must be selected. Statistics, selectivity, table size, and estimated costs influence the plan; reading much of the table can make a sequential scan appropriate. Compare estimated and actual row counts before blaming the access method.

Large flexible documents also increase write work. GIN maintains many entries, and updating a document still updates a row with associated locking and storage costs. Index only the questions the workload actually asks. A valid JSONB value is not evidence that required business fields exist or have acceptable types. If adding a CHECK, remember that SQL NULL passes a check expression; require presence and the intended type explicitly instead of relying only on nullable text extraction.

Containment was executed historically; scalar extraction was only planned. Matching text is not enough to prove typed containment equivalent.

Check it quickly

Save this as json-query.sql and run psql -X -v ON_ERROR_STOP=1 -f json-query.sql against a disposable PostgreSQL 18 database. The transaction creates temporary data and removes it on rollback. Keep the default unlimited gin_fuzzy_search_limit=0. Only one percent of rows have pending status.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE documents (id integer, payload jsonb NOT NULL);
INSERT INTO documents SELECT g, jsonb_build_object('status',
 CASE WHEN g%100=0 THEN 'pending' ELSE 'done' END)
FROM generate_series(1,10000) AS g;
CREATE INDEX documents_payload ON documents USING gin(payload);
ANALYZE documents;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT id FROM documents WHERE payload @> '{"status":"pending"}';
EXPLAIN (COSTS OFF)
SELECT id FROM documents WHERE payload->>'status'='pending';
CREATE INDEX documents_status ON documents ((payload->>'status'));
ANALYZE documents;
EXPLAIN (COSTS OFF)
SELECT id FROM documents WHERE payload->>'status'='pending';
SELECT value, value ? 'x' AS has_key,
 value->'x' IS NULL AS sql_null_json, value->>'x' IS NULL AS sql_null_text
FROM (VALUES ('{}'::jsonb),('{"x":null}'::jsonb),('{"x":"null"}'::jsonb)) v(value);
DO $$ BEGIN
 IF (SELECT count(*) FROM documents WHERE payload @> '{"status":"pending"}') <> 100 THEN
 RAISE EXCEPTION 'wrong containment count';
 END IF;
 IF '{"n":1}'::jsonb @> '{"n":"1"}'::jsonb THEN
 RAISE EXCEPTION 'number and string were conflated';
 END IF;
 IF '{"outer":{"x":1}}'::jsonb ? 'x' THEN
 RAISE EXCEPTION 'top-level existence searched recursively';
 END IF;
 IF ('{"x":null}'::jsonb->'x') IS NULL THEN
 RAISE EXCEPTION 'JSON null confused with missing key';
 END IF;
END $$;
ROLLBACK;

Retained PostgreSQL 18.6 standalone SQL logs show containment using documents_payload and returning 100 rows. This source review did not rerun it; the psql directive and client invocation were not exercised. EXPLAIN without ANALYZE planned the text predicate first as a sequential scan, then through documents_status. Those two calls did not execute the text query or compare its returned identifiers with containment. These are observed plans for this distribution, not fixed plan guarantees or production latency estimates.

The final three rows show missing, explicit JSON null, and the text string separately. Their existence flags are false, true, true; their text-is-SQL-null flags are true, true, false. Assertions also check typed containment and nonrecursive existence. Keep these semantic checks when changing query shape: a faster plan is useful only if it still returns the intended documents.

Sources

PostgreSQL JSON types and indexing, GIN operator classes and indexed expressions, checked 2026-10-15.
PostgreSQL JSON functions and operators, extraction, existence, containment, and null semantics, checked 2026-10-15.
PostgreSQL 18: CHECK constraint null behavior, CHECK acceptance of unknown results and explicit null constraints, checked 2026-10-15.

A Postgres index lives to answer a query

Tue, 20 Jul 2027 00:00:00 +0000

An index is valuable because it changes the work needed by a real query. Counting indexed columns is not a useful success measure. A narrow B-tree that finds one customer’s newest three orders can outperform a wider structure that cannot supply the needed ordering, while a query reading almost every row may correctly ignore both. Evaluate access, ordering, and result volume together.

Start from the requested result

Suppose the application asks for the newest three orders for one customer. Equality on customer narrows the population, and descending placed_at chooses the newest records within that customer. A final id key gives deterministic ordering when timestamps tie. The index (customer, placed_at DESC, id DESC) expresses that useful sequence of keys.

The benefit is not just avoiding a table scan. An ordered index path can stop after producing the limited result instead of collecting and sorting every matching order. Without LIMIT, or when most rows are needed, the balance can change. The same index must be judged against the full statement and its common parameter values, not an isolated WHERE fragment.

Textual condition order does not set index order. PostgreSQL plans logical expressions, so moving customer equality before or after another AND condition does not redefine the tree. Index direction, null ordering, leading-key constraints, and the requested output order are relevant structural properties. Verify them explicitly when adapting a query rather than applying a blanket column-order slogan. PostgreSQL 18 can also use B-tree skip scans for some conditions on later keys when the missing leading key has few distinct values; absence of that leading equality is not an absolute prohibition.

Read the chosen plan honestly

EXPLAIN shows the planner’s proposed work, and EXPLAIN ANALYZE executes it to show actual rows and loops. A sequential scan can be the right answer when a query returns most of a table, especially when it also needs wide columns absent from the index. Such a plan is not proof that column order is wrong or that index creation failed.

Conversely, seeing an index name does not prove a query is cheap. An index scan can visit many entries and heap tuples, a bitmap path can need a separate sort, and an index-only plan can still perform heap visibility checks. Read actual rows, Heap Fetches where applicable, buffer activity, and sorting together. A LIMIT near the top is useful only if lower nodes avoid doing excessive work first.

When it breaks

A production parameter can select thousands of times more rows than a development value. The index that looked excellent for a small customer may be less beneficial for an exceptionally large one. Data distribution, current statistics, caching, and concurrency all affect the plan and observed latency. Retain representative parameter cases when validating a change.

Indexes also impose persistent costs: storage, cache pressure, insert/update maintenance, and build time. Adding note as payload merely because one diagnostic query selected it can make a hot write path more expensive. Separate the narrow application lookup from occasional broad reporting, and require evidence that each extra key or included column serves a recurring need.

A final failure is using unspecified ordering in pagination. The first few rows can change across plans or tied timestamps unless ORDER BY defines a deterministic choice. An index does not independently establish an API ordering contract. Keep that order explicit in SQL and validate exact identifiers, not just the number of returned rows.

Ordered keys and LIMIT serve the narrow query. Historical index-only execution still fetched the heap; identifiers, rather than plan shape, are asserted.

Check it quickly

Save the following as topn.sql and run psql -X -v ON_ERROR_STOP=1 -f topn.sql against a disposable PostgreSQL 18 database. It creates twenty thousand orders spread over one hundred customers, including a wide note column for the broad-query comparison.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE orders (id integer, customer integer, placed_at integer, note text);
INSERT INTO orders SELECT g, g%100, g, repeat('x',100) FROM generate_series(1,20000) AS g;
CREATE INDEX orders_customer_recent ON orders (customer, placed_at DESC, id DESC);
ANALYZE orders;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT id,placed_at FROM orders WHERE customer=7 ORDER BY placed_at DESC,id DESC LIMIT 3;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT * FROM orders WHERE customer>=0;
SELECT array_agg(id ORDER BY placed_at DESC,id DESC) AS newest
FROM (SELECT id,placed_at FROM orders WHERE customer=7
 ORDER BY placed_at DESC,id DESC LIMIT 3) s;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY placed_at DESC,id DESC)
 FROM (SELECT id,placed_at FROM orders WHERE customer=7
 ORDER BY placed_at DESC,id DESC LIMIT 3) s)
 IS DISTINCT FROM ARRAY[19907,19807,19707] THEN
 RAISE EXCEPTION 'wrong newest orders';
 END IF;
END $$;
ROLLBACK;

The customer-seven query returns ids 19907,19807,19707 in that order. Retained PostgreSQL 18.6 standalone SQL logs show a Limit over an index-only scan, stopping after three matching rows with three heap visibility fetches. This source review did not rerun it; the psql directive and client invocation were not exercised. The all-customers query selected a sequential scan and returned twenty thousand rows including their notes. Both choices are consistent with their different tasks.

The assertion checks the deterministic top-three result; timing and buffer numbers are deliberately not fixed expectations. Rollback removes the fixture. This demonstrates workload-shaped index use, not a universal speed ratio or proof that the same plan will remain optimal as production data and query requirements evolve.

Sources

PostgreSQL multicolumn indexes, key constraints and scan scope, checked 2026-10-15.
PostgreSQL indexes and ORDER BY, ordered retrieval and LIMIT, checked 2026-10-15.
PostgreSQL using EXPLAIN, interpreting actual plan work, checked 2026-10-15.

A function index precomputes a value at write time

Wed, 05 May 2027 00:00:00 +0000

An expression index stores the result of a computation as an index key. A query comparing lower(email) can search those precomputed keys instead of evaluating lower over every table row. This moves work into index creation and maintenance. It does not change the stored email, guarantee that every query uses the index, or make every textual variation a different expression.

Precomputed keys move work into index creation and maintenance. Their meaning, uniqueness rule and immutability must stay valid.

Index the computed meaning

The index on lower(email) contains normalized values associated with rows. An equality predicate using that expression can become an index condition. Ordinary equality on the original email has different semantics and does not automatically gain the same search path. Likewise, applying trim before lower adds another operation; the existing index cannot assume trimming never changes data.

PostgreSQL matches parsed expressions and performs some simplification. Extra parentheses do not inherently defeat matching, so queries need not use identical source text. Conversely, mathematically equivalent expressions are not always recognized as interchangeable. Inspect the planned condition and the actual expression definition rather than relying on a cosmetic string comparison or a universal equivalence promise.

A unique expression index can also enforce a rule such as case-normalized uniqueness. That is a change to allowed data, not just an optimization. Existing conflicting rows will prevent its creation. Lowercasing alone is not a complete definition of email equivalence, and locale or collation behavior can matter for other text. Choose a business normalization rule intentionally before making it a stored constraint.

Keep indexed values stable

Functions used in index expressions must satisfy PostgreSQL’s immutability requirement: their results must depend on their arguments rather than changing external state or the current time. now() cannot be indexed as if it were a stable value forever. Marking a function IMMUTABLE merely to bypass the check does not make the claim true and can leave stored keys inconsistent with later evaluations.

The expression is computed during insertion and applicable updates. Its storage and maintenance cost remain even if the planner never chooses the index. Changing a function’s behavior or normalization policy also requires considering already stored index values; replacing code alone does not recompute them. Plan the rebuild and data validation as part of the semantic change.

When it breaks

An ORM refactor can introduce lower(trim(email)) while the index still stores lower(email). Queries remain correct but may scan more rows. A team can also change collation or a custom function and accidentally rely on old stored keys. Compare the final generated SQL, index definition, relevant function behavior, and target environment when diagnosing the missing path.

Another failure is treating a unique expression index as a harmless performance tweak. Duplicate normalized values can make index creation fail, and future inserts now face a new constraint. Audit and resolve real duplicates without silently deleting business records. For production builds, choose an appropriate index creation procedure and verify its completion; an index name existing in the catalog is not sufficient evidence of a usable finished index.

Check it quickly

Save this as expression.sql and execute psql -X -v ON_ERROR_STOP=1 -f expression.sql in a disposable PostgreSQL 18 database. The ten thousand addresses use the reserved example.test domain, and the table is temporary.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE accounts (id integer, email text NOT NULL);
INSERT INTO accounts SELECT g, 'User'||g||'@example.test' FROM generate_series(1,10000) AS g;
CREATE UNIQUE INDEX account_email_folded ON accounts (lower(email));
ANALYZE accounts;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT id FROM accounts WHERE ((lower(email)))='user42@example.test';
EXPLAIN (COSTS OFF)
SELECT id FROM accounts WHERE lower(trim(email))='user42@example.test';
DO $$ BEGIN
 INSERT INTO accounts VALUES (10001,'USER42@example.test');
 RAISE EXCEPTION 'case-folded duplicate accepted';
EXCEPTION WHEN unique_violation THEN RAISE NOTICE 'folded duplicate rejected';
END $$;
DO $$ BEGIN
 EXECUTE 'CREATE INDEX invalid_clock ON accounts ((now()))';
 RAISE EXCEPTION 'non-immutable index expression accepted';
EXCEPTION WHEN invalid_object_definition THEN RAISE NOTICE 'non-immutable expression rejected';
END $$;
DO $$ BEGIN
 IF (SELECT id FROM accounts WHERE lower(email)='user42@example.test') IS DISTINCT FROM 42 THEN
 RAISE EXCEPTION 'expression lookup failed';
 END IF;
END $$;
ROLLBACK;

The historical PostgreSQL 18.6 single-user check used account_email_folded even with redundant parentheses around lower(email); psql behavior was untested. The trim variant used a sequential scan. Inserting an uppercase duplicate of user forty-two failed the unique expression index, and indexing now() failed the immutability requirement. The final assertion confirms the supported lookup still returns id forty-two.

The exception handlers catch only the expected database errors; unexpected success fails the script. Rollback removes all test objects. These results distinguish harmless syntax from changed meaning without claiming the fixture covers Unicode normalization, every collation, production build locking, or arbitrary expression equivalence. Those require their own concrete data and operations.

Sources

PostgreSQL indexes on expressions, stored computations and unique expressions, checked 2026-10-15.
PostgreSQL CREATE INDEX, immutability and build requirements, checked 2026-10-15.

A partial index stores only the rows that matter

Tue, 27 Apr 2027 00:00:00 +0000

A partial index can keep a frequently queried subset of rows in a smaller structure. An index on job id WHERE status is open is a useful candidate when open jobs are rare and most requests target them. Its availability depends on what the planner can prove about each query, not merely on whether the application happens to pass “open” during one execution.

Index eligibility follows what planning can prove. Generic versus custom changes the available evidence; cost still decides the chosen path.

Prove that excluded rows are unnecessary

The index has no entries for closed jobs, so it cannot by itself answer a query that might need them. PostgreSQL must recognize at planning time that the query condition implies the index predicate. An explicit status = 'open' supplies that fact. A generic parameter status = $1 does not, because the same plan must remain valid when the parameter is closed.

Implication is not always textual identity. PostgreSQL recognizes some simple logical relationships, but it is not a general theorem prover. Standardizing the intended predicate helps prevent refactors from hiding an otherwise usable index. A condition on the indexed id alone cannot establish the status restriction, even if current data or application conventions make the row usually open.

Prepared plans change the evidence available

Prepared statements can use custom plans based on supplied parameter values or generic plans reusable across values. A custom plan for status open can know the needed predicate and consider the partial index. A generic plan with an unknown status cannot safely use that index as the complete answer path. The application’s SQL text has not changed, but the information available during planning has.

This does not mean parameterized queries are incompatible with partial indexes. A query with literal status = 'open' and a parameter only for id retains the predicate in a generic plan. Nor should parameters be replaced by string concatenation: parameter binding remains the correct way to carry untrusted values. Design the query around its real invariant rather than weakening data handling to chase an index.

The default plan policy chooses between custom and generic plans using its own cost considerations. For diagnosis, plan_cache_mode can force either behavior in a session. Such forcing is an experiment, not an instruction to disable generic plans globally. Replanning costs, parameter skew, and execution frequency determine the appropriate tradeoff.

When it breaks

A query can become slower after repeated prepared execution when a reusable plan replaces a parameter-specific one. The symptom may look like random index disappearance. Capture the actual prepared SQL and inspect EXPLAIN EXECUTE with representative values; an ad hoc literal query does not reproduce every prepared-plan condition. Confirm the predicate, plan mode, estimated rows, and runtime work together.

The subset can also stop being small. A backlog that leaves most jobs open reduces the expected size advantage, and status transitions still require index maintenance. Statistics and production-like distributions are needed before claiming lower write cost or better cache residency. A partial index is a workload decision, not a guarantee that the table’s future state will stay favorable.

The predicate limits index entries, not stored table rows or permission to read them. Closed jobs remain in the table and are still available through other access paths. A partial unique index would enforce uniqueness only within its predicate; it cannot replace a global uniqueness rule. Keep query acceleration, integrity, and row access policy separate when reviewing the schema.

Check it quickly

Save this as partial-plan.sql and run psql -X -v ON_ERROR_STOP=1 -f partial-plan.sql in a disposable PostgreSQL 18 database. The temporary table has one hundred open jobs among ten thousand rows, and the index includes only that open subset.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE jobs (id integer, status text NOT NULL);
INSERT INTO jobs SELECT g, CASE WHEN g<=100 THEN 'open' ELSE 'closed' END
FROM generate_series(1,10000) AS g;
CREATE INDEX jobs_open_id ON jobs (id) WHERE status='open';
ANALYZE jobs;
PREPARE chosen(text,integer) AS SELECT id FROM jobs WHERE status=$1 AND id=$2;
SET LOCAL plan_cache_mode=force_generic_plan;
EXPLAIN (COSTS OFF) EXECUTE chosen('open',7);
EXECUTE chosen('open',7);
SET LOCAL plan_cache_mode=force_custom_plan;
EXPLAIN (COSTS OFF) EXECUTE chosen('open',7);
EXECUTE chosen('open',7);
PREPARE always_open(integer) AS SELECT id FROM jobs WHERE status='open' AND id=$1;
SET LOCAL plan_cache_mode=force_generic_plan;
EXPLAIN (COSTS OFF) EXECUTE always_open(7);
EXECUTE always_open(7);
DEALLOCATE chosen;
DEALLOCATE always_open;
ROLLBACK;

In the recorded historical PostgreSQL 18.6 single-user SQL check, the unknown-status generic plan used a sequential scan. The psql client and connection-pool behavior were not executed. Forcing a custom plan for the same prepared statement and open value used jobs_open_id. The second statement, whose status predicate is a literal and only its id is parameterized, also used that partial index under a generic plan. All three executions returned id seven.

These plans demonstrate why a known predicate matters, not a promise that every machine selects the same access path. The explicit SET LOCAL changes end with rollback, and DEALLOCATE removes the prepared statements. The test does not reproduce automatic plan switching under a real connection pool; that requires observing the application’s actual sessions and repeated execution history.

Sources

PostgreSQL partial indexes, predicate implication at planning time, checked 2026-10-15.
PostgreSQL PREPARE, custom and generic execution plans, checked 2026-10-15.

Choose B-tree or hash from the required operations

Sun, 07 Feb 2027 00:00:00 +0000

PostgreSQL’s default B-tree index supports several common query requirements: equality, ranges, ordered access, and uniqueness enforcement. A hash index supports a narrower equality access path. Choosing between them starts with the required operators and constraints, then proceeds to measurement. Saying that hashes are always faster for equality, or that PostgreSQL almost never chooses them, replaces a workload question with an unsupported generalization.

Capabilities before timings

A B-tree retains ordering under its operator class and collation. That can support a bounded range and an ORDER BY without a separate sort, as well as a single-key equality lookup. B-trees can also support IS NULL and IS NOT NULL. An index’s existence still does not force its use: reading a large fraction of a table may favor another plan.

Text patterns have additional requirements. A constant anchored prefix such as LIKE 'abc%' can use an appropriate B-tree access path, but collation and operator class matter. Outside the C locale, a pattern-specific operator class may be necessary. A leading wildcard is not made into a selective ordered range merely by creating a normal B-tree index.

A PostgreSQL hash index stores a four-byte hash code rather than the complete indexed value. It supports equality, lacks key ordering, supports only a single indexed column, and cannot enforce uniqueness. Its smaller entries can help some equality workloads with large values, but heap verification and bucket behavior still contribute to cost. The data structure alone does not establish the winner.

When it breaks

Replacing a B-tree with a hash index can remove useful range or ordered access. It also removes an access path for IS NULL: PostgreSQL hash indexes do not support that predicate. Replacing the index behind a uniqueness requirement is a more serious mistake because a hash index cannot supply the required contract at all.

Null semantics deserve a separate check. A default unique B-tree permits multiple nulls because they are treated as distinct for uniqueness. If the application requires at most one null, use NULLS NOT DISTINCT on the unique constraint or index rather than inferring that policy from the word unique. Supported access operations and enforced data rules are related but different decisions.

Capabilities precede cost comparisons. Default uniqueness permits the two nulls; retained fixture plans are not a speed ranking or fresh client run.

Check it quickly

Run this experiment with PostgreSQL 18 and psql. The temporary table contains twenty thousand integer keys plus two null-key rows. It first has a hash index, then replaces that index with a unique B-tree. Rollback removes the experiment.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE points (key integer,value text);
INSERT INTO points SELECT g,'value-'||g FROM generate_series(1,20000) AS g;
INSERT INTO points VALUES (NULL,'no-key-a'),(NULL,'no-key-b');
CREATE INDEX points_hash ON points USING hash(key);
ANALYZE points;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT value FROM points WHERE key=12345;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT key,value FROM points WHERE key>=12340 AND key<12350 ORDER BY key;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT value FROM points WHERE key IS NULL;
DO $$ BEGIN
 CREATE UNIQUE INDEX points_hash_unique ON points USING hash(key);
 RAISE EXCEPTION 'hash uniqueness unexpectedly supported';
EXCEPTION WHEN feature_not_supported THEN RAISE NOTICE 'hash cannot enforce uniqueness';
END $$;
DROP INDEX points_hash;
CREATE UNIQUE INDEX points_btree ON points(key);
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT value FROM points WHERE key=12345;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT key,value FROM points WHERE key>=12340 AND key<12350 ORDER BY key;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT value FROM points WHERE key IS NULL;
DO $$ BEGIN
 IF (SELECT value FROM points WHERE key=12345) IS DISTINCT FROM 'value-12345' OR
 (SELECT count(*) FROM points WHERE key>=12340 AND key<12350) <> 10 OR
 (SELECT count(*) FROM points WHERE key IS NULL) <> 2 THEN RAISE EXCEPTION 'index changed answers'; END IF;
END $$;
ROLLBACK;

Retained PostgreSQL 18.6 standalone SQL logs, not a rerun in this source review, show that the hash equality query used points_hash and returned value-12345. The ten-key range used a sequential scan plus sort, while the null query used a sequential scan returning two rows. Attempting a unique hash index raised the expected unsupported-feature exception, which the block catches deliberately; any different failure still stops the script.

After replacement, the B-tree supported the equality, bounded ordered range, and null lookup in this fixture. The range returned ten rows and the null lookup still returned two. Building the unique index succeeded despite those nulls, matching PostgreSQL’s default null-distinct uniqueness behavior. The final assertions check answers after B-tree creation; the earlier hash-stage row counts come from EXPLAIN ANALYZE. The retained run omitted the psql directive and did not test the client invocation.

Plan choice and page counts can vary with relation size, cache state, statistics, and configuration. This experiment establishes supported operations and an observed selection, not a benchmark proving one method faster on every machine. The unsupported uniqueness capability does not become supported when costs or data sizes change.

Measure the complete workload

For a real candidate, compare representative equality lookups, range queries, ordering requirements, write activity, storage, and maintenance. Test alternatives in a controlled environment instead of building competing large indexes casually on production. A hash index may earn its storage on a measured equality workload, while the B-tree may still be required for uniqueness or other queries. Keep every retained index tied to an actual access or integrity requirement, and account for its write cost as well as its read benefit.

Sources

PostgreSQL 18: Index types, supported predicates for B-tree and hash indexes, checked 2026-10-15.
PostgreSQL 18: Hash indexes, collision rechecks and bucket maintenance, checked 2026-10-15.
PostgreSQL 18: Constraints, uniqueness and null semantics, checked 2026-10-15.

Index · XB Field Notes

InnoDB covering indexes carry the query's columns

When it breaks

Check it quickly

Pay for coverage deliberately

Sources

Partial indexes and writes

Separate membership from update cost

Eligibility is separate from usefulness

When it breaks

Check it quickly

Sources

Further reading

Hash indexes recheck collisions and maintain changed keys

What lives in a bucket

When it breaks

Check it quickly

Evaluate maintenance with lookup performance

Sources

Postgres JSONB pairs with GIN for flexible queries

Match the operator to the index

Missing and null are different questions

When it breaks

Check it quickly

Sources

A Postgres index lives to answer a query

Start from the requested result

Read the chosen plan honestly

When it breaks

Check it quickly

Sources

A function index precomputes a value at write time

Index the computed meaning

Keep indexed values stable

When it breaks

Check it quickly

Sources

A partial index stores only the rows that matter

Prove that excluded rows are unnecessary

Prepared plans change the evidence available

When it breaks

Check it quickly

Sources

Choose B-tree or hash from the required operations

Capabilities before timings

When it breaks

Check it quickly

Measure the complete workload

Sources