Query · XB Field Notes

InnoDB covering indexes carry the query's columns

Fri, 03 Mar 2028 00:00:00 +0000

A covering index contains the columns needed to evaluate a particular query and return its requested values. In InnoDB, secondary-index records also carry the primary-key columns, so a query can be covered without explicitly listing the primary key again in the secondary-index definition. Coverage is a relationship between one query and an index, not a permanent property that applies to every query against the table.

When it breaks

Adding a projected column can remove coverage. An index that supplies an identifier and timestamp may not contain the large payload that a later application change begins selecting. The optimizer can still use that same index to locate matching rows, then obtain missing values from the clustered records. Index use and covering access are different facts.

Coverage also does not guarantee a selective access path. A wide scan can be fully covered while examining many entries. Column order, predicates, sorting requirements, and row estimates still matter. Do not interpret Using index as evidence that a query is fast, or confuse it with Using index condition, which describes index condition pushdown rather than complete projection coverage.

InnoDB visibility checks qualify the usual index-only explanation. Secondary records do not contain the same hidden transaction metadata as clustered records. Delete-marked secondary entries or pages changed by newer transactions can require clustered-record lookups to obtain the correct visible version. A covering plan is therefore not a universal promise of zero clustered access under concurrent MVCC activity.

Projection coverage and MVCC visibility are separate. Historical MySQL plans lack a recovered raw log; neither coverage nor pushdown proves speed.

Check it quickly

The unchanged fixture was not rerun in this source review. The retained review records say MySQL 8.4 was unavailable, and no raw MySQL run log was recovered in the retained validation files; the reported plans remain attributed historical observations.

Use a fresh MySQL 8.4 instance and a client session with permission to create a disposable schema. The recorded run used MySQL 8.4.11. Ensure xbcatm_index_lab is unused, stop on unexpected SQL errors, and keep the same session so its selected database remains active. Create ten thousand rows, with one hundred paid orders and a deliberately wider payload:

CREATE DATABASE xbcatm_index_lab;
USE xbcatm_index_lab;
CREATE TABLE orders (
 id INT PRIMARY KEY,
 status VARCHAR(10) NOT NULL,
 created_at DATETIME NOT NULL,
 payload VARCHAR(1000) NOT NULL,
 INDEX by_status_created (status,created_at)
) ENGINE=InnoDB;
INSERT INTO orders
WITH digits(n) AS (
 SELECT 0 UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3
 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6
 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9
)
SELECT number+1,IF(MOD(number+1,100)=0,'paid','open'),
 TIMESTAMP '2026-01-01 00:00:00' + INTERVAL (number+1) SECOND,
 REPEAT('x',640)
FROM (SELECT a.n+10*b.n+100*c.n+1000*d.n AS number
 FROM digits a CROSS JOIN digits b CROSS JOIN digits c CROSS JOIN digits d) AS numbered;
ANALYZE TABLE orders;
SELECT COUNT(*),SUM(status='paid') FROM orders;

Manually verify that the final counts are ten thousand and one hundred; these SELECT statements print results without automated assertions. The secondary index contains status and creation time explicitly; InnoDB supplies the primary identifier implicitly. Inspect and run a query that needs only those values:

EXPLAIN SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded tabular plan reports by_status_created and Using index. The analyzed plan identifies a covering index lookup and returns twenty rows. Result identifiers are one hundred, two hundred, and so on through two thousand, ordered by the timestamp and identifier. These are checked results on populated data, not a plan inferred from an empty schema.

Now request the payload as well:

EXPLAIN SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded plan still uses by_status_created, but loses Using index; the analyzed plan describes an index lookup without the covering qualification. This isolates the effect of changing the requested columns while keeping the filter, ordering, and index unchanged. The diagnostic execution does run the SELECT, so account for that when applying EXPLAIN ANALYZE to expensive real queries.

On success, remove the schema created for this exercise. MySQL DDL can commit implicitly, so an error or interruption does not roll the setup back; inspect and remove only this fixture schema afterward:

DROP DATABASE xbcatm_index_lab;

The timings from this tiny warm experiment are not a throughput or latency benchmark. Scheduling, caching, and instrumentation can outweigh the difference between two twenty-row reads. The verified conclusion concerns access shape and returned values, not a promised percentage reduction in physical I/O.

Pay for coverage deliberately

Appending columns to a secondary index consumes storage and buffer-pool capacity and increases maintenance when those indexed values change. Long primary keys enlarge secondary entries too. Cover the important narrow query rather than automatically copying a large payload into every index. Keep projected columns explicit so later application changes make their access requirements visible in review.

Compare representative plans, examined rows, foreground latency, and write cost before retaining a new index. Test with realistic selectivity and concurrent activity; a quiet fixture does not measure MVCC fallback frequency or replica maintenance cost. An index should earn its place through the workload it serves, not through the appearance of a desirable label in one explanation.

Sources

MySQL 8.4 clustered and secondary indexes, primary-key columns carried by secondary records, checked 2026-09-12.
MySQL 8.4 EXPLAIN output, index-use and covering-plan indicators, checked 2026-09-12.
MySQL 8.4 InnoDB multiversioning, clustered lookups required for secondary-index visibility checks, checked 2026-09-12.

Partition pruning skips whole partitions a query cannot need

Sun, 13 Feb 2028 00:00:00 +0000

Partition pruning excludes entire partitions that cannot contain matching rows. It is different from using an index to find rows within a remaining partition, and it is not a page-level cache trick. PostgreSQL can perform this exclusion while planning a query or later when parameter values become available during execution.

Prove which partitions cannot match

Declarative partitions have bounds derived from their partition key. A predicate restricting a date key to October can prove that September and November are irrelevant. Pruning uses those bounds, not an estimate that another partition probably contains few matches. Statistics still influence other planning decisions, but an inaccurate row estimate does not make partition membership a guess.

Indexes are not required for pruning. After excluding irrelevant partitions, PostgreSQL may choose a sequential scan of a remaining partition because most of its rows are needed. An Index Scan label is therefore not the test for successful pruning. Inspect the relation names and plan structure, then separately evaluate how each remaining relation is accessed.

A restriction must be usable with the partition key and its comparison semantics. Converting a date column to text can hide the direct bound relationship even when a particular string comparison happens to select the same rows. Use typed parameters and predicates that describe the intended interval directly. Do not assume that arbitrary expressions around a partition key preserve pruning or even preserve the original date semantics.

Parameters can prune during execution

A constant can allow planning-time pruning, leaving only the relevant child relation in the plan. A prepared statement’s generic plan cannot know every future parameter value in advance. That does not mean the statement must scan every partition: execution-time pruning can use values known when the plan initializes.

EXPLAIN ANALYZE reports Subplans Removed for partitions discarded during initialization. Other cases, such as parameters changing on the inner side of a nested-loop join, can prune during execution itself. Inspect loop counts and whether subplans actually ran when reasoning about those cases. A never-executed node alone can have other causes, such as an empty outer input, so interpret it in its full plan context.

Avoid turning these observations into a blanket rule that every nonconstant expression disables pruning. Parameter availability and expression support determine when exclusion is possible. Custom and generic prepared plans can differ while returning the same answer. In particular, a generic parameter may support runtime partition pruning while failing the planning-time implication test for a partial index. Those two optimizations have different proof boundaries. The useful question is which partitions were accessed for the actual values used by the workload.

When it breaks

A filter on an unrelated column may leave every partition relevant. Casting the key into an incompatible expression can do the same. Check the exact predicate and enable_partition_pruning setting before changing the partition layout. Manually inventing a contradictory CHECK constraint is not a repair; ordinary local declarative partitions enforce their bounds, and constraint exclusion is a separate optimization using CHECK constraints.

Pruning also is not a promise of no locking or negligible planning cost. Partitions removed during execution initialization are still locked at the start of execution. Large partition hierarchies can consume planning time and per-session metadata memory even when a query eventually scans only a few children. Choose the hierarchy for realistic query and maintenance patterns rather than maximizing partition count.

Finally, pruning two partitions can still leave thousands of irrelevant rows inside the third. A matching local index may help that second stage. Returning only a hundred rows does not prove that only a hundred rows were examined, and buffer hits are not the same as physical disk reads.

Partition-bound proof, initialization pruning and local scan work are separate. Historical plan observations are scoped to the fixture; exclusion does not establish no locking.

Check it quickly

Save this as partition-pruning.sql and run psql -X -v ON_ERROR_STOP=1 -f partition-pruning.sql in a disposable PostgreSQL 18 database. The script enables pruning within its transaction and uses a generic prepared plan to expose initialization-time pruning. It creates no indexes on the temporary partitions.

\set ON_ERROR_STOP on
BEGIN;
SET LOCAL datestyle='ISO, YMD';
SET LOCAL enable_partition_pruning=on;
CREATE TEMP TABLE observations (day date,id integer) PARTITION BY RANGE(day);
CREATE TEMP TABLE obs_sep PARTITION OF observations FOR VALUES FROM ('2026-09-01') TO ('2026-10-01');
CREATE TEMP TABLE obs_oct PARTITION OF observations FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');
CREATE TEMP TABLE obs_nov PARTITION OF observations FOR VALUES FROM ('2026-11-01') TO ('2026-12-01');
INSERT INTO observations SELECT DATE '2026-09-01'+g%90,g FROM generate_series(0,8999) AS g;
ANALYZE observations;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT count(*) FROM observations WHERE day>=DATE '2026-10-01' AND day<DATE '2026-11-01';
SET LOCAL plan_cache_mode=force_generic_plan;
PREPARE by_day(date) AS SELECT count(*) FROM observations WHERE day=$1;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) EXECUTE by_day(DATE '2026-10-10');
EXECUTE by_day(DATE '2026-10-10');
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT count(*) FROM observations WHERE day::text='2026-10-10';
DO $$ BEGIN
 IF (SELECT count(*) FROM observations WHERE day=DATE '2026-10-10') <> 100 OR
 (SELECT count(*) FROM observations WHERE day>=DATE '2026-10-01' AND day<DATE '2026-11-01') <> 3100 THEN
 RAISE EXCEPTION 'unexpected pruned result';
 END IF;
END $$;
DEALLOCATE by_day;
ROLLBACK;

Retained logs show that the PostgreSQL 18.6 standalone SQL adaptation scanned obs_oct alone for October, reading 3,100 rows. This source review did not repeat that run. No psql transport or concurrent partition-lock test was performed. The generic date lookup reported Subplans Removed: 2 and returned 100 rows, but still filtered 3,000 other rows within October. This demonstrates both successful partition exclusion and remaining local work.

The text-cast predicate scanned all three children and returned the same 100 rows under the explicitly chosen ISO DateStyle. Assertions verify the typed query’s counts, and DEALLOCATE plus rollback clean up the prepared statement and temporary tables. These results establish the predicate and pruning relationship for this fixture; measure representative plans separately before choosing an index or changing production partition granularity.

Sources

PostgreSQL table partitioning, planning and execution pruning, bounds, and locking, checked 2026-10-15.
PostgreSQL PREPARE, generic and custom plans and parameter handling, checked 2026-10-15.
PostgreSQL partial indexes, planning-time predicate implication, checked 2026-10-15.

A Common Table Expression names a query you can reuse

Sun, 19 Dec 2027 00:00:00 +0000

A common table expression gives a query result a name within one SQL statement. It can make a complex report readable by naming populations and intermediate calculations before the final SELECT. That name is not a persistent table, and it does not by itself promise a cached result, a particular scan count, or a sequence of procedural steps.

Scope first, execution second

A WITH clause belongs to its statement. A later statement cannot reuse its CTE name, and there is no schema object to index or query after execution. Use a temporary table, view, or another appropriate database object if results or definitions must outlive that statement. A CTE is useful when its scope and its named result match the work being expressed.

In modern PostgreSQL, a nonrecursive SELECT CTE without volatile functions can be folded into its parent query. By default, this happens when it is referenced once. Folding lets outer restrictions participate in optimization, so an outer primary-key lookup can reach an index on the underlying table instead of filtering a separately produced intermediate result.

When the same eligible CTE is referenced more than once, PostgreSQL normally materializes it instead. That can avoid repeating expensive calculations, but it can also prevent a restrictive condition at one use from being pushed into the shared computation. This distinction has applied since PostgreSQL 12; assuming that every WITH is always an optimization fence describes older behavior, not the current default.

Materialization is a tradeoff

MATERIALIZED requests separate computation. NOT MATERIALIZED permits folding even with repeated references, potentially repeating work while allowing each use to access only the rows it needs. PostgreSQL ignores NOT MATERIALIZED for recursive or non-side-effect-free CTEs; it does not authorize duplicating arbitrary writes or volatile operations.

A materialized SELECT CTE is not necessarily evaluated to exhaustion. Its consumers may demand only part of its result, though sorts, joins, or other operations can require more. Materialization also does not create a reusable indexed table for later statements. Inspect the plan and measure the actual workload before forcing a boundary merely because the word sounds like a cache.

The useful comparison is total work: repeated selective index probes may be cheaper than building and scanning a large shared result, while repeating an expensive calculation may be worse. A readable name and an efficient execution plan are compatible, but neither implies the other. Keep the business predicate in one authoritative definition and choose execution controls only when the evidence supports them.

When it breaks

Data-modifying CTEs follow different rules. INSERT, UPDATE, DELETE, or PostgreSQL 18 MERGE inside WITH executes once and to completion even if its output is not consumed. The named relation is the RETURNING output, not a freshly rescanned target table. A modifying CTE without RETURNING still performs its modification but cannot supply rows to a later reference.

The modifying substatements and main statement use the same snapshot, and their actual modification order is not a procedural top-to-bottom guarantee. Pass changed rows through RETURNING when another part of the statement needs them. Avoid attempting to modify the same row twice in one statement and expecting textual order to decide the winner. A plain SELECT-looking outer query does not make a data-modifying WITH read-only.

A CTE name has statement scope. Static illustration of the article’s example and its stated limits; no SQL was executed for this graphic.

Check it quickly

Save this as cte.sql and run psql -X -v ON_ERROR_STOP=1 -f cte.sql in a disposable PostgreSQL 18 database. It compares four read plans, then moves two temporary rows using DELETE RETURNING and INSERT. The final rollback removes the fixture and reverses its changes.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE items (id integer PRIMARY KEY, value text);
INSERT INTO items SELECT g,'item-'||g FROM generate_series(1,10000) AS g;
ANALYZE items;
EXPLAIN (COSTS OFF)
WITH source AS (SELECT * FROM items) SELECT * FROM source WHERE id=7;
EXPLAIN (COSTS OFF)
WITH source AS MATERIALIZED (SELECT * FROM items) SELECT * FROM source WHERE id=7;
EXPLAIN (COSTS OFF)
WITH source AS (SELECT * FROM items)
SELECT a.id FROM source a JOIN source b USING(id) WHERE a.id=7;
EXPLAIN (COSTS OFF)
WITH source AS NOT MATERIALIZED (SELECT * FROM items)
SELECT a.id FROM source a JOIN source b USING(id) WHERE a.id=7;
CREATE TEMP TABLE moved (id integer,value text);
WITH removed AS (DELETE FROM items WHERE id<=2 RETURNING *)
INSERT INTO moved SELECT * FROM removed;
SELECT (SELECT count(*) FROM items) AS remaining,(SELECT count(*) FROM moved) AS moved;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM moved) IS DISTINCT FROM ARRAY[1,2] THEN
 RAISE EXCEPTION 'RETURNING handoff failed';
 END IF;
 IF (SELECT count(*) FROM items) <> 9998 THEN RAISE EXCEPTION 'DELETE did not execute'; END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, the single-reference default plan used items_pkey directly. Explicit MATERIALIZED introduced a CTE Scan over a planned sequential scan. The default double reference also shared a materialized CTE, while NOT MATERIALIZED produced two index-based accesses. These plain EXPLAIN statements show planned structure; they do not measure execution time or prove a production performance improvement.

The modifying statement actually executed and left 9,998 rows in items and two in moved. Assertions verify that the transferred identifiers are exactly one and two and that the source rows were deleted. RETURNING provides the handoff between operations. Changing the outer statement’s demand is not a safe method for suppressing a modifying CTE; keep such experiments transactional and inspect all WITH bodies before executing unfamiliar SQL.

Sources

PostgreSQL WITH queries, folding, materialization, and data-modifying CTEs, checked 2026-10-15.
PostgreSQL SELECT reference, WITH scope and MATERIALIZED controls, checked 2026-10-15.

EXPLAIN plans reveal the route of a query

Fri, 21 May 2027 00:00:00 +0000

EXPLAIN shows the execution plan selected for a statement. It helps locate expensive work and mistaken estimates, but a slow request can also involve locks, network transfer, or application behavior outside the plan. Use plan evidence to test a specific hypothesis rather than assuming that every performance problem requires a different scan or another index.

Read the tree without double-counting

In PostgreSQL, leaves commonly scan relations while upper nodes join, sort, aggregate, or modify rows. Upper nodes do real work too. Their costs include child costs, so adding all displayed costs double-counts work. The rows estimate is the number a node emits, not necessarily how many rows it examines. Filters can explain a large difference between input and output without proving a planning mistake.

EXPLAIN alone reports estimates. EXPLAIN ANALYZE actually executes the statement and adds observations. Actual rows and times are averages per execution when loops exceeds one; interpret them together with loops. Compare estimates with corresponding observed output, not with unrelated table size. Buffer counts include child activity and are not counts of unique pages, so they also require careful interpretation.

Statistics influence a choice

Row counts, distributions, correlations, query structure, and cost settings influence planning. ANALYZE refreshes statistics, but a large estimate error does not uniquely diagnose stale statistics. Sampling, correlated predicates, missing extended statistics, and parameter selectivity can also matter. Refresh the relevant statistics when appropriate, then check whether the mismatch actually improved.

The planner does not simply watch the latest observed cache-hit ratio and automatically select a new route. Settings such as effective_cache_size express assumptions about caching; actual warm-cache conditions can change execution time even with the same plan. Record data size, parameters, relevant settings, and measurement conditions when comparing results. A sequential scan can be the best choice for a large fraction of a table.

When it breaks

Prepared statements need version-specific interpretation. PostgreSQL’s automatic choice can compare custom plans with a generic plan; it does not simply cache the first parameter’s plan for all later values. Use EXPLAIN EXECUTE with representative values and inspect whether parameters appear as constants or placeholders. Different database products have different caching rules, so avoid importing one product’s explanation into another.

Functions, casts, and collations can prevent a particular index from matching the predicate, but do not universally force a full scan. An expression index may support lower(column), and some cross-type operators remain indexable. PostgreSQL can reject an unsupported varchar-to-integer comparison instead of quietly converting it. Inspect the actual operator, types, and index condition before deciding on a rewrite.

EXPLAIN ANALYZE is not harmless inspection. Writes, triggers, locks, and called functions can have effects. A transaction rollback does not undo every possible effect, such as consumed sequence values or external actions. Use controlled data and an appropriate test environment; even a SELECT may invoke side-effecting functions. Budget the load and avoid treating a diagnostic command as automatically safe for production.

Confirm an improvement, not merely a different shape

Capture comparable before-and-after results. A changed plan can be slower, and the same plan shape can become faster through less work, different data, or improved execution conditions. Inspect elapsed time, output correctness, row estimates, buffers, loops, and spills together. Cost units are not milliseconds, and a low estimated cost is not an operational latency guarantee.

All bars use the same relative-cost scale. The reversal comes from a toy formula and supplied selectivity, not measured database execution. Model not rerun.

Check it quickly

Save the following as plan_route_check.py and run python3 plan_route_check.py. It uses only the standard library to model two candidate paths under a deliberately wrong selectivity estimate. It neither invokes PostgreSQL nor reproduces its optimizer. The fixture’s stale input is known by construction; the warning asks for investigation rather than diagnosing every real estimate error as stale statistics.

#!/usr/bin/env python3
"""Toy planner model: estimated cost picks a route, actual rows judge it.

Stdlib only. It simulates stale statistics: the planner believes 500 rows
match, while 600,000 really do, and checks that the estimate-error alarm fires.
"""

from __future__ import annotations


def seq_scan_cost(pages: float, rows: float) -> float:
 return pages * 1.0 + rows * 0.01


def index_scan_cost(selectivity_est: float, pages: float, rows: float) -> float:
 heap_pages = selectivity_est * pages
 return 4.0 + heap_pages * 4.0 + selectivity_est * rows * 0.01


def main() -> None:
 pages, rows = 8_000.0, 1_000_000.0
 stale_selectivity = 0.0005 # planner belief: 500 rows
 true_selectivity = 0.60 # reality: 600,000 rows
 est_rows = stale_selectivity * rows
 actual_rows = true_selectivity * rows

 seq = seq_scan_cost(pages, rows)
 idx_est = index_scan_cost(stale_selectivity, pages, rows)
 idx_true = index_scan_cost(true_selectivity, pages, rows)
 chosen = "index scan" if idx_est < seq else "seq scan"
 print(f"estimated rows: {est_rows:.0f}; actual rows: {actual_rows:.0f}")
 print(f"estimated cost: seq={seq:.1f} index={idx_est:.1f} -> planner picks {chosen}")
 print(f"true cost at actual selectivity: seq={seq:.1f} index={idx_true:.1f}")

 error = abs(actual_rows - est_rows) / actual_rows
 print(f"estimate error: {error:.1%}")
 warned = False
 if error > 0.50:
 warned = True
 print("WARNING: row estimate off by >50% - inspect statistics, predicates, and parameter selectivity.")
 if chosen == "index scan" and idx_true > seq:
 warned = True
 print("WARNING: chosen route loses at actual selectivity - check EXPLAIN ANALYZE.")
 assert warned, "expected a stale-estimate warning for this fixture"
 assert est_rows < actual_rows / 10, "fixture must keep the estimate an order of magnitude low"
 print("PASS: planner picked the stale route and the estimate-error alarm fired")


if __name__ == "__main__":
 main()

The model chooses an index scan at estimated cost 25 versus 18,000 for the sequential path. At the supplied actual selectivity, its index cost becomes 25,204. It prints an estimate error of 99.9% and a PASS line after the warnings. Both costs are outputs of these simple formulas, not measured database execution times.

This confirms the arithmetic and the deliberately constructed counterexample. It does not validate a particular PostgreSQL plan, justify adding an index, or prove that ANALYZE repairs a production query. Repeat the investigation with actual plans and controlled execution evidence before changing a real workload.

Sources

PostgreSQL: EXPLAIN, generating the plan, checked 2026-10-15.
PostgreSQL: using EXPLAIN, interpreting the output, checked 2026-10-15.
PostgreSQL: planner/optimizer, how routes are chosen, checked 2026-10-15.
MySQL: EXPLAIN statement, plan format and output columns, checked 2026-10-15.
PostgreSQL PREPARE, generic and custom plan selection, checked 2026-10-15.
PostgreSQL expression indexes, indexed expressions including lower(column), checked 2026-10-15.

Query · XB Field Notes

InnoDB covering indexes carry the query's columns

When it breaks

Check it quickly

Pay for coverage deliberately

Sources

Partition pruning skips whole partitions a query cannot need

Prove which partitions cannot match

Parameters can prune during execution

When it breaks

Check it quickly

Sources

A Common Table Expression names a query you can reuse

Scope first, execution second

Materialization is a tradeoff

When it breaks

Check it quickly

Sources

EXPLAIN plans reveal the route of a query

Read the tree without double-counting

Statistics influence a choice

When it breaks

Confirm an improvement, not merely a different shape

Check it quickly

Further reading

Sources