Schema · XB Field Notes

Protobuf evolution preserves meaning only within a defined contract

Sat, 26 Feb 2028 00:00:00 +0000

A gRPC rollout can leave old producers, new consumers, and intermediate services running together. Protobuf helps them exchange binary messages by identifying fields with numbers and wire types. That tolerance is narrower than application compatibility: a message can parse successfully while a consumer misunderstands its values, ignores a new instruction, or loses the distinction between absent and explicitly zero.

Separate numbers, meaning, and bytes

A binary tag combines the field number with its wire type; the field name is not transmitted there. Keeping the number and type while renaming a field can preserve binary decoding. It still changes generated accessors and may change JSON or text interfaces. A gRPC method name or service name belongs to another contract and is not protected by a field-number rule.

Never reuse a retired number for another meaning. Different declared types can share a wire type: signed and unsigned integers both use varints, yet a negative signed value can become a large unsigned value. A successful parse is therefore insufficient validation. Reserve retired numbers and names in the schema so later declarations cannot accidentally reclaim them; reserved numbers are unavailable, not placeholders to activate later.

Unknown fields can survive a parse-and-serialize binary relay. However, copying only known fields or converting through JSON can discard them. Preservation of decoded information also does not promise byte-identical serialization. Field order and encoding choices can differ, including across implementations. Compare required values and presence in compatibility tests; do not use ordinary reserialization as a canonical digest algorithm.

Presence carries a separate bit of meaning

In proto3, singular scalar fields without optional generally use implicit presence. Their default value is indistinguishable from absence through the normal API. Explicit optional scalars track whether a value was supplied, including zero. Message fields and oneof members already have presence; repeated and map fields do not distinguish an empty collection from an absent one.

Changing a scalar between implicit and explicit presence can remain binary-readable while changing round-trip semantics. An older reader that knows the field with implicit presence can parse explicit zero and then omit it when forwarding. Adding optional at one endpoint cannot force every intermediary to preserve that distinction. Test the entire path that actually modifies or forwards the message.

When it breaks

A new consumer may require a field that old writers never supplied. Although proto3 has no required declaration, application validation can create exactly this rolling-upgrade failure. Introduce an agreed fallback or sequence the rollout before enforcing the requirement. Optional syntax alone does not define a sensible default, authorize an operation, or make an old service implement a new feature.

Also test persisted messages, delayed jobs, and rollback versions. Their useful lifetime can exceed the deployment window. A schema check against only the immediately previous release misses older records that can still reappear. Keep retired numbers reserved even when current traffic no longer uses them.

Both relayed results return zero, but the unknown-field relay retains explicit presence while the known implicit-presence relay loses it. The historical Python fixture passed within its local scope; it did not test reserved declarations, real gRPC calls or cross-language clients. No example was rerun for this drawing.

Check it quickly

Run the complete block with Python and Google’s protobuf package 7.36.2 available. It was executed with Python 3.12.14. Separate descriptor pools create real old and new message classes without a compiler, filesystem fixture, gRPC server, or network connection. This checks the Python binary runtime, not cross-language interoperability or generated-client compatibility.

from google.protobuf import descriptor_pb2 as schema, descriptor_pool, message_factory


def message(fields, explicit=()):
 file = schema.FileDescriptorProto(name='probe.proto', package='probe', syntax='proto3')
 record = file.message_type.add(name='Record')
 for name, number, kind in fields:
 field = record.field.add(name=name, number=number, type=kind, label=1)
 if name in explicit:
 field.oneof_index = len(record.oneof_decl)
 record.oneof_decl.add(name='_' + name)
 field.proto3_optional = True
 pool = descriptor_pool.DescriptorPool()
 pool.Add(file)
 return message_factory.GetMessageClass(pool.FindMessageTypeByName('probe.Record'))


INT32 = schema.FieldDescriptorProto.TYPE_INT32
UINT32 = schema.FieldDescriptorProto.TYPE_UINT32
STRING = schema.FieldDescriptorProto.TYPE_STRING
Old = message([('id', 1, INT32), ('name', 2, STRING)])
New = message([('event_id', 1, INT32), ('name', 2, STRING), ('count', 3, INT32)], ('count',))
Implicit = message([('event_id', 1, INT32), ('name', 2, STRING), ('count', 3, INT32)])

old = Old(id=7, name='alpha')
new_reader = New.FromString(old.SerializeToString())
assert new_reader.event_id == 7 and not new_reader.HasField('count')
# The added explicit zero is unknown to Old but retained in binary forwarding.
new_writer = New(event_id=7, name='alpha', count=0)
relay = Old.FromString(new_writer.SerializeToString())
returned = New.FromString(relay.SerializeToString())
assert returned.event_id == 7 and returned.HasField('count') and returned.count == 0
print('old/new binary values survive; unknown explicit zero survives the old relay')

# An implicit-presence reader knows field 3, so zero is omitted on serialization.
implicit_relay = Implicit.FromString(new_writer.SerializeToString())
lost_presence = New.FromString(implicit_relay.SerializeToString())
assert lost_presence.count == 0 and not lost_presence.HasField('count')
print('an implicit-presence relay preserves zero but loses its explicit presence')

Signed = message([('value', 1, INT32)])
Unsigned = message([('value', 1, UINT32)])
assert Unsigned.FromString(Signed(value=-1).SerializeToString()).value == 4294967295
# Distinct field order is legal; compare decoded meaning, not byte equality.
reordered = b'\x18\x00' + old.SerializeToString()
assert New.FromString(reordered) == new_writer
print('reusing an integer tag changes meaning; legal field orders decode equivalently')

The first output confirms that an old writer remains readable and that an unknown explicit zero survives the old binary relay. The next output exposes a different result when the relay knows that field with implicit presence: its value remains zero, but HasField becomes false after forwarding. These are different compatibility questions, not contradictory observations.

The final assertions decode negative one as 4294967295 under an incompatible unsigned interpretation and accept a legal alternative field order. They demonstrate why wire readability does not establish semantic equivalence or canonical bytes. The descriptor exercise does not compile reserved declarations; compiler checks and real gRPC request/response behavior remain separate rollout tests.

Sources

Protobuf: encoding, tags, varints and serialization ordering, checked 2026-10-15.
Protobuf: proto3 language guide, binary changes, reserved fields and unknown-field retention, checked 2026-10-15.
Protobuf: field presence, implicit and explicit presence across relays, checked 2026-10-15.
Protobuf: dos and don’ts, retired numbers and compatibility discipline, checked 2026-10-15.

Foreign keys keep references honest

Tue, 25 May 2027 00:00:00 +0000

A foreign key makes a relationship enforceable during ordinary database writes. A task can reference an existing project, and the database can reject an unknown project identifier even when an import or a new application code path forgets to check it. The constraint defines referential integrity; it does not decide whether the selected project is the correct one for the business operation.

State the relationship precisely

A referencing key points to eligible unique columns in the parent table, commonly its primary key. Several child rows may reference the same parent, so a foreign key alone describes neither one-to-one cardinality nor a requirement that every parent have children. Add the separate uniqueness or other rules that the actual relationship requires instead of inferring them from the reference.

Nullability is an independent decision. A single-column foreign key normally permits SQL NULL, representing no reference. Add NOT NULL when every child must have a parent. For composite keys, PostgreSQL’s default MATCH SIMPLE allows a row to avoid matching when any referencing component is null; MATCH FULL instead requires all components to be null or all to match. A partially filled composite reference is therefore not automatically invalid under the default.

Scope belongs in the key too. If identifiers are only unique within a tenant, reference a suitable tenant-and-identifier key rather than assuming a matching identifier in another tenant is acceptable. Even a correctly scoped foreign key is not an authorization check: an existing project can still be the wrong project for the current user or workflow.

Parent deletion needs a policy

NO ACTION is PostgreSQL’s default delete action. With the normal immediate checking mode, deleting a referenced parent fails unless the resulting statement satisfies the constraint. A deferrable NO ACTION constraint can postpone the check; RESTRICT does not permit that deferral of the delete restriction. The two names should not be treated as unconditional synonyms.

CASCADE removes referencing child rows when the parent is deleted. It fits a component that has no independent lifecycle, but its reach can extend through several relationships. SET NULL retains the child while clearing its reference, which requires compatible nullability and other constraints. SET DEFAULT also must leave a valid reference; choosing a default does not waive referential integrity.

ON UPDATE separately describes what happens when referenced key values change. An update cascade copies the new key into referencing rows; it is not a request to overwrite unrelated child attributes. Choose actions according to ownership and retention requirements. Replacing a restrictive rule with cascade merely to silence an error can turn a useful integrity failure into unintended data deletion.

When it breaks

An application-side existence query can race with another transaction deleting the parent. The database constraint coordinates the actual write, so keep it as the authoritative relationship check rather than relying only on the earlier read. Privileged operations that disable enforcement, or migrations that have not validated old rows, are separate states and cannot be assumed to satisfy a fully enforced, validated constraint.

The write cost is also workload-dependent. Parent deletion can require finding many children, taking locks, and executing cascades. The referenced key must already have an eligible primary/unique constraint or suitable unique index; adding a foreign key does not create an index on the child reference columns. Review those access paths and the size of the deletion graph. A declared relationship does not guarantee join elimination or make every join inexpensive.

The first parent deletion fails atomically. Removing the blocking task allows CASCADE and SET NULL to apply their distinct lifecycles. Earlier source outcomes; no new database run.

Check it quickly

Save this as foreign-key-actions.sql and run psql -X -v ON_ERROR_STOP=1 -f foreign-key-actions.sql in a disposable PostgreSQL 18 database. The fixture uses temporary tables for tasks, project-owned files, and retained history, with different delete actions. Rollback removes all test objects and changes.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE projects (id integer PRIMARY KEY);
CREATE TEMP TABLE tasks (id integer PRIMARY KEY,project_id integer REFERENCES projects ON DELETE NO ACTION);
CREATE TEMP TABLE files (id integer PRIMARY KEY,project_id integer REFERENCES projects ON DELETE CASCADE);
CREATE TEMP TABLE audit_history (id integer PRIMARY KEY,project_id integer REFERENCES projects ON DELETE SET NULL);
INSERT INTO projects VALUES (1),(2);
INSERT INTO tasks VALUES (10,1),(11,NULL);
INSERT INTO files VALUES (20,1);
INSERT INTO audit_history VALUES (30,1);
DO $$ BEGIN
 INSERT INTO tasks VALUES (12,999);
 RAISE EXCEPTION 'dangling reference accepted';
EXCEPTION WHEN foreign_key_violation THEN RAISE NOTICE 'missing project rejected';
END $$;
DO $$ BEGIN
 DELETE FROM projects WHERE id=1;
 RAISE EXCEPTION 'referenced project deleted';
EXCEPTION WHEN foreign_key_violation THEN RAISE NOTICE 'NO ACTION blocked parent deletion';
END $$;
DO $$ BEGIN
 IF (SELECT count(*) FROM files) <> 1 OR
 (SELECT project_id FROM audit_history WHERE id=30) IS DISTINCT FROM 1 THEN
 RAISE EXCEPTION 'failed deletion left partial side effects';
 END IF;
END $$;
DELETE FROM tasks WHERE id=10;
DELETE FROM projects WHERE id=1;
SELECT * FROM tasks;
SELECT * FROM audit_history;
SELECT count(*) AS remaining_files FROM files;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM projects) IS DISTINCT FROM ARRAY[2] THEN
 RAISE EXCEPTION 'wrong surviving project';
 END IF;
 IF (SELECT count(*) FROM files) <> 0 OR
 (SELECT count(*) FROM audit_history WHERE id=30 AND project_id IS NULL) <> 1 OR
 (SELECT count(*) FROM tasks WHERE id=11 AND project_id IS NULL) <> 1 THEN
 RAISE EXCEPTION 'wrong delete action or null handling';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 rejected the task pointing to project 999, while accepting task 11 with a null reference. The first deletion of project one failed because task 10 still referenced it. That failed statement did not partially remove the other dependent data.

After explicitly removing task 10, the parent deletion succeeded. Its file disappeared through CASCADE, history row 30 remained with a null project, and the unrelated null-reference task remained. Project two survived without any children. Assertions verify these distinct outcomes, showing that “references stay valid” and “children must be deleted” are different contracts. Use the same lifecycle examples when reviewing schema changes with application owners.

Sources

PostgreSQL constraints, referential integrity, nulls, actions, and indexing, checked 2026-09-10.
PostgreSQL CREATE TABLE, foreign-key matching and deferrability syntax, checked 2026-09-10.

Zero-downtime migrations change schema while old code runs

Tue, 03 Nov 2026 00:00:00 +0000

A zero-downtime migration is a compatibility and availability goal, not a guarantee attached to small statements. Old and new application versions may overlap while data is changing. Plan the schema, writer behavior, backfill, reader cutover, and recovery together. Dropping information is not automatically reversible, even when the preceding steps were additive.

Lock acquisition, row conflicts, resource usage, and application assumptions all matter. A metadata-only change can wait behind another transaction; a backfill can exhaust I/O or replication capacity without holding an exclusive table lock.

Expand, backfill, contract

Expand introduces a representation that existing readers and writers can tolerate. Deploy compatible writers before relying on new data, backfill existing rows, verify the invariant, switch readers, and contract only after old binaries, background jobs, and rollback requirements no longer need the old shape. Adding a nullable column is often useful, but still needs a lock and compatibility checks.

Dual writing must define failure and ordering behavior. Two columns updated in one database transaction can remain consistent; separate writes can partially succeed. Old writers that still update only the old column can invalidate an already backfilled value. Retire them or provide a tested synchronization mechanism before declaring convergence. A rollback is safe only while its required representation remains accurate.

Use restartable batches and a stable progress key. Prevent stale backfill data from overwriting a concurrent application update, for example through a transactionally checked version or a suitable conditional update. A null-only predicate is sufficient only for the intended invariant and writer behavior. Recheck skipped or concurrently changed rows and validate after writers converge. Batching limits individual transactions but can add total work; it is not a substitute for correct phase ordering.

Locks are one failure boundary

In PostgreSQL, ACCESS EXCLUSIVE conflicts with ordinary reads and writes. Many ALTER TABLE forms take it even when no table rewrite is needed. In contrast, an ordinary UPDATE takes ROW EXCLUSIVE plus row locks; it does not globally exclude every writer. Conflicts depend on affected rows and other operations, while long transactions retain locks until completion.

A queued strong lock can delay later conflicting requests. Identify actual blockers, transaction boundaries, and wait events rather than blaming whichever transaction is oldest. SQLite rollback-journal and WAL behavior are different engine contracts. The earlier article’s millisecond and page-count figures have no exact benchmark evidence established here and are not PostgreSQL predictions or measurements from this review.

Validate under a weaker lock

For PostgreSQL 17, add an appropriate CHECK ... NOT VALID only once current writers satisfy it: new inserts and updates are checked immediately, while old rows await validation. VALIDATE CONSTRAINT scans under SHARE UPDATE EXCLUSIVE, which permits ordinary DML but conflicts with some maintenance and schema operations. It is not lock-free.

A validated CHECK proving the column cannot be null can let SET NOT NULL skip its scan, provided that check is not dropped in the same command. This does not eliminate the stronger lock needed by the final alteration, and setting NOT NULL does not inherently rewrite the table. PostgreSQL 18 adds not-null constraint capabilities, so consult the deployed major version rather than assuming every release supports identical syntax.

When it breaks

A backfill finishes once, then a surviving old writer makes the new representation stale. Monitor mismatch counts and writer versions before cutting reads over. A destructive contract phase may require restore or forward repair; rehearse that recovery before removing data.

Ordinary CREATE INDEX blocks writes during construction. CREATE INDEX CONCURRENTLY permits normal DML, but performs extra scans and waits, cannot run inside a transaction block, and can leave an invalid index after failure. Such an index may still impose update overhead or enforce uniqueness. Inspect its exact state before an approved recovery operation.

Migration compatibility: mechanism and evidence boundaries.

Check it quickly

These retained commands are an operator checklist, not an isolated migration test. Use only an explicitly selected disposable PostgreSQL database where an owned events table exists and note does not. Confirm connection and search_path, suitable catalog visibility, DDL permission, and a separate total statement timeout. The fourth command changes schema; do not paste it into production.

psql -c "SELECT pid, state, now()-xact_start AS age, left(query,50) FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY xact_start;"
psql -c "SELECT pid, mode, granted, left(relation::regclass::text,40) FROM pg_locks WHERE NOT granted;"
psql -c "SELECT indexrelid::regclass, indisvalid FROM pg_index WHERE NOT indisvalid;"
psql -c "SET lock_timeout='3s'; ALTER TABLE events ADD COLUMN note text;"
psql -c "SELECT conname, convalidated FROM pg_constraint WHERE conrelid='events'::regclass;"

The activity query lists candidates, not proven blockers; pg_blocking_pids supplies blocking relationships. Ungranted locks do not identify every holder. lock_timeout bounds each acquisition wait, not execution time or total impact. convalidated alone does not show which condition was proven. No commands ran here; rollout compatibility, concurrent backfill, constraint transitions, and recovery still need an authorized, version-pinned fixture.

Sources

PostgreSQL: ALTER TABLE, SET NOT NULL, NOT VALID, and VALIDATE CONSTRAINT, checked 2026-09-16.
PostgreSQL: CREATE INDEX, building indexes concurrently and the invalid-index failure mode, checked 2026-09-16.
PostgreSQL: explicit locking, which commands take ACCESS EXCLUSIVE and SHARE, checked 2026-09-16.
Martin Fowler: parallel change, the expand/contract pattern under another name, checked 2026-09-16.
PostgreSQL 17: ALTER TABLE, version-specific CHECK validation and NOT NULL transitions, checked 2026-10-18.
PostgreSQL 17: client defaults, lock wait timeout versus statement duration, checked 2026-10-18.
PostgreSQL 17: system information functions, actual blocker relationships through pg_blocking_pids, checked 2026-10-18.
PostgreSQL 17: transaction isolation, concurrent updates and predicate rechecks, checked 2026-10-18.