Protobuf · XB Field Notes

Protobuf evolution preserves meaning only within a defined contract

Sat, 26 Feb 2028 00:00:00 +0000

A gRPC rollout can leave old producers, new consumers, and intermediate services running together. Protobuf helps them exchange binary messages by identifying fields with numbers and wire types. That tolerance is narrower than application compatibility: a message can parse successfully while a consumer misunderstands its values, ignores a new instruction, or loses the distinction between absent and explicitly zero.

Separate numbers, meaning, and bytes

A binary tag combines the field number with its wire type; the field name is not transmitted there. Keeping the number and type while renaming a field can preserve binary decoding. It still changes generated accessors and may change JSON or text interfaces. A gRPC method name or service name belongs to another contract and is not protected by a field-number rule.

Never reuse a retired number for another meaning. Different declared types can share a wire type: signed and unsigned integers both use varints, yet a negative signed value can become a large unsigned value. A successful parse is therefore insufficient validation. Reserve retired numbers and names in the schema so later declarations cannot accidentally reclaim them; reserved numbers are unavailable, not placeholders to activate later.

Unknown fields can survive a parse-and-serialize binary relay. However, copying only known fields or converting through JSON can discard them. Preservation of decoded information also does not promise byte-identical serialization. Field order and encoding choices can differ, including across implementations. Compare required values and presence in compatibility tests; do not use ordinary reserialization as a canonical digest algorithm.

Presence carries a separate bit of meaning

In proto3, singular scalar fields without optional generally use implicit presence. Their default value is indistinguishable from absence through the normal API. Explicit optional scalars track whether a value was supplied, including zero. Message fields and oneof members already have presence; repeated and map fields do not distinguish an empty collection from an absent one.

Changing a scalar between implicit and explicit presence can remain binary-readable while changing round-trip semantics. An older reader that knows the field with implicit presence can parse explicit zero and then omit it when forwarding. Adding optional at one endpoint cannot force every intermediary to preserve that distinction. Test the entire path that actually modifies or forwards the message.

When it breaks

A new consumer may require a field that old writers never supplied. Although proto3 has no required declaration, application validation can create exactly this rolling-upgrade failure. Introduce an agreed fallback or sequence the rollout before enforcing the requirement. Optional syntax alone does not define a sensible default, authorize an operation, or make an old service implement a new feature.

Also test persisted messages, delayed jobs, and rollback versions. Their useful lifetime can exceed the deployment window. A schema check against only the immediately previous release misses older records that can still reappear. Keep retired numbers reserved even when current traffic no longer uses them.

Both relayed results return zero, but the unknown-field relay retains explicit presence while the known implicit-presence relay loses it. The historical Python fixture passed within its local scope; it did not test reserved declarations, real gRPC calls or cross-language clients. No example was rerun for this drawing.

Check it quickly

Run the complete block with Python and Google’s protobuf package 7.36.2 available. It was executed with Python 3.12.14. Separate descriptor pools create real old and new message classes without a compiler, filesystem fixture, gRPC server, or network connection. This checks the Python binary runtime, not cross-language interoperability or generated-client compatibility.

from google.protobuf import descriptor_pb2 as schema, descriptor_pool, message_factory


def message(fields, explicit=()):
 file = schema.FileDescriptorProto(name='probe.proto', package='probe', syntax='proto3')
 record = file.message_type.add(name='Record')
 for name, number, kind in fields:
 field = record.field.add(name=name, number=number, type=kind, label=1)
 if name in explicit:
 field.oneof_index = len(record.oneof_decl)
 record.oneof_decl.add(name='_' + name)
 field.proto3_optional = True
 pool = descriptor_pool.DescriptorPool()
 pool.Add(file)
 return message_factory.GetMessageClass(pool.FindMessageTypeByName('probe.Record'))


INT32 = schema.FieldDescriptorProto.TYPE_INT32
UINT32 = schema.FieldDescriptorProto.TYPE_UINT32
STRING = schema.FieldDescriptorProto.TYPE_STRING
Old = message([('id', 1, INT32), ('name', 2, STRING)])
New = message([('event_id', 1, INT32), ('name', 2, STRING), ('count', 3, INT32)], ('count',))
Implicit = message([('event_id', 1, INT32), ('name', 2, STRING), ('count', 3, INT32)])

old = Old(id=7, name='alpha')
new_reader = New.FromString(old.SerializeToString())
assert new_reader.event_id == 7 and not new_reader.HasField('count')
# The added explicit zero is unknown to Old but retained in binary forwarding.
new_writer = New(event_id=7, name='alpha', count=0)
relay = Old.FromString(new_writer.SerializeToString())
returned = New.FromString(relay.SerializeToString())
assert returned.event_id == 7 and returned.HasField('count') and returned.count == 0
print('old/new binary values survive; unknown explicit zero survives the old relay')

# An implicit-presence reader knows field 3, so zero is omitted on serialization.
implicit_relay = Implicit.FromString(new_writer.SerializeToString())
lost_presence = New.FromString(implicit_relay.SerializeToString())
assert lost_presence.count == 0 and not lost_presence.HasField('count')
print('an implicit-presence relay preserves zero but loses its explicit presence')

Signed = message([('value', 1, INT32)])
Unsigned = message([('value', 1, UINT32)])
assert Unsigned.FromString(Signed(value=-1).SerializeToString()).value == 4294967295
# Distinct field order is legal; compare decoded meaning, not byte equality.
reordered = b'\x18\x00' + old.SerializeToString()
assert New.FromString(reordered) == new_writer
print('reusing an integer tag changes meaning; legal field orders decode equivalently')

The first output confirms that an old writer remains readable and that an unknown explicit zero survives the old binary relay. The next output exposes a different result when the relay knows that field with implicit presence: its value remains zero, but HasField becomes false after forwarding. These are different compatibility questions, not contradictory observations.

The final assertions decode negative one as 4294967295 under an incompatible unsigned interpretation and accept a legal alternative field order. They demonstrate why wire readability does not establish semantic equivalence or canonical bytes. The descriptor exercise does not compile reserved declarations; compiler checks and real gRPC request/response behavior remain separate rollout tests.

Sources

Protobuf: encoding, tags, varints and serialization ordering, checked 2026-10-15.
Protobuf: proto3 language guide, binary changes, reserved fields and unknown-field retention, checked 2026-10-15.
Protobuf: field presence, implicit and explicit presence across relays, checked 2026-10-15.
Protobuf: dos and don’ts, retired numbers and compatibility discipline, checked 2026-10-15.

Protobuf compatibility changes at a JSON boundary

Fri, 19 Nov 2027 00:00:00 +0000

A schema change can be safe for binary messages and unsafe for a JSON gateway handling the same service. Protobuf’s binary format identifies fields by number; ProtoJSON normally exposes their names. Review compatibility as a matrix of producers, consumers, and transformations. A successful compile or one pair of compatible endpoints does not prove that every relay preserves the information they exchange.

Additive changes depend on the representation

Adding a fresh numbered field is generally binary wire-safe. An old parser can retain the unknown field while continuing to read familiar ones. A new parser reading older data receives the appropriate default or absence state. Whether that absence is acceptable remains an application decision; do not require a newly introduced value before old writers can supply it.

ProtoJSON has different rules. Parsers reject unknown fields by default, so an old JSON consumer can reject a message containing a newly added property. Ignoring unknown fields can make parsing succeed, but discarded information is not preserved for a later consumer. That option is a deliberate data-loss policy, not an equivalent implementation of binary forwarding.

Renaming a field while retaining its number illustrates the distinction. Binary readers can still associate the value with that number. Generated accessors and the default JSON property change, however, and an old property can become unknown to the new JSON parser. The json_name option participates in that contract too. Inventory the names actually emitted and accepted before rolling out a rename.

Follow transformations, not just endpoints

An old message object can preserve unknown binary data internally even when its public fields cannot describe it. Serializing that object as JSON emits its known fields, losing the unknown binary information. Constructing a fresh message by copying only known fields creates a similar loss. Prefer message-oriented copy operations when retaining unknown data is required, and test that retention explicitly.

Do not confuse this with stable serialization bytes. Equivalent messages need not produce identical binary output across versions or implementations. A comparison should establish the required values and presence, rather than demand identical byte ordering unless the protocol defines its own canonical representation. Signing or hashing reserialized messages requires a separate, carefully defined contract.

Retire field numbers permanently, and reserve removed names to prevent accidental schema reuse. A reservation does not preserve runtime JSON acceptance of the old property. A field marked deprecated also remains available; that annotation does not remove old writers, persisted records, or downstream dependencies. Removal needs an inventory and a migration plan.

When it breaks

A gateway can accept a new binary request, convert it through an old-schema JSON object, and silently erase a new instruction. The final consumer may parse successfully and take its default branch. Conversely, a strict old JSON parser may reject the entire message. Inspect both paths: rejection and silent omission have different operational symptoms but arise from the same representation boundary.

Also distinguish schema compatibility from code-generation/runtime compatibility. Generated code can require particular supported runtime versions even when the wire schema is unchanged. Maintain the vendor’s supported combinations and test the oldest relevant deployed reader, delayed messages, and rollback path. A homemade dictionary checker cannot establish any of these runtime properties.

The same new note is retained by the old binary relay but lost when its old-schema message object passes through JSON. New JSON can instead be rejected by a strict old parser, or parsed with note discarded. The historical Python fixture passed only for its specific schemas and runtime; it does not certify gRPC or every generated client. No example was rerun for this drawing.

Check it quickly

Run this complete script with Python and Google’s protobuf package 7.36.2. It passed on Python 3.12.14. Independent descriptor pools create real message classes for three schemas. The binary and JSON parsers are the library’s actual implementations; there is no hand-written serialization model, compiler invocation, server, or network request.

from google.protobuf import descriptor_pb2 as schema, descriptor_pool, message_factory, json_format


def message(fields):
 file = schema.FileDescriptorProto(name='probe.proto', package='probe', syntax='proto3')
 record = file.message_type.add(name='Record')
 for name, number, kind in fields:
 record.field.add(name=name, number=number, type=kind, label=1)
 pool = descriptor_pool.DescriptorPool()
 pool.Add(file)
 return message_factory.GetMessageClass(pool.FindMessageTypeByName('probe.Record'))


INT32 = schema.FieldDescriptorProto.TYPE_INT32
STRING = schema.FieldDescriptorProto.TYPE_STRING
Old = message([('id', 1, INT32)])
New = message([('id', 1, INT32), ('note', 2, STRING)])
Renamed = message([('event_id', 1, INT32)])
producer = New(id=7, note='retain me')
old_relay = Old.FromString(producer.SerializeToString())
assert New.FromString(old_relay.SerializeToString()).note == 'retain me'

# Known-field JSON serialization omits unknown binary fields.
relay_json = json_format.MessageToJson(old_relay)
through_json = json_format.Parse(relay_json, Old())
assert New.FromString(through_json.SerializeToString()).note == ''
print('binary relay keeps note; old-schema JSON relay drops it')

new_json = json_format.MessageToJson(producer)
try:
 json_format.Parse(new_json, Old())
except json_format.ParseError:
 pass
else:
 raise AssertionError('unknown JSON field was accepted by the strict old parser')
ignored = json_format.Parse(new_json, Old(), ignore_unknown_fields=True)
assert ignored.id == 7
assert New.FromString(ignored.SerializeToString()).note == ''
print('strict old JSON parser rejects new field; permissive parsing discards it')

assert Renamed.FromString(Old(id=7).SerializeToString()).event_id == 7
try:
 json_format.Parse(json_format.MessageToJson(Old(id=7)), Renamed())
except json_format.ParseError:
 pass
else:
 raise AssertionError('renamed JSON field unexpectedly accepted')
print('same-number rename reads binary but rejects the previous JSON name')

The binary relay retains note, while the same old message passed through JSON loses it. Parsing new JSON with the strict old schema raises ParseError. Enabling unknown-field ignoring keeps the known identifier but still loses the new note. Finally, a same-number rename preserves binary reading yet rejects the prior JSON property.

These assertions test specific schema versions in one runtime. They do not certify an arbitrary change, a gRPC transport, or every language’s generated client. For a release gate, extend the same cross-version matrix with actual schemas, required semantic values, presence-sensitive defaults, and each intermediary representation used in production.

Sources

Protobuf: proto3 language guide, binary-safe additions and unknown-field retention, checked 2026-10-15.
Protobuf: ProtoJSON, unknown-field handling, property names and compatibility, checked 2026-10-15.
Protobuf: serialization is not canonical, limits of deterministic output, checked 2026-10-15.
Protobuf: runtime version guarantee, supported generated-code/runtime combinations, checked 2026-10-15.