Protobuf evolution preserves meaning only within a defined contract

Sat, 26 Feb 2028 00:00:00 +0000

A gRPC rollout can leave old producers, new consumers, and intermediate services running together. Protobuf helps them exchange binary messages by identifying fields with numbers and wire types. That tolerance is narrower than application compatibility: a message can parse successfully while a consumer misunderstands its values, ignores a new instruction, or loses the distinction between absent and explicitly zero.

Separate numbers, meaning, and bytes

A binary tag combines the field number with its wire type; the field name is not transmitted there. Keeping the number and type while renaming a field can preserve binary decoding. It still changes generated accessors and may change JSON or text interfaces. A gRPC method name or service name belongs to another contract and is not protected by a field-number rule.

Never reuse a retired number for another meaning. Different declared types can share a wire type: signed and unsigned integers both use varints, yet a negative signed value can become a large unsigned value. A successful parse is therefore insufficient validation. Reserve retired numbers and names in the schema so later declarations cannot accidentally reclaim them; reserved numbers are unavailable, not placeholders to activate later.

Unknown fields can survive a parse-and-serialize binary relay. However, copying only known fields or converting through JSON can discard them. Preservation of decoded information also does not promise byte-identical serialization. Field order and encoding choices can differ, including across implementations. Compare required values and presence in compatibility tests; do not use ordinary reserialization as a canonical digest algorithm.

Presence carries a separate bit of meaning

In proto3, singular scalar fields without optional generally use implicit presence. Their default value is indistinguishable from absence through the normal API. Explicit optional scalars track whether a value was supplied, including zero. Message fields and oneof members already have presence; repeated and map fields do not distinguish an empty collection from an absent one.

Changing a scalar between implicit and explicit presence can remain binary-readable while changing round-trip semantics. An older reader that knows the field with implicit presence can parse explicit zero and then omit it when forwarding. Adding optional at one endpoint cannot force every intermediary to preserve that distinction. Test the entire path that actually modifies or forwards the message.

When it breaks

A new consumer may require a field that old writers never supplied. Although proto3 has no required declaration, application validation can create exactly this rolling-upgrade failure. Introduce an agreed fallback or sequence the rollout before enforcing the requirement. Optional syntax alone does not define a sensible default, authorize an operation, or make an old service implement a new feature.

Also test persisted messages, delayed jobs, and rollback versions. Their useful lifetime can exceed the deployment window. A schema check against only the immediately previous release misses older records that can still reappear. Keep retired numbers reserved even when current traffic no longer uses them.

Both relayed results return zero, but the unknown-field relay retains explicit presence while the known implicit-presence relay loses it. The historical Python fixture passed within its local scope; it did not test reserved declarations, real gRPC calls or cross-language clients. No example was rerun for this drawing.

Check it quickly

Run the complete block with Python and Google’s protobuf package 7.36.2 available. It was executed with Python 3.12.14. Separate descriptor pools create real old and new message classes without a compiler, filesystem fixture, gRPC server, or network connection. This checks the Python binary runtime, not cross-language interoperability or generated-client compatibility.

from google.protobuf import descriptor_pb2 as schema, descriptor_pool, message_factory


def message(fields, explicit=()):
 file = schema.FileDescriptorProto(name='probe.proto', package='probe', syntax='proto3')
 record = file.message_type.add(name='Record')
 for name, number, kind in fields:
 field = record.field.add(name=name, number=number, type=kind, label=1)
 if name in explicit:
 field.oneof_index = len(record.oneof_decl)
 record.oneof_decl.add(name='_' + name)
 field.proto3_optional = True
 pool = descriptor_pool.DescriptorPool()
 pool.Add(file)
 return message_factory.GetMessageClass(pool.FindMessageTypeByName('probe.Record'))


INT32 = schema.FieldDescriptorProto.TYPE_INT32
UINT32 = schema.FieldDescriptorProto.TYPE_UINT32
STRING = schema.FieldDescriptorProto.TYPE_STRING
Old = message([('id', 1, INT32), ('name', 2, STRING)])
New = message([('event_id', 1, INT32), ('name', 2, STRING), ('count', 3, INT32)], ('count',))
Implicit = message([('event_id', 1, INT32), ('name', 2, STRING), ('count', 3, INT32)])

old = Old(id=7, name='alpha')
new_reader = New.FromString(old.SerializeToString())
assert new_reader.event_id == 7 and not new_reader.HasField('count')
# The added explicit zero is unknown to Old but retained in binary forwarding.
new_writer = New(event_id=7, name='alpha', count=0)
relay = Old.FromString(new_writer.SerializeToString())
returned = New.FromString(relay.SerializeToString())
assert returned.event_id == 7 and returned.HasField('count') and returned.count == 0
print('old/new binary values survive; unknown explicit zero survives the old relay')

# An implicit-presence reader knows field 3, so zero is omitted on serialization.
implicit_relay = Implicit.FromString(new_writer.SerializeToString())
lost_presence = New.FromString(implicit_relay.SerializeToString())
assert lost_presence.count == 0 and not lost_presence.HasField('count')
print('an implicit-presence relay preserves zero but loses its explicit presence')

Signed = message([('value', 1, INT32)])
Unsigned = message([('value', 1, UINT32)])
assert Unsigned.FromString(Signed(value=-1).SerializeToString()).value == 4294967295
# Distinct field order is legal; compare decoded meaning, not byte equality.
reordered = b'\x18\x00' + old.SerializeToString()
assert New.FromString(reordered) == new_writer
print('reusing an integer tag changes meaning; legal field orders decode equivalently')

The first output confirms that an old writer remains readable and that an unknown explicit zero survives the old binary relay. The next output exposes a different result when the relay knows that field with implicit presence: its value remains zero, but HasField becomes false after forwarding. These are different compatibility questions, not contradictory observations.

The final assertions decode negative one as 4294967295 under an incompatible unsigned interpretation and accept a legal alternative field order. They demonstrate why wire readability does not establish semantic equivalence or canonical bytes. The descriptor exercise does not compile reserved declarations; compiler checks and real gRPC request/response behavior remain separate rollout tests.

Sources