Protobuf compatibility changes at a JSON boundary

Fri, 19 Nov 2027 00:00:00 +0000

A schema change can be safe for binary messages and unsafe for a JSON gateway handling the same service. Protobuf’s binary format identifies fields by number; ProtoJSON normally exposes their names. Review compatibility as a matrix of producers, consumers, and transformations. A successful compile or one pair of compatible endpoints does not prove that every relay preserves the information they exchange.

Additive changes depend on the representation

Adding a fresh numbered field is generally binary wire-safe. An old parser can retain the unknown field while continuing to read familiar ones. A new parser reading older data receives the appropriate default or absence state. Whether that absence is acceptable remains an application decision; do not require a newly introduced value before old writers can supply it.

ProtoJSON has different rules. Parsers reject unknown fields by default, so an old JSON consumer can reject a message containing a newly added property. Ignoring unknown fields can make parsing succeed, but discarded information is not preserved for a later consumer. That option is a deliberate data-loss policy, not an equivalent implementation of binary forwarding.

Renaming a field while retaining its number illustrates the distinction. Binary readers can still associate the value with that number. Generated accessors and the default JSON property change, however, and an old property can become unknown to the new JSON parser. The json_name option participates in that contract too. Inventory the names actually emitted and accepted before rolling out a rename.

Follow transformations, not just endpoints

An old message object can preserve unknown binary data internally even when its public fields cannot describe it. Serializing that object as JSON emits its known fields, losing the unknown binary information. Constructing a fresh message by copying only known fields creates a similar loss. Prefer message-oriented copy operations when retaining unknown data is required, and test that retention explicitly.

Do not confuse this with stable serialization bytes. Equivalent messages need not produce identical binary output across versions or implementations. A comparison should establish the required values and presence, rather than demand identical byte ordering unless the protocol defines its own canonical representation. Signing or hashing reserialized messages requires a separate, carefully defined contract.

Retire field numbers permanently, and reserve removed names to prevent accidental schema reuse. A reservation does not preserve runtime JSON acceptance of the old property. A field marked deprecated also remains available; that annotation does not remove old writers, persisted records, or downstream dependencies. Removal needs an inventory and a migration plan.

When it breaks

A gateway can accept a new binary request, convert it through an old-schema JSON object, and silently erase a new instruction. The final consumer may parse successfully and take its default branch. Conversely, a strict old JSON parser may reject the entire message. Inspect both paths: rejection and silent omission have different operational symptoms but arise from the same representation boundary.

Also distinguish schema compatibility from code-generation/runtime compatibility. Generated code can require particular supported runtime versions even when the wire schema is unchanged. Maintain the vendor’s supported combinations and test the oldest relevant deployed reader, delayed messages, and rollback path. A homemade dictionary checker cannot establish any of these runtime properties.

The same new note is retained by the old binary relay but lost when its old-schema message object passes through JSON. New JSON can instead be rejected by a strict old parser, or parsed with note discarded. The historical Python fixture passed only for its specific schemas and runtime; it does not certify gRPC or every generated client. No example was rerun for this drawing.

Check it quickly

Run this complete script with Python and Google’s protobuf package 7.36.2. It passed on Python 3.12.14. Independent descriptor pools create real message classes for three schemas. The binary and JSON parsers are the library’s actual implementations; there is no hand-written serialization model, compiler invocation, server, or network request.

from google.protobuf import descriptor_pb2 as schema, descriptor_pool, message_factory, json_format


def message(fields):
 file = schema.FileDescriptorProto(name='probe.proto', package='probe', syntax='proto3')
 record = file.message_type.add(name='Record')
 for name, number, kind in fields:
 record.field.add(name=name, number=number, type=kind, label=1)
 pool = descriptor_pool.DescriptorPool()
 pool.Add(file)
 return message_factory.GetMessageClass(pool.FindMessageTypeByName('probe.Record'))


INT32 = schema.FieldDescriptorProto.TYPE_INT32
STRING = schema.FieldDescriptorProto.TYPE_STRING
Old = message([('id', 1, INT32)])
New = message([('id', 1, INT32), ('note', 2, STRING)])
Renamed = message([('event_id', 1, INT32)])
producer = New(id=7, note='retain me')
old_relay = Old.FromString(producer.SerializeToString())
assert New.FromString(old_relay.SerializeToString()).note == 'retain me'

# Known-field JSON serialization omits unknown binary fields.
relay_json = json_format.MessageToJson(old_relay)
through_json = json_format.Parse(relay_json, Old())
assert New.FromString(through_json.SerializeToString()).note == ''
print('binary relay keeps note; old-schema JSON relay drops it')

new_json = json_format.MessageToJson(producer)
try:
 json_format.Parse(new_json, Old())
except json_format.ParseError:
 pass
else:
 raise AssertionError('unknown JSON field was accepted by the strict old parser')
ignored = json_format.Parse(new_json, Old(), ignore_unknown_fields=True)
assert ignored.id == 7
assert New.FromString(ignored.SerializeToString()).note == ''
print('strict old JSON parser rejects new field; permissive parsing discards it')

assert Renamed.FromString(Old(id=7).SerializeToString()).event_id == 7
try:
 json_format.Parse(json_format.MessageToJson(Old(id=7)), Renamed())
except json_format.ParseError:
 pass
else:
 raise AssertionError('renamed JSON field unexpectedly accepted')
print('same-number rename reads binary but rejects the previous JSON name')

The binary relay retains note, while the same old message passed through JSON loses it. Parsing new JSON with the strict old schema raises ParseError. Enabling unknown-field ignoring keeps the known identifier but still loses the new note. Finally, a same-number rename preserves binary reading yet rejects the prior JSON property.

These assertions test specific schema versions in one runtime. They do not certify an arbitrary change, a gRPC transport, or every language’s generated client. For a release gate, extend the same cross-version matrix with actual schemas, required semantic values, presence-sensitive defaults, and each intermediary representation used in production.

Sources