Required fields and protobuf 3

The required field rule is often used by well-meaning developers to ensure that a field regarded as important by the application layer has a value in a given message.

Unfortunately the semantics behind this rule require that a valid value for this field must be present in the serialized form of the message.

If the field is missing or the value is not understood the entire message should be rejected.

This means that unless you can guarantee that no consumers of the message will be using the older .proto definition you can never remove a required field once it's been added, even if some change has occurred that means it no longer makes sense for your domain model.

Since the value needs to be understood by the decoding end it also means you can't add additional values to any enums used as required fields since there's no guarantee the recipient will know about the new values.

Protobuf 2 vs Protobuf 3

The proto2 language guide marks the required with Do not use and proto3 has removed it entirely.

Protons strongly prefers proto3 semantics and will emit a warning (or fail with an error in strict mode) when a required field is encountered in a proto3 message. This can happen if, for example, you import a proto2 message into a proto3 definition file.

Workarounds

The default field rule in proto3 is singular which for our purposes means "only write a value if set, when reading populate with the value if sent or the default value if not". The optional type means "only write a value if set, when reading only populate with the value if sent", therefore one must explicitly set a default value if we wish the remote to read a default value.

Consequently a buffer created from a proto3 definition can be successfully decoded by a recipient that uses an older proto2 version of the .proto file with a required field, if the sender marks the field as optional and (important!) also ensures that a value is set at the application level.

Unfortunately this trick doesn't work with values added to required enum fields. If you have an enum used in a required field and the older proto2-preferring recipients cannot upgrade you can never add a new value to the enum (sad-trombone).

If you have no need to add new values to the enum though, you can still mark the field optional in the proto3 version and set a value at the application level to ensure a value is sent.

How does Protons handle this?

When generating types for a message with a required field in non-strict mode, Protons will mark that field as non-optional in the interface for the message type.

When serializing, Protons will write a value for a required field into the serialized output. If the value has not been set on the message object the default value will be written.

When reading it will treat a required field as singular, setting it to the default value for the field if it is missing.

This gives us the maximum amount of flexibility and compatibility with older proto2-preferring implementations.