Add support for protobuf serialization for plugin communication #13120
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lbajolet-hashicorp
force-pushed
the
feature/protobuf-serialization
branch
from
cf5cf79 to
1fb5b20
Compare
lbajolet-hashicorp
force-pushed
the
feature/protobuf-serialization
branch
2 times, most recently
from
e3a2f63 to
c567587
Compare
lbajolet-hashicorp
force-pushed
the
feature/protobuf-serialization
branch
from
f668d07 to
9ff6fe7
Compare
lbajolet-hashicorp
force-pushed
the
feature/protobuf-serialization
branch
from
57739d9 to
7bb4c19
Compare
lbajolet-hashicorp
force-pushed
the
feature/protobuf-serialization
branch
from
7bb4c19 to
4b1c912
Compare
As we're trying to move away from gob for serialising data over the wire, this commit adds the capability for Packer to pick dynamically between gob or protobuf for the serialisation format to communicate with plugins. As it stands, if all the plugins discovered are compatible with protobuf, and we have not forced gob usage, protobuf will be the serialisation format picked. If any plugin is not compatible with protobuf, gob will be used for communicating with all the plugins that will be used over the course of a command.
With the draft to support both gob and protobuf as serialisation formats for Packer, along with the SDK changes that propel them, we add a series of tests that make sure the logic that picks which protocol is solid and functional. These tests rely on building several versions of the tester plugin, with and without protobuf support, to then install them in the tests as needed to test the logic of Packer using packer build with them, and templates that require multiple plugins.
lbajolet-hashicorp
force-pushed
the
feature/protobuf-serialization
branch
from
4b1c912 to
78cfe63
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In HashiCorp Packer 1.12, we are working to remove Packer's reliance on gob encoding for its plugin system architecture. We will introduce protos and custom HCL ObjectSpec wrappers for sending ObjectSpecs types over the wire using protobufs.
This work will look similar to the work already done for Nomad. The first implementation will use an environment variable to support the protocol switching server for the existing net/RPC implementation. Users of a plugin via the environment can toggle net/rpc with gob or net/rpc with protobuf communication for any supported Platform. Since Packer can not mix and match serialization formats at runtime, Packer will determine the supported formats by each plugin and use the appropriate wire protocol. If all plugins support the new serialization, protobuf/msgpack will be selected as the wire protocol. If the most compatible format is gob then the old serialization protocol will be used. Users wishing to force the old serialization for compatibility reasons can specify the PACKER_FORCE_GOB environment variable to turn off protobuf and msgpack serialization detection.