Container-based proto compilation #136
Conversation
use cmd variables added newline
n0shut
force-pushed
the
docker-protos
branch
from
aa2a542
to
1fe5eaf
Compare
compile-protos-docker.sh
Outdated
|
|
||
| DOCKER_CMD="docker run -v $(pwd):/in \ | ||
| -v $(pwd)/proto:/in/github.com/openconfig/gnmi/proto \ | ||
| ghcr.io/srl-labs/protoc:0.0.3" |
There was a problem hiding this comment.
For my edification, how does this map to the protoc version that is in the container?
There was a problem hiding this comment.
@robshakir it doesn't direclty map to the protoc version, as there are more than just protoc in the package (namely grpc plugins for both py and go version).
Therefore, the idea is that ghcr.io/srl-labs/protoc:0.0.3 is tagged in a semver fashion and the image tag matches the tag in the git repo for which the Dockerfile reveals the used versions: https://github.com/srl-labs/protoc-container/blob/v0.0.3/Dockerfile
There was a problem hiding this comment.
ACK - no problem. I was just wondering whether it is possible to directly map these together so that it's clearer, but it's written into the pb.go as to what versions are used so I don't see this is a huge issue.
The only downside here is that we stay pinned to that 0.0.3 version until an explicit change is made. If there's a semver convention here - could we consider tagging something like 1.x.x-latest? (It'd be nice to stay close to HEAD for the generated files.)
There was a problem hiding this comment.
@robshakir good call, I see that the upstream changed the logic to include protoc__protoc-gen-go version in the image tag, so that it's clear which sw components versions are used.
https://github.com/srl-labs/protoc-container#versioning
I have therefore update the branch in 4a11050 to reflect that.
Now it is explicit which version of protoc and protoc-gen-go are used in the image.
There was a problem hiding this comment.
I can also port it to gnoi repo where it is even more of a pressing issue.
|
Hi @robshakir @wenovus # to compile all protos
./run.sh compile-all
# to compile selected protos
./run.sh compile-go-gnmi-ext |
|
tsk tsk tsk |
|
Thanks for this -- let me look again this week. Unfortunately, I was on an extended leave during the period you previously requested review. |
|
No worries @robshakir, I didn't mean to push, just a slight nudge. I can update it with the updated versions of protoc and go plugins |
| set -o pipefail | ||
|
|
||
| # container image tag, to see all available tags go to | ||
| # https://github.com/srl-labs/protoc-container/pkgs/container/protoc |
There was a problem hiding this comment.
Is this container something that you expect to continue to be updated?
There was a problem hiding this comment.
yes, I just bumped it to a more recent version tuple 227be07
run.sh
Outdated
| function compile-go-gnmi { | ||
| ${DOCKER_CMD} ash -c "${PROTOC_GO_CMD} proto/gnmi/gnmi.proto" | ||
| echo "finished Go compilation for gnmi.proto" | ||
|
|
There was a problem hiding this comment.
(nit) is this newline needed (here and below)?
|
@marcushines - PTAL too. |
|
compile_protos.sh brought back, @robshakir |
|
closing as stale |
Hi @robshakir @marcushines
I would like to propose a hermetic way to compile protos used in this repository which is based on the container approach.
Currently, the compile-protos.sh script has no pinned dependencies resulting in users defaulting to whatever latest/oldest version they have installed on their machines when updating the protos.
Not only it adds unnecessary churn to the PRs, it also makes the process overly complicated. The dev machine should have protos installed in the specified directories and we see that people struggle with that - #134.
In the newly added run.sh I introduced a container-based workflow where I leverage ghcr.io/srl-labs/protoc container that has pinned dependencies for protoc and the relevant grpc plugins - https://github.com/srl-labs/protoc-container/blob/main/Dockerfile
Using this container every user of gnmi repo will be guaranteed to compile the protos in one and only possible way, without the need to keep local dependencies and maintain their lifecycle.
At the time of this writing the latest version of protoc and plugins were used. I hope you will find that useful.
In #137 I generated the Go/Py bindings for you to verify how it works and potentially try it yourself to ensure that it works the same for everyone in a consistent way.