Add container-based build workflow and improve build tooling #93
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently if get_git_commit_id() fails, its not clear why:
Generation for Bootloader to /build_dir has completed.
...snip...
File "/usr/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', '-C', '/build', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
So lets fix it by providing stderr output, thus making the issue
obvious:
Generation for Bootloader to /build_dir has completed.
...snip...
File "/build/tools/build_project.py", line 113, in git
raise RuntimeError(
RuntimeError: Git command `git -C /build rev-parse HEAD` failed: fatal: detected dubious ownership in repository at '/build'
To add an exception for this directory, call:
git config --global --add safe.directory /build
Signed-off-by: Petr Štetiar <[email protected]>
Using the build workflow with rootless podman containers and volumes
results into following permssions issues:
$ make build_firmware MANIFESTS=manifests/nabucasa/yellow_openthread_rcp.yaml
podman run --rm -it \
-v /silabs/silabs-firmware-builder:/build:z \
-v /silabs/silabs-firmware-builder/outputs:/outputs:z \
-v /silabs/silabs-firmware-builder/build_dir:/build_dir:z silabs-firmware-builder \
bash -c " \
build_firmware.sh \
--build-dir /build_dir \
--output-dir /outputs \
--manifest manifests/nabucasa/skyconnect_openthread_rcp.yaml \
"
The sdk /gecko_sdk_4.4.4/ ( com.silabs.sdk.stack.super:4.4.4._1207041799 ) is now trusted.
ln: failed to create symbolic link '/gecko_sdk_4.4.4/extension': Permission denied
The sdk /simplicity_sdk_2024.6.2/ ( com.silabs.sdk.stack.sisdk:2024.6.2._-620023087 ) is now trusted.
ln: failed to create symbolic link '/simplicity_sdk_2024.6.2/extension': Permission denied
This is happening due to the user/group mapping between container and
the host, where currently the simplicity_sdk and gecko_sdk directories
are owned as root, thus builder user won't be able to create an
extension symlink, resulting in this failures.
So lets fix it by chown-ing the simplicity_sdk and gecko_sdk folders for builder user.
References: https://www.redhat.com/en/blog/debug-rootless-podman-mounted-volumes
Signed-off-by: Petr Štetiar <[email protected]>
…ality Currently its not possible to easily reuse the steps taken on the GitHub CI to build the firmware, so lets factor out those common bits into new build_firmware.sh script help which basically mimics the current firmware build flow on the GitHub CI and can be as well reused for example in local container based workflow. Signed-off-by: Petr Štetiar <[email protected]>
ynezz
force-pushed
the
ynezz/build-improvements
branch
from
d799dd1 to
8a7e053
Compare
|
Just fixed docker run --rm -it --user 0:0 -v /builds/prpl-foundation/mirrors/silabs-firmware-builder:/build -v /builds/prpl-foundation/mirrors/silabs-firmware-builder/outputs:/outputs -v /builds/prpl-foundation/mirrors/silabs-firmware-builder/build_dir:/build_dir silabs-firmware-builder \
bash -c " \
build_firmware.sh \
--build-dir /build_dir \
--output-dir /outputs \
--manifest manifests/nabucasa/skyconnect_zigbee_ncp.yaml --manifest manifests/nabucasa/yellow_bootloader.yaml --manifest manifests/nabucasa/skyconnect_openthread_rcp.yaml --manifest manifests/nabucasa/yellow_openthread_rcp.yaml --manifest manifests/nabucasa/yellow_zigbee_ncp.yaml --manifest manifests/nabucasa/zwave_stick.yaml --manifest manifests/nabucasa/skyconnect_bootloader.yaml --manifest manifests/prpl_foundation/wnc_freedom_openthread_rcp.yaml --manifest manifests/prpl_foundation/wnc_freedom_bootloader.yaml \
"
the input device is not a TTY |
Currently it needs a lot of steps to build single firmware, so lets streamline this workflow by using container. Usage: make [all|build_container|build_firmware] Targets: all Build container and firmware build_container Build container build_firmware Build firmware help Show this help message Options: build_firmware MANIFESTS=<path> Override default manifest files (default: all .yaml/.yml files in manifests/) Examples: # Build the container image make build_container # Build all firmware manifests make build_firmware # Build a specific firmware manifest make build_firmware MANIFESTS=manifests/nabucasa/yellow_bootloader.yaml Signed-off-by: Petr Štetiar <[email protected]>
In commit 3eae968 ("tools: add build_firmware.sh providing firmware build functionality") new build_firmware.sh shell script was added, so lets keep the code quality with reviewdog's shfmt and shellcheck based GitHub actions. Signed-off-by: Petr Štetiar <[email protected]>
Use new build_firmware.sh script in GitHub actions as well, so the build steps are shared with local container based workflow. Signed-off-by: Petr Štetiar <[email protected]>
Currently, the Dockerfile downloads various tools and SDKs from external sources without verifying their integrity. This poses a potential security risk as the downloaded files could be tampered with during transit or at the source (supply chain attack). This change introduces SHA256 checksums for all downloaded artifacts and verifies them before installation. This ensures that the files we receive match exactly what we expect, mitigating the risk of supply chain attacks where malicious actors might try to inject compromised versions of these tools. Signed-off-by: Petr Štetiar <[email protected]>
ynezz
force-pushed
the
ynezz/build-improvements
branch
from
8a7e053 to
c20b59a
Compare
|
just improved the UX by providing |
|
Thanks for the PR! I will give it a thorough review in January. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a new container-based build workflow and several improvements to the build system:
Major Changes
tools/build_firmware.shscript to standardize build stepsNew Build Workflow Features
make-based commands for building firmwareTechnical Improvements
Usage Example
Build all firmware
Build specific manifest
Documentation
Added comprehensive documentation in README.md covering:
Fixes