Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update ngen Docker image to build t-route master branch #429

Merged
merged 14 commits into from
Sep 18, 2023

Conversation

robertbartel
Copy link
Contributor

@robertbartel robertbartel commented Sep 13, 2023

Switch to building/installing t-route master instead of t-route ngen branch. Very similar to changes from #318, but with some modifications based on initial review and evolution of the repo since.

Additions

  • Several small build system improvements including:
    • using ${BUILD_PARALLEL_JOBS} instead of $(nproc)
    • an improved way to optionally pass a pre-downloaded boost tarball to the image creation process, without reducing Docker build cache utilization
    • remove of the requirement for t-route repo/branch values to be explicitly provided in the .env file

Removals

  • Does not work out of the box to use legacy t-route (e.g. ngen branch), which might limit capability with some older hydrofabric datasets.

Changes

  • Now compiles and builds master t-route by default (instead of ngen branch)
  • Several Python elements are now installed in the global (root) site. This enables use-cases where a user other than mpi(1000) is used, e.g. using docker run --user=$(uid -u), so now the routing modules are available to that user

mattw-nws and others added 13 commits September 13, 2023 10:24
…er python modules will be installed there (instead of in mpi's user site), and can be readily used if the container is launched with uid=X... a useful scenario for reusing this container image.
Updating docker-build.yml for main stack compose config to default to
official OWP repo and "master" branch in relevant t-route build ARGs,
instead of failing with an error message if they were not set within the
.env config.
Adding dummy file to help avoid copy issues during Docker image builds
if there is no Boost tar file present, while allowing the build to
take advantage of a local Boost tar file when it is.
Taking advantage of new dummy file, rather than the image's entrypoint
(which would be prone to change and thus reduce build cache hits) to
help avoid copy issues during Docker image builds if there is no Boost
tar file present, while allowing the build to take advantage of a local
Boost tar file when it is.
- Create symlinks for python/pip from python3/pip3
- Run dnf clean all
- Use --no-cache-dir with pip installs
Separating layers handling Python dependencies for troute that needed
some of the from-source dependencies.
@robertbartel robertbartel added enhancement New feature or request maas MaaS Workstream labels Sep 13, 2023
Fixing issue where main ngen worker and calibration images would not
necessarily be built using the same ngen branch or commit due to the
Docker compose build config not applying all ngen .env-sourced ARGS to
the calibration image build.
Copy link
Member

@aaraney aaraney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@robertbartel robertbartel merged commit f1050f1 into NOAA-OWP:master Sep 18, 2023
1 check failed
@robertbartel robertbartel deleted the f/troute_master/1/rebase branch September 18, 2023 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request maas MaaS Workstream
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants