Skip to content

Commit

Permalink
Set up Docker test job to allow publishing release images (chapel-lan…
Browse files Browse the repository at this point in the history
…g#26046)

Refactor the Docker nightly testing script to optionally also push
release-tagged images.

Configured by the `RELEASE_VERSION` environment variable, assumed to get
set via Jenkins parameter. If set, the script pushes the images tagged
as `latest` and `$RELEASE_VERSION`. All nightly build/pushes still run
before release-tagged pushes, so we don't push a release if any image
variants are broken. Script aborts without building anything if
`RELEASE_VERSION` is set but we're not on the appropriate release
branch.

As part of this PR I attempted to change our process to test the image
before pushing, rather than build, push, then test. However, on
`chapelmac-m1` this ran into docker/buildx#59,
so I reverted it. Since we only push release images after all nightly
images, we're still safe from pushing a broken release-tagged image.
Noted this limitation and the requirement to build nightly images first
in a comment.

Also includes:
- delete unused `util/cron/publish-docker-images.bash` which didn't
contain as much functionality as `test-docker.bash`
- more comments and some small refactors for clarity

Resolves Cray/chapel-private#6743.

[reviewed by @tzinsky , thanks!]

Associated pre-merge tasks:
- [x] merge corresponding CI config adjustments PR
https://github.hpe.com/hpe/hpc-chapel-ci-config/pull/1291
- [x] disable unused previous job
https://chapel-ci.us.cray.com/job/publish-docker-images/
- [x] update Docker release best-practices to tell you to use this
script
(https://github.hpe.com/hpe/hpc-chapel-docs/commit/d159f7e8a0be122660cd8af39e3b49ddb8b59486)
- [x] delete temporarily private Docker repositories created for testing
(`chapel-test{,-gasnet,-gasnet-smp}`)

Testing:
- [x] manual run in non-release mode still works
- [x] manual run in release mode (temporarily modified to push to a
scratch repo)
riftEmber authored Oct 10, 2024
2 parents 9547ee1 + 3a3f38d commit 5f55606
Showing 2 changed files with 114 additions and 114 deletions.
79 changes: 0 additions & 79 deletions util/cron/publish-docker-images.bash

This file was deleted.

149 changes: 114 additions & 35 deletions util/cron/test-docker.bash
Original file line number Diff line number Diff line change
@@ -1,68 +1,147 @@
#!/usr/bin/env bash
#
# This script will build, run, and verify chapel, gasnet, and gasnet-smp docker images.
# This script will build, (sanity) test, and push the chapel, chapel-gasnet, and
# chapel-gasnet-smp Docker images. Uses the 'nightly' tag by default, or
# 'latest' and a release version tag if specified.
#
# Assumes Docker is already running on the system, logged into an account with
# appropriate permissions to push the images.
#
# Expected environment variables:
# - RELEASE_VERSION (optional): If set, will also push the image tagged as
# 'latest' and this version. Should match version in release branch name.


CWD=$(cd $(dirname $0) ; pwd)
source $CWD/common.bash
export CHPL_HOME=$(cd $CWD/../.. ; pwd)
log_info "Setting CHPL_HOME to: ${CHPL_HOME}"
export CHPL_NIGHTLY_TEST_CONFIG_NAME="docker"

# build_image function takes image name and docker script location as arguments.
# Builds the image with the name from arg$1, runs the container and execute the install and verify script located in the location $2.
build_image() {
local imageName="$1"
# BEGIN FUNCTIONS

# Patch the Dockerfile to build FROM the nightly image instead of latest.
# Assumes the Dockerfile is available at ./Dockerfile.
# Arguments are forwarded to `patch` command.
dockerfile_nightly_patch() {
local patch_args="$*"

local nightlypatch="
1c1
< FROM chapel/chapel:latest
---
> FROM chapel/chapel:nightly
"

patch $patch_args ./Dockerfile << EOF
$nightlypatch
EOF
}

# Build, test, and push a Docker image.
# Args:
# - image name without tag
# - test script location
# - release version tag to use (optional, builds nightly otherwise)
update_image() {
local baseImageName="$1"
local script="$2"
# Remove any existing image with the tag before building docker image
docker image rm --force $imageName
local release_tag="$3"

# Use specified release version tag, or 'nightly' if not specified
local imageName="${baseImageName}:${release_tag:-nightly}"

log_info "Starting $imageName..."

# Remove any existing image with the tag before building
if [ -n "$release_tag" ]
then
docker image rm --force "$imageName"
fi

docker buildx build --platform=linux/amd64,linux/arm64 . --push -t $imageName
# Build and push image
# Note: We push before testing due to a limitation of Docker
# (https://github.com/docker/buildx/issues/59) which prevents loading a
# multi-arch image without pushing. This means we may push a broken nightly
# image before erroring out; it's important that release pushes come after
# all nightly pushes so we can't push a broken release image.
# Anna, 2024-10-07
docker buildx build --platform=linux/amd64,linux/arm64 --push . -t "$imageName"
BUILD_RESULT=$?
# Also push as 'latest' tag if this is a release build
if [ -n "$release_tag" ]
then
# Use base image name (without tag) to use Docker's default tag 'latest'
docker buildx build --platform=linux/amd64,linux/arm64 --push . -t "$baseImageName"
fi

if [ $BUILD_RESULT -ne 0 ]
then
echo "docker build failed for $imageName image"
exit 1
fi

containerid= docker image ls | grep $imageName | awk '{print$3}'
cd ${CHPL_HOME}/util/cron
# Run test script inside container
echo 'writeln("Hello, world!");' > hello.chpl

docker run --rm -i $imageName < $script

docker run --rm -i "$imageName" < "$script"
CONTAINER_RUN=$?
# Clean up scratch chpl file for testing
rm hello.chpl

if [ $CONTAINER_RUN -ne 0 ]
then
echo "docker commands failed inside chapel $imageName container"
exit 1
else
echo "docker commands succeeded inside chapel $imageName container"
fi

log_info "Completed $imageName"
}

# Patch the Dockerfile to build FROM the nightly image instead of latest.
# Assumes the Dockerfile is available at ./Dockerfile.
dockerfile_nightly_patch() {
local nightlypatch="
1c1
< FROM chapel/chapel:latest
---
> FROM chapel/chapel:nightly
"
patch ./Dockerfile << EOF
$nightlypatch
EOF
# Build, test, and push all Chapel Docker images.
# Args:
# - release version tag to use (optional, builds nightly otherwise)
update_all_images() {
local release_tag="$1"

cd "$CHPL_HOME"
update_image chapel/chapel "${CHPL_HOME}/util/cron/docker-chapel.bash" "$release_tag"

cd "$CHPL_HOME/util/packaging/docker/gasnet"
dockerfile_nightly_patch
update_image chapel/chapel-gasnet "${CHPL_HOME}/util/cron/docker-gasnet.bash" "$release_tag"
# Clean up after patch changes
dockerfile_nightly_patch -R

cd "$CHPL_HOME/util/packaging/docker/gasnet-smp"
dockerfile_nightly_patch
update_image chapel/chapel-gasnet-smp "${CHPL_HOME}/util/cron/docker-gasnet.bash" "$release_tag"
# Clean up after patch changes
dockerfile_nightly_patch -R
}
# END FUNCTIONS

# Build chapel Docker images
cd $CHPL_HOME
build_image chapel/chapel:nightly ${CHPL_HOME}/util/cron/docker-chapel.bash

cd $CHPL_HOME/util/packaging/docker/gasnet
dockerfile_nightly_patch
build_image chapel/chapel-gasnet:nightly ${CHPL_HOME}/util/cron/docker-gasnet.bash
if [ -n "$RELEASE_VERSION" ]
then
log_info "Building and pushing nightly and release-tagged images for version: $RELEASE_VERSION"
release_branch="release/$RELEASE_VERSION"
if [ "$(git rev-parse --abbrev-ref HEAD)" != "$release_branch" ]
then
log_error "Not on expected release branch $release_branch for version $RELEASE_VERSION, aborting"
exit 1
fi
else
log_info "Building and pushing nightly images"
fi

cd $CHPL_HOME/util/packaging/docker/gasnet-smp
dockerfile_nightly_patch
build_image chapel/chapel-gasnet-smp:nightly ${CHPL_HOME}/util/cron/docker-gasnet.bash
# Build and push nightly images
update_all_images

export CHPL_NIGHTLY_TEST_CONFIG_NAME="docker"
# Build and push release-tagged images, if RELEASE_VERSION was specified.
# Runs after all nightly images, to abort if any fail.
if [ -n "$RELEASE_VERSION" ]
then
update_all_images "$RELEASE_VERSION"
fi

0 comments on commit 5f55606

Please sign in to comment.