Skip to content

Commit

Permalink
docker: moves everything to consistent docker base layer (#3613)
Browse files Browse the repository at this point in the history
This moves to consistent Alpine 3.18.5 and Java 17.0.9_p8. This updates
test images to the latest minor or patch version of the corresponding
software (e.g. Elasticsearch to latest 6.x). This does not change any
major version, including the JRE. We can do these as a separate task.

Signed-off-by: Adrian Cole <[email protected]>
  • Loading branch information
codefromthecrypt authored Dec 5, 2023
1 parent 2a6b4fb commit d36186e
Show file tree
Hide file tree
Showing 20 changed files with 67 additions and 142 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@ GenericContainer<?> createZipkinContainer(@Nullable GenericContainer<?> storage)

final GenericContainer<?> zipkin;
if (RELEASE_VERSION == null) {
zipkin = new GenericContainer<>(parse("ghcr.io/openzipkin/java:17.0.8_p7"));
zipkin = new GenericContainer<>(parse("ghcr.io/openzipkin/java:17.0.9_p8"));
List<String> classpath = new ArrayList<>();
for (String item : System.getProperty("java.class.path").split(File.pathSeparator)) {
Path path = Paths.get(item);
Expand Down
121 changes: 14 additions & 107 deletions build-bin/README.md
100755 → 100644
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ On deploy:
`build-bin` holds portable scripts used in CI to test and deploy the project.

The scripts here are portable. They do not include any CI provider-specific logic or ENV variables.
This helps `.travis.yml` and `test.yml` (GitHub Actions) contain nearly the same contents, even if
This helps `test.yml` (GitHub Actions) and alternatives contain nearly the same contents, even if
certain OpenZipkin projects need slight adjustments here. Portability has proven necessary, as
OpenZipkin has had to transition CI providers many times due to feature and quota constraints.

Expand Down Expand Up @@ -66,9 +66,6 @@ the scripts it uses.
The `on:` section obviates job creation and resource usage for irrelevant events. Notably, GitHub
Actions includes the ability to skip documentation-only jobs.

Combine [configure_test] and [test] into the same `run:` when `configure_test` primes file system
cache.

Here's a partial `test.yml` including only the aspects mentioned above.
```yaml
on:
Expand All @@ -84,54 +81,13 @@ jobs:
test:
steps:
- name: Checkout Repository
uses: actions/checkout@v2
uses: actions/checkout@v4
with:
fetch-depth: 0 # full git history
fetch-depth: 0 # full git history for license check
- name: Test
run: build-bin/configure_test && build-bin/test
```
### Example Travis setup
`.travis.yml` is a monolithic configuration file broken into stages, of which the default is "test".
A simplest Travis `test` job configures tests in `install` and runs them as `script`, but only on
relevant event conditions.

The `if:` section obviates job creation and resource usage for irrelevant events. Travis does not
support file conditions. A `before_install` step to skip documentation-only commits will likely
complete in less than a minute (10 credit cost).

Here's a partial `.travis.yml` including only the aspects mentioned above.
```yaml
git:
depth: false # TRAVIS_COMMIT_RANGE requires full commit history.
jobs:
include:
- stage: test
if: branch = master AND tag IS blank AND type IN (push, pull_request)
name: Run unit and integration tests
before_install: | # Prevent test build of a documentation-only change.
if [ -n "${TRAVIS_COMMIT_RANGE}" ] && ! git diff --name-only "${TRAVIS_COMMIT_RANGE}" -- | grep -qv '\.md$'; then
echo "Stopping job as changes only affect documentation (ex. README.md)"
travis_terminate 0
fi
install: ./build-bin/configure_test
script: ./build-bin/test
```

When Travis only runs tests (something else does deploy), there's no need to use stages:
```yaml
git:
depth: false # TRAVIS_COMMIT_RANGE requires full commit history.
if: branch = master AND tag IS blank AND type IN (push, pull_request)
before_install: | # Prevent test build of a documentation-only change.
if [ -n "${TRAVIS_COMMIT_RANGE}" ] && ! git diff --name-only "${TRAVIS_COMMIT_RANGE}" -- | grep -qv '\.md$'; then
echo "Stopping job as changes only affect documentation (ex. README.md)"
travis_terminate 0
fi
install: ./build-bin/configure_test
script: ./build-bin/test
run: |
build-bin/configure_test
build-bin/test
```
## Deploy
Expand All @@ -154,77 +110,28 @@ The `on:` section obviates job creation and resource usage for irrelevant events
cannot implement "master, except documentation only-commits" in the same file. Hence, deployments of
master will happen even on README change.

Combine [configure_deploy] and [deploy] into the same `run:` when `configure_deploy` primes file
system cache.

Here's a partial `deploy.yml` including only the aspects mentioned above. Notice env variables are
explicitly defined and `on.tags` is a [glob pattern](https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet).

```yaml
on:
push:
tags: '[0-9]+.[0-9]+.[0-9]+**' # Ex. 8.272.10 or 17.0.8_p7
tags: '[0-9]+.[0-9]+.[0-9]+**' # e.g. 8.272.10 or 15.0.1_p9
branches: master
jobs:
deploy:
steps:
- name: Checkout Repository
uses: actions/checkout@v2
uses: actions/checkout@v4
with:
fetch-depth: 1 # only needed to get the sha label
- name: Deploy
- name: Configure Deploy
run: build-bin/configure_deploy
env:
GH_USER: ${{ secrets.GH_USER }}
GH_TOKEN: ${{ secrets.GH_TOKEN }}
run: | # GITHUB_REF will be refs/heads/master or refs/tags/MAJOR.MINOR.PATCH
build-bin/configure_deploy &&
build-bin/deploy $(echo ${GITHUB_REF} | cut -d/ -f 3)
```

### Example Travis setup
`.travis.yml` is a monolithic configuration file broken into stages. This means `test` and `deploy`
are in the same file. A simplest Travis `deploy` stage has two jobs: one for master pushes and
another for version tags. These jobs are controlled by event conditions.

The `if:` section obviates job creation and resource usage for irrelevant events. Travis does not
support file conditions. A `before_install` step to skip documentation-only commits will likely
complete in less than a minute (10 credit cost).

As billing is by the minute, it is most cost effective to combine test and deploy on master push.

Here's a partial `.travis.yml` including only the aspects mentioned above. Notice YAML anchors work
in Travis and `tag =~` [condition](https://github.com/travis-ci/travis-conditions) is a regular
expression.
```yaml
git:
depth: false # full git history for license check, and doc-only skipping
_terminate_if_only_docs: &terminate_if_only_docs |
if [ -n "${TRAVIS_COMMIT_RANGE}" ] && ! git diff --name-only "${TRAVIS_COMMIT_RANGE}" -- | grep -qv '\.md$'; then
echo "Stopping job as changes only affect documentation (ex. README.md)"
travis_terminate 0
fi
jobs:
include:
- stage: test
if: branch = master AND tag IS blank AND type IN (push, pull_request)
before_install: *terminate_if_only_docs
install: |
if [ "${TRAVIS_SECURE_ENV_VARS}" = "true" ] && [ "${TRAVIS_PULL_REQUEST}" = "false" ]; then
export SHOULD_DEPLOY=true
./build-bin/configure_deploy
else
export SHOULD_DEPLOY=false
./build-bin/configure_test
fi
script:
- ./build-bin/test || travis_terminate 1
- if [ "${SHOULD_DEPLOY}" != "true" ]; then travis_terminate 0; fi
- travis_wait ./build-bin/deploy master
- stage: deploy
# Ex. 8.272.10 or 17.0.8_p7
if: tag =~ /^[0-9]+\.[0-9]+\.[0-9]+/ AND type = push AND env(GH_TOKEN) IS present
install: ./build-bin/configure_deploy
script: ./build-bin/deploy ${TRAVIS_TAG}
- name: Deploy
# GITHUB_REF will be refs/heads/master or refs/tags/1.2.3
run: build-bin/deploy $(echo ${GITHUB_REF} | cut -d/ -f 3)
```
7 changes: 5 additions & 2 deletions build-bin/docker/configure_docker_push
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/sh
#
# Copyright 2015-2020 The OpenZipkin Authors
# Copyright 2015-2021 The OpenZipkin Authors
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
Expand Down Expand Up @@ -50,5 +50,8 @@ fi
# See https://github.com/multiarch/qemu-user-static
#
# Mirrored image use to avoid docker.io pulls:
# docker tag multiarch/qemu-user-static:5.1.0-7 ghcr.io/openzipkin/multiarch-qemu-user-static:latest
# docker tag multiarch/qemu-user-static:7.2.0-1 ghcr.io/openzipkin/multiarch-qemu-user-static:latest
#
# Note: This image only works on x86_64/amd64 architecture.
# See: https://github.com/multiarch/qemu-user-static#supported-host-architectures
docker run --rm --privileged ghcr.io/openzipkin/multiarch-qemu-user-static --reset -p yes
5 changes: 5 additions & 0 deletions build-bin/docker/docker_arch
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@
set -ue

# Normalize docker_arch to what's available
#
# Note: s390x and ppc64le were added for Knative
docker_arch=${DOCKER_ARCH:-$(uname -m)}
case ${docker_arch} in
amd64* )
Expand All @@ -38,6 +40,9 @@ case ${docker_arch} in
s390x* )
docker_arch=s390x
;;
ppc64le* )
docker_arch=ppc64le
;;
* )
>&2 echo "Unsupported DOCKER_ARCH: ${docker_arch}"
exit 1;
Expand Down
12 changes: 6 additions & 6 deletions build-bin/docker/docker_args
Original file line number Diff line number Diff line change
Expand Up @@ -46,21 +46,21 @@ if [ -n "${DOCKER_TARGET}" ]; then
fi

# When non-empty, becomes the base layer including tag appropriate for the image being built.
# Ex. ghcr.io/openzipkin/java:17.0.8_p7-jre
# e.g. ghcr.io/openzipkin/java:21.0.1_p12-jre
#
# This is not required to be a base (FROM scratch) image like ghcr.io/openzipkin/alpine:3.14.2
# This is not required to be a base (FROM scratch) image like ghcr.io/openzipkin/alpine:3.12.3
# See https://docs.docker.com/glossary/#parent-image
if [ -n "${DOCKER_PARENT_IMAGE}" ]; then
docker_args="${docker_args} --build-arg docker_parent_image=${DOCKER_PARENT_IMAGE}"
fi

# When non-empty, becomes the build-arg alpine_version. Ex. "3.18.2"
# When non-empty, becomes the build-arg alpine_version. e.g. "3.12.3"
# Used to align base layers from https://github.com/orgs/openzipkin/packages/container/package/alpine
if [ -n "${ALPINE_VERSION}" ]; then
docker_args="${docker_args} --build-arg alpine_version=${ALPINE_VERSION}"
fi

# When non-empty, becomes the build-arg java_version. Ex. "17.0.8_p7"
# When non-empty, becomes the build-arg java_version. e.g. "21.0.1_p12"
# Used to align base layers from https://github.com/orgs/openzipkin/packages/container/package/java
if [ -n "${JAVA_VERSION}" ]; then
docker_args="${docker_args} --build-arg java_version=${JAVA_VERSION}"
Expand All @@ -69,13 +69,13 @@ if [ -n "${JAVA_VERSION}" ]; then
java_major_version=$(echo ${JAVA_VERSION}| cut -f1 -d .)
case ${java_major_version} in
8) java_home=/usr/lib/jvm/java-1.8-openjdk;;
1?) java_home=/usr/lib/jvm/java-${java_major_version}-openjdk;;
1?|2?|3?) java_home=/usr/lib/jvm/java-${java_major_version}-openjdk;;
esac

if [ -n "${java_home}" ]; then docker_args="${docker_args} --build-arg java_home=${java_home}"; fi
fi

# When non-empty, becomes the build-arg maven_classifier. Ex. "module" or "exec"
# When non-empty, becomes the build-arg maven_classifier. e.g. "module" or "exec"
# Used as the classifier arg to ./build-bin/maven/maven_unjar. Allows building two images with the
# same Dockerfile, varying on classifier, like openzipkin/zipkin vs openzipkin/zipkin-slim
if [ -n "${MAVEN_CLASSIFIER}" ]; then
Expand Down
2 changes: 1 addition & 1 deletion build-bin/docker/docker_build
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ version=${2:-}
docker_args=$($(dirname "$0")/docker_args ${version})

echo "Building image ${docker_tag}"
DOCKER_BUILDKIT=1 docker build --pull ${docker_args} --tag ${docker_tag} .
DOCKER_BUILDKIT=1 docker build --network=host --pull ${docker_args} --tag ${docker_tag} .
4 changes: 2 additions & 2 deletions build-bin/docker/docker_push
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,8 @@ for repo in ${docker_repos}; do
done

docker_args=$($(dirname "$0")/docker_args ${version})
docker_archs=${DOCKER_ARCHS:-amd64 arm64}
# Note: s390x and ppc64le were added for Knative
docker_archs=${DOCKER_ARCHS:-amd64 arm64 s390x ppc64le}

echo "Will build the following architectures: ${docker_archs}"

Expand Down Expand Up @@ -114,4 +115,3 @@ else
docker manifest push -p ${tag}
done
fi

7 changes: 7 additions & 0 deletions build-bin/docker_push
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,18 @@ case ${version} in
;;
esac

# Don't attempt ppc64le until there's a package for recent LTS versions.
# See https://github.com/openzipkin/zipkin/issues/3443
export DOCKER_ARCHS="amd64 arm64 s390x"

build-bin/docker/docker_push openzipkin/zipkin ${version}
DOCKER_TARGET=zipkin-slim build-bin/docker/docker_push openzipkin/zipkin-slim ${version}

# testing images only push to ghcro.io
export DOCKER_RELEASE_REPOS=ghcr.io
# Don't attempt unfamiliar archs on test images
export DOCKER_ARCHS="amd64 arm64"

for name in $(ls docker/test-images/*/Dockerfile|cut -f3 -d/); do
DOCKER_FILE=docker/test-images/${name}/Dockerfile build-bin/docker/docker_push openzipkin/${name} ${version}
done
2 changes: 1 addition & 1 deletion build-bin/maven/maven_unjar
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ fi

if ! test -f ${artifact_id}.jar && [ ${is_release} = "true" ]; then
mvn_get="mvn -q --batch-mode -Denforcer.fail=false \
org.apache.maven.plugins:maven-dependency-plugin:3.1.2:get \
org.apache.maven.plugins:maven-dependency-plugin:3.6.1:get \
-Dtransitive=false -DgroupId=${group_id} -DartifactId=${artifact_id} -Dversion=${version}"

if [ -n "${classifier}" ]; then
Expand Down
4 changes: 2 additions & 2 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Copyright 2015-2021 The OpenZipkin Authors
# Copyright 2015-2023 The OpenZipkin Authors
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
Expand All @@ -17,7 +17,7 @@
# Use latest version here: https://github.com/orgs/openzipkin/packages/container/package/java
# This is defined in many places because Docker has no "env" script functionality unless you use
# docker-compose: When updating, update everywhere.
ARG java_version=17.0.8_p7
ARG java_version=17.0.9_p8

# We copy files from the context into a scratch container first to avoid a problem where docker and
# docker-compose don't share layer hashes https://github.com/docker/compose/issues/883 normally.
Expand Down
2 changes: 1 addition & 1 deletion docker/test-images/zipkin-cassandra/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
# Use latest version here: https://github.com/orgs/openzipkin/packages/container/package/java
# This is defined in many places because Docker has no "env" script functionality unless you use
# docker-compose: When updating, update everywhere.
ARG java_version=17.0.8_p7
ARG java_version=17.0.9_p8

# We copy files from the context into a scratch container first to avoid a problem where docker and
# docker-compose don't share layer hashes https://github.com/docker/compose/issues/883 normally.
Expand Down
2 changes: 1 addition & 1 deletion docker/test-images/zipkin-cassandra/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ cat > pom.xml <<-'EOF'
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.30</version>
<version>1.7.36</version>
</dependency>
</dependencies>
</project>
Expand Down
8 changes: 4 additions & 4 deletions docker/test-images/zipkin-elasticsearch6/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Copyright 2015-2021 The OpenZipkin Authors
# Copyright 2015-2023 The OpenZipkin Authors
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
Expand All @@ -17,7 +17,7 @@
# Use latest version here: https://github.com/orgs/openzipkin/packages/container/package/java
# This is defined in many places because Docker has no "env" script functionality unless you use
# docker-compose: When updating, update everywhere.
ARG java_version=17.0.8_p7
ARG java_version=17.0.9_p8

# We copy files from the context into a scratch container first to avoid a problem where docker and
# docker-compose don't share layer hashes https://github.com/docker/compose/issues/883 normally.
Expand All @@ -35,7 +35,7 @@ WORKDIR /install
# Use latest 6.x version from https://www.elastic.co/downloads/past-releases#elasticsearch
# This is defined in many places because Docker has no "env" script functionality unless you use
# docker-compose: When updating, update everywhere.
ARG elasticsearch6_version=6.8.13
ARG elasticsearch6_version=6.8.23

# Download only the OSS distribution (lacks X-Pack)
RUN \
Expand All @@ -49,7 +49,7 @@ COPY --from=scratch /config/ ./config/

FROM ghcr.io/openzipkin/java:${java_version}-jre as zipkin-elasticsearch6
LABEL org.opencontainers.image.description="Elasticsearch OSS distribution on OpenJDK and Alpine Linux"
ARG elasticsearch6_version=6.8.13
ARG elasticsearch6_version=6.8.23
LABEL elasticsearch-version=$elasticsearch6_version

# Add HEALTHCHECK and ENTRYPOINT scripts into the default search path
Expand Down
Loading

0 comments on commit d36186e

Please sign in to comment.