Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci(docker): use Depot for multiarch images #10039

Merged
merged 3 commits into from
Sep 13, 2024
Merged

ci(docker): use Depot for multiarch images #10039

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
eefe3cf
build(deps): upgrade `@agoric/synthetic-chain`
michaelfig Sep 12, 2024
99b3d93
ci(docker): use Depot for multiarch images
michaelfig Sep 12, 2024
91d2990
ci(docker): stop building obsolete `ibc-alpha`
michaelfig Sep 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
201 changes: 61 additions & 140 deletions .github/workflows/docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ on:
workflow_dispatch:
env:
REGISTRY: ghcr.io
DOCKER_PLATFORMS: linux/amd64,linux/arm64/v8

jobs:
snapshot:
Expand All @@ -33,109 +34,50 @@ jobs:
SNAPSHOT_TAG="${TIMESTAMP}-${SHORT_SHA}"
echo "tag=$SNAPSHOT_TAG" >> $GITHUB_OUTPUT

docker-parallel-build:
needs: snapshot
runs-on: ubuntu-latest
docker-sdk:
permissions:
# allow issuing OIDC tokens for this workflow run
id-token: write
# allow at least reading the repo contents, add other permissions if necessary
contents: read
# to push the resulting images
packages: write
strategy:
matrix:
platform:
- linux/amd64
- linux/arm64/v8
needs: snapshot
runs-on: 'depot-ubuntu-22.04-16' # ubuntu-latest
outputs:
tag: '${{ steps.docker-tags.outputs.tags }}'
tags: '${{ steps.docker-tags.outputs.tags }} ${{ needs.snapshot.outputs.tag }}'
steps:
- uses: actions/checkout@v4
- uses: depot/setup-action@v1
with:
oidc: true # to set DEPOT_TOKEN for later steps
- run: depot configure-docker
- name: Log in to the Container registry
uses: docker/login-action@v3
# see https://docs.github.com/en/actions/publishing-packages/publishing-docker-images
with:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: ${{ env.REGISTRY }}
- name: Save BUILD_TAG
run: |
ARCH=$(echo '${{ matrix.platform }}' | tr / _)
echo "BUILD_TAG=${{ needs.snapshot.outputs.tag }}-$ARCH" >> $GITHUB_ENV
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we still generating -$ARCH tags? I have used those in the past, but mostly because the arm64 and thus combined image took forever to publish, so possibly this is no longer necessary.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right; building and publishing the combined image now takes about the same time as the -linux_amd64 tag used to take. So, no more -$ARCH tags.

echo "BUILD_TAG=${{ needs.snapshot.outputs.tag }}" >> $GITHUB_ENV
- name: Save GIT_REVISION
run: echo "GIT_REVISION=$(git rev-parse HEAD)" >> $GITHUB_ENV
- name: Save GIT_COMMIT
run: echo "GIT_COMMIT=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
- name: Save commit hash, url of submodules to environment
run: |
node packages/xsnap/src/build.js --show-env >> $GITHUB_ENV
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
with:
buildkitd-flags: --debug
- name: Login to Docker Registry
uses: docker/login-action@v2
with:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: ${{ env.REGISTRY }}
- name: Build and Push ssh-node
uses: docker/build-push-action@v4
uses: depot/build-push-action@v1
with:
file: packages/deployment/Dockerfile.ssh-node
context: packages/deployment/docker
platforms: ${{ matrix.platform }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: true
tags: '${{ env.REGISTRY }}/agoric/ssh-node:${{ env.BUILD_TAG }}'
- name: Build and Push sdk
uses: docker/build-push-action@v4
with:
file: packages/deployment/Dockerfile.sdk
context: ./
platforms: ${{ matrix.platform }}
push: true
tags: '${{ env.REGISTRY }}/agoric/agoric-sdk:${{ env.BUILD_TAG }}'
# When changing/adding entries here, make sure to search the whole
# project for `@@AGORIC_DOCKER_SUBMODULES@@`
build-args: |
GIT_COMMIT=${{env.GIT_COMMIT}}
MODDABLE_COMMIT_HASH=${{env.MODDABLE_COMMIT_HASH}}
MODDABLE_URL=${{env.MODDABLE_URL}}
XSNAP_NATIVE_COMMIT_HASH=${{env.XSNAP_NATIVE_COMMIT_HASH}}
XSNAP_NATIVE_URL=${{env.XSNAP_NATIVE_URL}}
GIT_REVISION=${{env.GIT_REVISION}}
- name: Build and Push setup
uses: docker/build-push-action@v4
with:
file: packages/deployment/Dockerfile
context: packages/deployment
platforms: ${{ matrix.platform }}
tags: '${{ env.REGISTRY }}/agoric/cosmic-swingset-setup:${{ env.BUILD_TAG }}'
push: true
build-args: |
TAG=${{ env.BUILD_TAG }}
- name: notify on failure
if: failure()
uses: ./.github/actions/notify-status
with:
webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
from: ${{ secrets.NOTIFY_EMAIL_FROM }}
to: ${{ secrets.NOTIFY_EMAIL_TO }}
password: ${{ secrets.NOTIFY_EMAIL_PASSWORD }}

# Publish the build's multiarch images to Docker Registry.
docker-sdk:
needs: [docker-parallel-build, snapshot]
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
outputs:
tag: '${{ steps.docker-tags.outputs.tags }}'
tags: '${{ steps.docker-tags.outputs.tags }} ${{ needs.snapshot.outputs.tag }}'
steps:
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
with:
buildkitd-flags: --debug
- name: Login to Docker Registry
uses: docker/login-action@v2
with:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: ${{ env.REGISTRY }}
- name: Compute tags
id: docker-tags
run: |
Expand Down Expand Up @@ -168,67 +110,44 @@ jobs:
DOCKER_TAGS="latest $SDK_TAG"
;;
esac
DOCKER_TAGS="$DOCKER_TAGS $BUILD_TAG"
echo "tags=$DOCKER_TAGS" >> $GITHUB_OUTPUT

- name: Push SDK multiarch
run: |
set -ex
for IMAGE in agoric/agoric-sdk agoric/ssh-node agoric/cosmic-swingset-setup; do
for TAG in ${{ steps.docker-tags.outputs.tags }} ${{ needs.snapshot.outputs.tag }}; do
sources=
for ARCH in linux/amd64 linux/arm64/v8; do
uarch=$(echo "$ARCH" | tr / _)
BUILD_TAG="${{ needs.snapshot.outputs.tag }}-$uarch"
sources="$sources $REGISTRY/$IMAGE:$BUILD_TAG"
done
docker buildx imagetools create --tag "$REGISTRY/$IMAGE:$TAG"$sources
done
done

# This is currently needed for the relayer integration test framework.
# It just runs agoric/agoric-sdk with a "single-node" argument.
docker-ibc-alpha:
needs: [docker-sdk, snapshot]
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
if: ${{ needs.docker-sdk.outputs.tag }} != dev
steps:
- uses: actions/checkout@v4
- name: Save SDK_TAG
run: echo "SDK_TAG=${{ needs.snapshot.outputs.tag }}" >> $GITHUB_ENV
- name: Prefix tags
id: prefix-tags
run: |
IMAGE="$REGISTRY/agoric/agoric-sdk"
for TAG in ibc-alpha; do
for TAG in ${{ steps.docker-tags.outputs.tags }}; do
PREFIXED="$PREFIXED$sep$IMAGE:$TAG"
sep=,
done
echo "tags=$PREFIXED" >> $GITHUB_OUTPUT
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
with:
buildkitd-flags: --debug
- name: Login to Docker Registry
uses: docker/login-action@v2
with:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: ${{ env.REGISTRY }}
- name: Build and Push
uses: docker/build-push-action@v4
- name: Build and Push sdk
uses: depot/build-push-action@v1
with:
file: packages/deployment/Dockerfile.ibc-alpha
context: packages/deployment/docker
platforms: linux/amd64,linux/arm64/v8
file: packages/deployment/Dockerfile.sdk
context: ./
platforms: ${{ env.DOCKER_PLATFORMS }}
push: true
tags: '${{ steps.prefix-tags.outputs.tags }}'
# When changing/adding entries here, make sure to search the whole
# project for `@@AGORIC_DOCKER_SUBMODULES@@`
build-args: |
GIT_COMMIT=${{env.GIT_COMMIT}}
MODDABLE_COMMIT_HASH=${{env.MODDABLE_COMMIT_HASH}}
MODDABLE_URL=${{env.MODDABLE_URL}}
XSNAP_NATIVE_COMMIT_HASH=${{env.XSNAP_NATIVE_COMMIT_HASH}}
XSNAP_NATIVE_URL=${{env.XSNAP_NATIVE_URL}}
GIT_REVISION=${{env.GIT_REVISION}}
- name: Build and Push setup
uses: depot/build-push-action@v1
with:
file: packages/deployment/Dockerfile
context: packages/deployment
platforms: ${{ env.DOCKER_PLATFORMS }}
tags: '${{ env.REGISTRY }}/agoric/cosmic-swingset-setup:${{ env.BUILD_TAG }}'
push: true
build-args: |
SDK_TAG=${{env.SDK_TAG}}
TAG=${{ env.BUILD_TAG }}
- name: notify on failure
if: failure()
uses: ./.github/actions/notify-status
Expand All @@ -240,9 +159,13 @@ jobs:

docker-solo:
needs: [docker-sdk, snapshot]
runs-on: ubuntu-latest
runs-on: 'depot-ubuntu-22.04-16' # ubuntu-latest
permissions:
# allow issuing OIDC tokens for this workflow run
id-token: write
# allow at least reading the repo contents, add other permissions if necessary
contents: read
# to push the resulting images
packages: write
steps:
- uses: actions/checkout@v4
Expand All @@ -257,24 +180,22 @@ jobs:
sep=,
done
echo "tags=$PREFIXED" >> $GITHUB_OUTPUT
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- uses: depot/setup-action@v1
with:
buildkitd-flags: --debug
oidc: true # to set DEPOT_TOKEN for later steps
- run: depot configure-docker
- name: Login to Docker Registry
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: ${{ env.REGISTRY }}
- name: Build and Push
uses: docker/build-push-action@v4
uses: depot/build-push-action@v1
with:
file: packages/solo/Dockerfile
context: packages/solo
platforms: linux/amd64,linux/arm64/v8
platforms: ${{ env.DOCKER_PLATFORMS }}
push: true
tags: '${{ steps.prefix-tags.outputs.tags }}'
build-args: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,7 @@ jobs:
with:
node-version: '18.x'
# Rebuilding the SDK image with resolved endo packages is not currently supported
# and since we can't build core eval submissions form the SDK using a newer endo,
# and since we can't build core eval submissions from the SDK using a newer endo,
# simply ignore any endo branch integration (this means we don't have full coverage)
ignore-endo-branch: 'true'
id: restore-node
Expand Down
13 changes: 0 additions & 13 deletions a3p-integration/.yarn/patches/@agoric-synthetic-chain-npm-0.1.0-148de716a6.patch
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion a3p-integration/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"doctor": "yarn synthetic-chain doctor"
},
"dependencies": {
"@agoric/synthetic-chain": "patch:@agoric/synthetic-chain@npm%3A0.1.0#~/.yarn/patches/@agoric-synthetic-chain-npm-0.1.0-148de716a6.patch",
"@agoric/synthetic-chain": "^0.3.0",
"@types/better-sqlite3": "^7.6.9"
},
"packageManager": "[email protected]",
Expand Down
Loading
Loading