ci(docker): use Depot for multiarch images #10039

michaelfig · 2024-09-06T16:20:12Z

closes: #9043

Description

Update docker.yml image publishing to use Depot for efficient multiarch builds.

Before (~2h):

After (~15m):

Also, stop building the agoric/agoric-sdk:ibc-alpha tag since it is no longer relevant (only was needed by an IBC test long, long ago).

Security Considerations

Introduces CI dependency on Depot. agoric-3-proposals already uses this successfully, so it is not new to our organisation.

Scaling Considerations

Improves scalability of our CI system (2h Docker multiarch build times are reduced to 15m).

Documentation Considerations

n/a

Testing Considerations

Manually launched Docker builds show that the new CI works correctly.

Upgrade Considerations

n/a

cloudflare-workers-and-pages · 2024-09-06T16:38:17Z

Deploying agoric-sdk with Cloudflare Pages

Latest commit:	`91d2990`
Status:	✅ Deploy successful!
Preview URL:	https://99e7711b.agoric-sdk.pages.dev
Branch Preview URL:	https://mfig-docker-depot.agoric-sdk.pages.dev

View logs

mhofman

This one is flying a bit above my head. One thing however, I don't see an equivalent PR in agoric-3-proposals for the patch here.

.github/workflows/docker.yml

turadg

So far this seems overly complex to me. Let's figure out how to simplify.

turadg · 2024-09-10T17:48:46Z

a3p-integration/.yarn/patches/@agoric-synthetic-chain-npm-0.1.1-d26a740f05.patch

+-  fromTag: "latest"
+  fromTag: "latest",
+  repositoryColon: "ghcr.io/agoric/agoric-3-proposals:",
+  sdkRepositoryColon: "ghcr.io/agoric/agoric-sdk:",


where is this value changed?

or repositoryColon for that matter?

turadg · 2024-09-10T17:48:57Z

a3p-integration/.yarn/patches/@agoric-synthetic-chain-npm-0.1.1-d26a740f05.patch

+   // Must match the Bake file and CI config
+-  fromTag: "latest"
+  fromTag: "latest",
+  repositoryColon: "ghcr.io/agoric/agoric-3-proposals:",


why is the colon necessary to include?

given there are multiple repositories in play, both git and Docker, please give this a more descriptive name. assuming we can drop the colon,

Suggested change

+ repositoryColon: "ghcr.io/agoric/agoric-3-proposals:",

+ a3pDockerRepository: "ghcr.io/agoric/agoric-3-proposals",

turadg · 2024-09-10T17:54:24Z

a3p-integration/.yarn/patches/@agoric-synthetic-chain-npm-0.1.1-d26a740f05.patch

+   RESUME(fromTag) {
+     return `
+ ## RESUME
+-FROM ghcr.io/agoric/agoric-3-proposals:${fromTag} as use-${fromTag}


why are these Docker vars necessary? the Dockerfile is generated by the CI job. Couldn't it just produce the desired file contents?

The buildConfig is part of the package.json. Would you prefer that CI edited that before prepare-build instead of creating a docker-bake.override.json to modify the Dockerfile's behaviour at build time?

yes, very much

I've removed any dependencies on a synthetic-chain patch, and reduced the postprocessing of the prepare-build-generated files to a single sed. Let's talk about the best way to make that change robust.

turadg · 2024-09-12T17:50:55Z

a3p-integration/package.json

@@ -12,7 +12,7 @@
    "doctor": "yarn synthetic-chain doctor"
  },
  "dependencies": {
-    "@agoric/synthetic-chain": "patch:@agoric/synthetic-chain@npm%3A0.1.0#~/.yarn/patches/@agoric-synthetic-chain-npm-0.1.0-148de716a6.patch",
+    "@agoric/synthetic-chain": "^0.1.1",


might as well bump to 0.3.0

.github/workflows/docker.yml

michaelfig · 2024-09-12T23:55:55Z

I extracted the controversial "publish a3p-use-upgrade-next" code into #10083.

mhofman

I don't feel knowledgeable enough with the depot stuff to approve, but from what I'm seeing this look fine to me.

mhofman · 2024-09-13T05:06:04Z

.github/workflows/docker.yml

      - name: Save BUILD_TAG
        run: |
-          ARCH=$(echo '${{ matrix.platform }}' | tr / _)
-          echo "BUILD_TAG=${{ needs.snapshot.outputs.tag }}-$ARCH" >> $GITHUB_ENV


Are we still generating -$ARCH tags? I have used those in the past, but mostly because the arm64 and thus combined image took forever to publish, so possibly this is no longer necessary.

You're right; building and publishing the combined image now takes about the same time as the -linux_amd64 tag used to take. So, no more -$ARCH tags.

turadg

Nice improvement!

LGTM save for moving : into the variable. I trust you'll address before merging

turadg · 2024-09-13T16:38:57Z

.github/workflows/docker.yml

+          PREFIX="$REGISTRY/agoric/cosmic-swingset-solo:"
          for TAG in ${{ needs.docker-sdk.outputs.tags }}; do
-            PREFIXED="$PREFIXED$sep$IMAGE:$TAG"
+            PREFIXED="$PREFIXED$sep$PREFIX$TAG"


why this change? the previous was more clear. please revert or justify.

Project ID file accidentally missing from #10039 which caused the following error in CI: ``` Error: could not configure buildx: unknown project ID (run `depot init` or use --project or $DEPOT_PROJECT_ID) ``` This was confirmed fixed with a manual run of `Docker Build Release Images`.

michaelfig requested review from turadg and mhofman September 6, 2024 16:20

michaelfig marked this pull request as ready for review September 6, 2024 16:20

michaelfig force-pushed the mfig-docker-depot branch from 8817440 to a083021 Compare September 6, 2024 16:38

michaelfig force-pushed the mfig-docker-depot branch from a083021 to 02aa9e2 Compare September 6, 2024 21:00

mhofman reviewed Sep 7, 2024

View reviewed changes

.github/workflows/docker.yml Outdated Show resolved Hide resolved

michaelfig force-pushed the mfig-docker-depot branch 2 times, most recently from 82c3e3f to e7fb573 Compare September 9, 2024 20:54

turadg requested changes Sep 10, 2024

View reviewed changes

michaelfig force-pushed the mfig-docker-depot branch 14 times, most recently from 62d8aaf to d901bda Compare September 12, 2024 17:27

turadg reviewed Sep 12, 2024

View reviewed changes

build(deps): upgrade @agoric/synthetic-chain

eefe3cf

michaelfig changed the title ~~ci(docker): use Depot multiarch, publish use-upgrade-next~~ ci(docker): use Depot for multiarch images Sep 12, 2024

michaelfig force-pushed the mfig-docker-depot branch from d901bda to 5820004 Compare September 12, 2024 23:35

michaelfig requested a review from turadg September 12, 2024 23:53

mhofman reviewed Sep 13, 2024

View reviewed changes

michaelfig requested review from Muneeb147 and usmanmani1122 September 13, 2024 15:55

turadg approved these changes Sep 13, 2024

View reviewed changes

michaelfig added 2 commits September 13, 2024 14:20

ci(docker): use Depot for multiarch images

99b3d93

ci(docker): stop building obsolete ibc-alpha

91d2990

michaelfig force-pushed the mfig-docker-depot branch from 5820004 to 91d2990 Compare September 13, 2024 21:05

michaelfig self-assigned this Sep 13, 2024

michaelfig added tooling repo-wide infrastructure automerge:no-update (expert!) Automatically merge without updates labels Sep 13, 2024

mergify bot merged commit 2abdc28 into master Sep 13, 2024
100 checks passed

mergify bot deleted the mfig-docker-depot branch September 13, 2024 21:41

michaelfig mentioned this pull request Sep 13, 2024

chore(depot): add depot.json project ID file #10087

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci(docker): use Depot for multiarch images #10039

ci(docker): use Depot for multiarch images #10039

michaelfig commented Sep 6, 2024 •

edited

Loading

cloudflare-workers-and-pages bot commented Sep 6, 2024 •

edited

Loading

mhofman left a comment

turadg left a comment

turadg Sep 10, 2024

turadg Sep 10, 2024

turadg Sep 10, 2024

turadg Sep 10, 2024

michaelfig Sep 10, 2024

turadg Sep 10, 2024

michaelfig Sep 12, 2024

turadg Sep 12, 2024

michaelfig commented Sep 12, 2024

mhofman left a comment

mhofman Sep 13, 2024

michaelfig Sep 13, 2024

turadg left a comment

turadg Sep 13, 2024

michaelfig Sep 13, 2024

	+ repositoryColon: "ghcr.io/agoric/agoric-3-proposals:",
	+ a3pDockerRepository: "ghcr.io/agoric/agoric-3-proposals",

ci(docker): use Depot for multiarch images #10039

ci(docker): use Depot for multiarch images #10039

Conversation

michaelfig commented Sep 6, 2024 • edited Loading

Description

Security Considerations

Scaling Considerations

Documentation Considerations

Testing Considerations

Upgrade Considerations

cloudflare-workers-and-pages bot commented Sep 6, 2024 • edited Loading

Deploying agoric-sdk with Cloudflare Pages

mhofman left a comment

Choose a reason for hiding this comment

turadg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelfig commented Sep 12, 2024

mhofman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turadg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelfig commented Sep 6, 2024 •

edited

Loading

cloudflare-workers-and-pages bot commented Sep 6, 2024 •

edited

Loading