Use new docker build provider #1278

blampe · 2024-04-26T22:31:37Z

This replaces the old Docker provider with the new Docker Build provider.

User-facing AWSX types are unchanged.

Behavior is unchanged, except we will now automatically store and use an inline cache if the user hasn't configured anything explicitly.

Fixes #1349
Fixes #1072

lukehoban

SDK types are unchanged -- those remain the old Docker API.

The @pulumi/docker typess don't show up in the public API of this component, right? So what do you mean that "SDK types are unchanged"? I assume we can remove the dependency on @pulumi/docker from both the implementation, and update the implied depdendency in the schema (which I assume is only to hint at the internal dependnecy - even though not strictly needed?). Feel like maybe I'm missing something that motivated this comment though?

lukehoban · 2024-05-07T23:44:49Z

awsx/package.json

@@ -22,8 +22,9 @@
  },
  "//": "Pulumi sub-provider dependencies must be pinned at an exact version because we extract this value to generate the correct dependency in the schema",
  "dependencies": {
-    "@pulumi/aws": "6.32.0",
+    "@pulumi/aws": "6.9.0",
    "@pulumi/docker": "4.5.1",


Can we remove this dependency?

I think removing this will break anyone who indirectly depends on it -- is that OK? In other words I can specify only awsx in package.json but still use docker in my program.

blampe · 2024-08-09T17:35:38Z

So what do you mean that "SDK types are unchanged"?

Sorry, this meant that awsx's own wrapper types around Docker are unchanged. Only the internals are changing, the user-facing API remains the same.

flostadler

LGTM, just two small questions.
I was thinking that we should add an upgrade test for going from the old to the new docker provider, what do you think?

flostadler · 2024-08-19T15:57:30Z

awsx/ecr/image.ts

+    tags: [canonicalImageName],
+    buildArgs: dockerInputs.args,
+    cacheFrom: cacheFrom,
+    cacheTo: [{ inline: {} }],


Do you know if this will increase image sizes substantially? IIRC this is only metadata about the layers so the size increase shouldn't be substantial.

Great question! It's actually full cache artifacts, so it can increase the size a bit. I think it would be best to take this out and stay as close to the existing behavior of not doing any caching stuff unless cacheFrom is set.

However this makes me realize we probably do want inline caching when cacheFrom is set. Currently it's using a registry cache output, but this isn't compatible with the existing behavior because users currently point this at an image (not a cache artifact) due to the way the old Docker provider did caching. (It would simply pull images to get layers into your local build cache.) In other words we don't want to export a cache to image:latest, but it would be reasonable to push an image with inline caching.

Caching Docker images is hard enough as it is and the old provider's behavior was also easy to get wrong. For example cacheFrom might not do anything without a magic env var:

new docker.Image("my-app-image", { build: { args: { BUILDKIT_INLINE_CACHE: "1", }, cacheFrom: { images: ["foo:latest"], }, context: "app/", dockerfile: "app/Dockerfile", }, imageName: "foo:latest", });

So for us to stay as close to the existing behavior while also allowing caching to somewhat work, I think we basically just want to enable this BUILDKIT_INLINE_CACHE behavior when cacheFrom is present.

I tried a couple of different sample images and there was no substantial increase in image size. For some the image was even smaller. So the current approach sounds good to me

flostadler · 2024-08-19T16:04:47Z

awsx/ecr/image.ts

+  if (dockerInputs.context !== undefined) {
+    context = dockerInputs.context;
+  }
+
  const dockerImageArgs: docker.ImageArgs = {


Are there any new input properties we should expose as well?

Probably! There's a lot more we can do now, e.g. multiple tags, multiple platforms, more refined caching behavior, etc.

I didn't want to add any of those right now because (a) these are opinionated APIs and opinions take more time to get concensus on, and (b) it's not clear how much (if any) we want to continue investing in this wrapper versus giving users more composable building blocks (like your example of registry auth helpers). The latter is my preference, but it's not my call to make.

t0yv0 · 2024-11-06T16:50:31Z

Are we going forward with this @blampe or should we pick up on AWS squad?

What is your confidence that the upgrade is going to be "invisible" to users? Does new Docker Build provider Image have an alias to the old image and is ready to accept old image states and rewire them? Or is there going to be a replacement during upgrade but we believe it is benign?

I was thinking that we should add an upgrade test for going from the old to the new docker provider, what do you think?

This would be good but I also wonder if we know enough to cover the upgrade space well enough to test it properly.

Perhaps instead we can just include this change in the next major version?

t0yv0 · 2024-11-06T16:51:01Z

This likely will also fix #1401

flostadler · 2024-12-23T21:25:59Z

@t0yv0 In my opinion we shouldn't wait for the next major with this. Even if the image does get rebuilt it shouldn't be a problem. It's not a breaking change like infrastructure being replaced.

As part of another change I'm going to add a function to the provider for retrieving ECR creds. If there's users that do not want to go with docker-build, they could use that and pipe it into the docker.Image resource.

flostadler · 2025-01-02T14:41:11Z

I resolved the merge conflicts with the main branch, updated to the latest docker-build (0.0.8) and added an upgrade test. I think this is good to go now!

flostadler · 2025-01-02T16:03:59Z

@blampe I updated the PR, could you have a look as well?

blampe · 2025-01-04T05:43:17Z

awsx/package.json

@@ -25,7 +25,7 @@
  "//": "Pulumi sub-provider dependencies must be pinned at an exact version because we extract this value to generate the correct dependency in the schema",
  "dependencies": {
    "@pulumi/aws": "6.65.0",
-    "@pulumi/docker": "4.5.1",
+    "@pulumi/docker-build": "0.0.8",


Is this possibly a breaking change? Like if I pull in p/docker indirectly via awsx and then reference it from my code, my imports would break?

I don't know if that's what would actually happen or if it's avoidable. Would you leave in the old p/docker dependency even if it's unused?

Mostly just asking to make sure we've thought through any edge cases.

IMO it's not a breaking change. Users shouldn't depend on transitive dependencies directly. They should add those to their own dependencies.

That being said, I'm currently workin on another awsx feature that'll add back the docker provider for push without build scenarios so we're gonna be covered here

blampe · 2025-01-04T05:44:41Z

@blampe I updated the PR, could you have a look as well?

@flostadler it looks great to me! Just one small question re: dependencies but not blocking. Feel free to merge whenever you want!

pulumi-bot · 2025-01-14T14:30:58Z

This PR has been shipped in release v2.20.0.

blampe changed the title ~~docker-build~~ Use new docker build provider Apr 26, 2024

blampe marked this pull request as draft April 26, 2024 22:32

lukehoban reviewed May 7, 2024

View reviewed changes

mjeffryes assigned blampe May 10, 2024

blampe added 3 commits August 7, 2024 12:16

docker-build

8859481

fix dependencies

1848695

regenerate schema + sdk

c1c5778

blampe force-pushed the buildx branch from b9b562f to c1c5778 Compare August 9, 2024 16:09

blampe requested a review from flostadler August 9, 2024 17:35

blampe marked this pull request as ready for review August 9, 2024 17:35

mjeffryes added this to the 0.108 milestone Aug 16, 2024

flostadler reviewed Aug 19, 2024

View reviewed changes

mikhailshilkov removed this from the 0.108 milestone Aug 22, 2024

mjeffryes modified the milestones: 0.109, 0.110 Sep 12, 2024

mjeffryes removed this from the 0.110 milestone Oct 2, 2024

flostadler added 3 commits January 2, 2025 13:02

Merge branch 'master' into buildx

08bdb05

Update docker-build to 0.0.8

17d5ada

Add upgrade test

32919e8

flostadler requested review from t0yv0 and corymhall January 2, 2025 14:41

corymhall approved these changes Jan 2, 2025

View reviewed changes

Correct versions from faulty merge

d45f1c6

Merge branch 'master' into buildx

c4e12fe

blampe commented Jan 4, 2025

View reviewed changes

flostadler merged commit fe08e89 into master Jan 6, 2025
11 checks passed

flostadler deleted the buildx branch January 6, 2025 12:19

flostadler mentioned this pull request Jan 8, 2025

Fix breaking change in imageUri output of ecr.Image #1467

Merged

This was referenced Jan 14, 2025

Update awsx to use docker-build #1349

Closed

ecr.NewImage CacheFrom adds --target to docker build #1072

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use new docker build provider #1278

Use new docker build provider #1278

blampe commented Apr 26, 2024 •

edited

Loading

lukehoban left a comment

lukehoban May 7, 2024

blampe Aug 9, 2024

blampe commented Aug 9, 2024

flostadler left a comment

flostadler Aug 19, 2024

blampe Aug 21, 2024

flostadler Jan 2, 2025

flostadler Aug 19, 2024

blampe Aug 21, 2024

t0yv0 commented Nov 6, 2024

t0yv0 commented Nov 6, 2024

flostadler commented Dec 23, 2024

flostadler commented Jan 2, 2025

flostadler commented Jan 2, 2025

blampe Jan 4, 2025 •

edited

Loading

flostadler Jan 6, 2025

blampe commented Jan 4, 2025

pulumi-bot commented Jan 14, 2025

Use new docker build provider #1278

Use new docker build provider #1278

Conversation

blampe commented Apr 26, 2024 • edited Loading

lukehoban left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blampe commented Aug 9, 2024

flostadler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t0yv0 commented Nov 6, 2024

t0yv0 commented Nov 6, 2024

flostadler commented Dec 23, 2024

flostadler commented Jan 2, 2025

flostadler commented Jan 2, 2025

blampe Jan 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blampe commented Jan 4, 2025

pulumi-bot commented Jan 14, 2025

blampe commented Apr 26, 2024 •

edited

Loading

blampe Jan 4, 2025 •

edited

Loading