Add implementation for CloudFormation Custom Resource Emulator #1806

flostadler · 2024-11-07T16:50:38Z

This PR adds support for CloudFormation Custom Resource to the aws-native provider. It implements an emulator that enables Pulumi programs to interact with Lambda-backed CloudFormation Custom Resources.

A CloudFormation custom resource is essentially an extension point to run arbitrary code as part of the CloudFormation lifecycle. It is similar in concept to the Pulumi Command Provider, the difference being that CloudFormation CustomResources are executed in the Cloud; either through Lambda or SNS.

For the first implementation we decided to limit the scope to Lambda backed Custom Resources, because the SNS variants are not widely used.

Custom Resource Protocol

The implementation follows the CloudFormation Custom Resource protocol. I derived the necessary parts by combining information from the docs, CDKs CustomResource Framework and trial&error.

Notable aspects of that protocol are:

primitive properties need to be string encoded when sending them to Custom Resource handlers. This includes deeply nested properties: AWS::CloudFormation::CustomResource - Lambda Backed CustomResource's ResourceProperties converts int and boolean properties to strings aws-cloudformation/cloudformation-coverage-roadmap#1037
The Lambda Function is invoked asynchronously. Lambda will retry the execution if the function fails unexpectedly (e.g. unhandled exception).
Due to the async invocation, the response is not returned from the Lambda Function, instead it's sent to a ResponseURL that needs to be included in the request payload.
- Similarly to CloudFormation, we decided to implement this using S3 Buckets and presigned URLs.

Custom Resource Lifecycle

sequenceDiagram
    participant A as aws-native
    participant S3 as S3 Bucket
    participant L as Lambda
    
    %% Create Flow
    Note over A,L: Create Operation
    A->>S3: Generate presigned URL
    A->>L: Invoke with CREATE event
    activate L
    loop Until response found or timeout
        A->>S3: Poll for response
        L-->>S3: Upload response
    end
    deactivate L
    A->>S3: Fetch response
    alt Success
        A->>A: Store PhysicalId & outputs
    else Failure
        A->>A: Return error
    end

    %% Update Flow
    Note over A,L: Update Operation
    A->>S3: Generate presigned URL
    A->>L: Invoke with UPDATE event
    activate L
    loop Until response found or timeout
        A->>S3: Poll for response
        L-->>S3: Upload response
    end
    deactivate L
    A->>S3: Fetch response
    alt Success
        A->>A: Check PhysicalId
        alt ID Changed
            A->>S3: Generate presigned URL for cleanup
            A->>L: Invoke with DELETE event for old resource
            activate L
            loop Until cleanup response found or timeout
                A->>S3: Poll for cleanup response
                L-->>S3: Upload cleanup response
            end
            deactivate L
            A->>S3: Fetch cleanup response
        end
    else Failure
        A->>A: Return error
    end

    %% Delete Flow
    Note over A,L: Delete Operation
    A->>S3: Generate presigned URL
    A->>L: Invoke with DELETE event
    activate L
    loop Until response found or timeout
        A->>S3: Poll for response
        L-->>S3: Upload response
    end
    deactivate L
    A->>S3: Fetch response
    alt Success
        A->>A: Return success
    else Failure
        A->>A: Return error
    end

Reviewer Notes

Key areas to review:

Error handling in the response collection mechanism
Timeout management, especially for the Update lifecycle
Documentation completeness and accuracy

Exposing this resource and schematizing it is part of this PR #1807.
Automatically cleaning up the response objects is not included in this PR in order to keep its size manageable. Implementing this is tracked here: #1813.

Please pay special attention to:

S3 response collection mechanism security
State management during updates
Cleanup handling when physical resource IDs change

Testing

Unit tests including error handling tests for various failure scenarios
Integration tests with actual Lambda functions are added in this stacked PR: Expose CloudFormation Custom Resource Emulator Resource #1807

Related Issues

flostadler · 2024-11-07T16:50:56Z

This change is part of the following stack:

Add implementation for CloudFormation Custom Resource Emulator #1806 ◀
- Expose CloudFormation Custom Resource Emulator Resource #1807

_{Change managed by git-spice.}

github-actions · 2024-11-07T16:52:31Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

codecov · 2024-11-07T18:06:08Z

Codecov Report

Attention: Patch coverage is 59.43152% with 157 lines in your changes missing coverage. Please review.

Project coverage is 48.13%. Comparing base (04539b4) to head (b184dc0).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
provider/pkg/resources/cfn_custom_resource.go	59.43%	142 Missing and 15 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1806      +/-   ##
==========================================
+ Coverage   46.81%   48.13%   +1.32%     
==========================================
  Files          42       43       +1     
  Lines        6167     6554     +387     
==========================================
+ Hits         2887     3155     +268     
- Misses       3052     3156     +104     
- Partials      228      243      +15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

corymhall

Just a couple of minor comments, otherwise this is awesome! Great work!

provider/pkg/resources/cfn_custom_resource.go

corymhall · 2024-11-08T14:10:52Z

provider/pkg/resources/cfn_custom_resource.go

+//    - Initiates cleanup of old resource
+//    - Sends DELETE event for old PhysicalResourceId
+// 5. Returns updated properties and new PhysicalResourceId
+func (c *cfnCustomResource) Update(ctx context.Context, urn urn.URN, id string, inputs, oldInputs, state resource.PropertyMap, timeout time.Duration) (resource.PropertyMap, error) {


Is there a way for us to get the resource options here, specifically the retainOnDelete option? I know there were times that I would set the retainOnDelete to true for custom resources because they didn't handle delete well.

Sadly, not that I'm aware of: https://pulumi-developer-docs.readthedocs.io/latest/docs/_generated/proto/provider.html#pulumirpc-updaterequest

Only MLCs have access to retainOnDelete it seems:
https://github.com/pulumi/pulumi/blob/master/proto/pulumi/provider.proto#L473

For CustomResources that are traditional I don't think the engine ever cooperates with the provider on this. The engine handles it all engine-side.

provider/pkg/resources/cfn_custom_resource.go

t0yv0 · 2024-11-08T14:59:04Z

provider/pkg/resources/cfn_custom_resource.go

+			},
+			"serviceToken": {
+				Description: "The service token to use for the Custom Resource. The service token is invoked when the resource is created, updated, or deleted.\n" +
+					"This can be a Lambda Function ARN with optional version or alias identifiers.\n\n" +


This sentence here is a bit confusing. "This can be a Lambda Function ARN", perhaps give a quick example?

Also should this be marked Secret/Sensitive in the scehema?

Lambda function ARNs (including alias or version) are a very common concept in AWS. I don't think we should go into too much detail about it here.

This is also not a secret, ARNs are just unique IDs

I was confused that this is called "serviceToken". This made me think of access tokens or some such. If there's precedent of calling lambda ARNs "tokens" then TIL.

I also don't like the name... It's what CloudFormation called this property. I think there's precedence to stick to their naming patterns given we're trying to emulate their behavior/API

t0yv0 · 2024-11-08T15:05:35Z

provider/pkg/resources/cfn_custom_resource.go

+// Read returns the current inputs and outputs of the custom resource because CFN custom resources do not store state.
+// They are just a stateless wrapper around a Lambda function or SNS topic.
+func (c *cfnCustomResource) Read(ctx context.Context, urn urn.URN, id string, oldInputs resource.PropertyMap, oldState resource.PropertyMap) (resource.PropertyMap, resource.PropertyMap, bool, error) {
+	if len(oldState) == 0 {


Comment here that len(oldState) is trying to guess if we are in a Read-for-import.
The developer docs are improving but not quite there yet but are being improved (cc @lunaris )

https://pulumi-developer-docs.readthedocs.io/latest/developer-docs/providers/implementers-guide.html#read

t0yv0 · 2024-11-08T15:07:01Z

provider/pkg/resources/cfn_custom_resource.go

+
+type Clock interface {
+	Now() time.Time
+	Since(time.Time) time.Duration


Nit: Go likes minimal interfaces so you only really need Now(), as Since can be computed from Now in a func.

Yes, but that would clutter the implementation. The purpose of this interface is to make the code testable. I'd rather just pass through to the stdlib than add custom code

Mhm, I guess this is a bit of a moot point from my side given that you can subtract times and get the duration that way

Yeah it's really a small nit, feel free to disregard.

t0yv0 · 2024-11-08T15:13:52Z

provider/pkg/resources/cfn_custom_resource.go

+// It returns the inputs, validation failures, and an error if the inputs cannot be unmarshalled.
+func (c *cfnCustomResource) Check(ctx context.Context, urn urn.URN, randomSeed []byte, inputs resource.PropertyMap, state resource.PropertyMap, defaultTags map[string]string) (resource.PropertyMap, []ValidationFailure, error) {
+	var typedInputs CfnCustomResourceInputs
+	_, err := resourcex.Unmarshal(&typedInputs, inputs, resourcex.UnmarshalOptions{})


I do not fully trust this function, and reading the docs quickly is not very reassuring some of the concerns I have.

I think we might need to think about this a bit (probably out of scope of this PR but before being fully confident).

What happens with unknowns, dependencies and secrets?

For secrets it's a design question - custom resources do not support them right? So we need to support secrets through the black-box transformation I think?

We're already quite heavily using the resourcex.Decode and resourcex.Unmarshal methods in the provider. From what I can tell they work fine.

Additionally we're not using the unmarshalled typedInputs to construct the outputs, but rather use the inputs resource.PropertyMap to construct the outputs.
From that aspect, we're properly handling secrets.

Unknowns are a good point though. I'll try to double check whether this can receive unknowns if SupportsPreview: false.

t0yv0 · 2024-11-08T15:17:17Z

provider/pkg/resources/cfn_custom_resource.go

+//    - On success: Returns PhysicalResourceId and properties
+//    - On failure: Returns error with reason
+//    - Handles `NoEcho` for sensitive data
+func (c *cfnCustomResource) Create(ctx context.Context, urn urn.URN, inputs resource.PropertyMap, timeout time.Duration) (*string, resource.PropertyMap, error) {


I was wondering if this is ever called in Preview, but looks like the provider is SupportsPreview: false so we're good. It never is! No unknowns then.

t0yv0 · 2024-11-08T15:19:50Z

provider/pkg/resources/cfn_custom_resource.go

+
+// Check validates the inputs of the resource and applies default values if necessary.
+// It returns the inputs, validation failures, and an error if the inputs cannot be unmarshalled.
+func (c *cfnCustomResource) Check(ctx context.Context, urn urn.URN, randomSeed []byte, inputs resource.PropertyMap, state resource.PropertyMap, defaultTags map[string]string) (resource.PropertyMap, []ValidationFailure, error) {


I think this might still receive unknowns but I'm not 100% sure.

I'll double check. It's not a big deal if we stick to using the PropertyMap for Check

t0yv0 · 2024-11-08T15:22:35Z

provider/pkg/resources/cfn_custom_resource_test.go

+			},
+		},
+		{
+			name:   "SecretResponse",


Would it be possible to see here what is returned to the engine from Create in this test case?

Perhaps with hexops/autogold if it's too much trouble to write by hand.

We're asserting this with the assertion in line 348. In case noEcho is set to true, we expect the data prop to be marked as a secret.

t0yv0 · 2024-11-08T15:25:46Z

provider/pkg/resources/cfn_custom_resource_test.go

+
+	expectedError := fmt.Errorf("failed to invoke lambda")
+	stackID := "stack-id"
+	serviceToken := "arn:aws:lambda:us-west-2:123456789012:function:my-function"


Here's the serviceToken example, I see. Indeed it looks like a lambda ARN. Interesting.

t0yv0

🚢

Co-authored-by: Cory Hall <[email protected]>

This was referenced Nov 7, 2024

Expose CloudFormation Custom Resource Emulator Resource #1807

Merged

Add AWS SDK clients for CFN Custom Resource support #1805

Merged

flostadler changed the base branch from flostadler/cfn-custom-resource-aws-clients to flostadler/cfn-custom-resource-helpers November 7, 2024 17:32

flostadler force-pushed the flostadler/cfn-custom-resource-impl branch from baf7ba4 to 1d4b118 Compare November 7, 2024 17:32

flostadler mentioned this pull request Nov 7, 2024

Add utility functions for supporting CloudFormation CustomResources #1808

Merged

flostadler self-assigned this Nov 7, 2024

flostadler requested review from t0yv0, corymhall and a team November 7, 2024 18:06

flostadler force-pushed the flostadler/cfn-custom-resource-impl branch from 59357d1 to 94cfa06 Compare November 8, 2024 10:19

flostadler marked this pull request as ready for review November 8, 2024 11:54

flostadler force-pushed the flostadler/cfn-custom-resource-helpers branch from 60b5047 to 4cd4ddb Compare November 8, 2024 12:01

flostadler force-pushed the flostadler/cfn-custom-resource-impl branch from 94cfa06 to 19b72f2 Compare November 8, 2024 12:01

Base automatically changed from flostadler/cfn-custom-resource-helpers to master November 8, 2024 13:52

flostadler force-pushed the flostadler/cfn-custom-resource-impl branch from 19b72f2 to bc9e116 Compare November 8, 2024 13:53

corymhall approved these changes Nov 8, 2024

View reviewed changes

t0yv0 reviewed Nov 8, 2024

View reviewed changes

provider/pkg/resources/cfn_custom_resource.go Show resolved Hide resolved

t0yv0 reviewed Nov 8, 2024

View reviewed changes

t0yv0 approved these changes Nov 8, 2024

View reviewed changes

Add implementation for CloudFormation Custom Resource Emulator

6928e80

flostadler and others added 5 commits November 8, 2024 17:46

Fix assertion in read test

0c794bc

Add tests for partial failure

cae0c3d

Add developer docs

ee8b0b6

Update provider/pkg/resources/cfn_custom_resource.go

798af0b

Co-authored-by: Cory Hall <[email protected]>

Review Comments

914125c

flostadler force-pushed the flostadler/cfn-custom-resource-impl branch from cdfdb9b to 914125c Compare November 8, 2024 16:49

Add missing dependency

b184dc0

flostadler merged commit 14a21ae into master Nov 8, 2024
18 checks passed

flostadler deleted the flostadler/cfn-custom-resource-impl branch November 8, 2024 18:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add implementation for CloudFormation Custom Resource Emulator #1806

Add implementation for CloudFormation Custom Resource Emulator #1806

flostadler commented Nov 7, 2024 •

edited

Loading

flostadler commented Nov 7, 2024 •

edited

Loading

github-actions bot commented Nov 7, 2024

codecov bot commented Nov 7, 2024 •

edited

Loading

corymhall left a comment

corymhall Nov 8, 2024

flostadler Nov 8, 2024

t0yv0 Nov 8, 2024

t0yv0 Nov 8, 2024

flostadler Nov 8, 2024

t0yv0 Nov 8, 2024

flostadler Nov 8, 2024

t0yv0 Nov 8, 2024

t0yv0 Nov 8, 2024

flostadler Nov 8, 2024

flostadler Nov 8, 2024

t0yv0 Nov 8, 2024

t0yv0 Nov 8, 2024

flostadler Nov 8, 2024

t0yv0 Nov 8, 2024

t0yv0 Nov 8, 2024

flostadler Nov 8, 2024

t0yv0 Nov 8, 2024

flostadler Nov 8, 2024

t0yv0 Nov 8, 2024

t0yv0 Nov 8, 2024

t0yv0 left a comment

Add implementation for CloudFormation Custom Resource Emulator #1806

Add implementation for CloudFormation Custom Resource Emulator #1806

Conversation

flostadler commented Nov 7, 2024 • edited Loading

Custom Resource Protocol

Custom Resource Lifecycle

Reviewer Notes

Testing

Related Issues

flostadler commented Nov 7, 2024 • edited Loading

github-actions bot commented Nov 7, 2024

Does the PR have any schema changes?

codecov bot commented Nov 7, 2024 • edited Loading

Codecov Report

corymhall left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t0yv0 left a comment

Choose a reason for hiding this comment

flostadler commented Nov 7, 2024 •

edited

Loading

flostadler commented Nov 7, 2024 •

edited

Loading

codecov bot commented Nov 7, 2024 •

edited

Loading