ignoreChanges seems to have some trouble ignoring changes

[gj]

Code

export const rootService = new awsx.ecs.FargateService(
  rootServiceName,
  rootServiceArgs,
  {
    customTimeouts: {
      create: '30m',
      update: '30m',
      delete: '30m',
    },
    ignoreChanges: ['loadBalancers', 'taskDefinition'],
  }
);

NOTE: I've also tried setting direct paths to the properties that are changing (e.g., 'loadBalancers[0].targetGroupArn', but Pulumi still doesn't ignore the property diff.

pulumi up output

✦ ❯ pulumi up
Previewing update (org/development):
     Type                          Name                              Plan        Info
     pulumi:pulumi:Stack           infra-development
     ├─ awsx:x:ecs:FargateService  development-root-fargate-service
 +-  │  └─ aws:ecs:Service         development-root-fargate-service  replace     [diff: ~loadBalancers,taskDefinition]

pulumi stack export output

            {
                "urn": "urn:pulumi:development::infra::awsx:x:ecs:FargateService::development-root-fargate-service",
                "custom": false,
                "type": "awsx:x:ecs:FargateService",
                "parent": "urn:pulumi:development::infra::pulumi:pulumi:Stack::infra-development",
                "customTimeouts": {
                    "create": 1800,
                    "update": 1800,
                    "delete": 1800
                }
            }

Use case

I create the Fargate service and initial Task Definition via Pulumi, but then I use AWS CodeDeploy for blue/green deployments of the service. CodeDeploy flips the 'blue' and 'green' target groups on every deployment and additionally creates a new revision of the Task Definition. Pulumi seems to store the Task Definition ARN including the revision (e.g., <ARN>:23) and then complains that it is different than the current Task Definition attached to the Fargate service (e.g., <same ARN>:24).

[pgavlin]

Similar to https://github.com/pulumi/pulumi/issues/3529, I'm guessing that the ignoreChanges property on the awsx.ecs.FargateService component resource is not being propagated to its children. Until that is fixed, I believe that you can use the transformations resource option to add the appropriate ignoreChanges options to the underlying ECS service: https://www.pulumi.com/docs/intro/concepts/programming-model/#transformations

[gj]

@pgavlin thanks for the tip! transformations is a good workaround for the time being.

[brandonkal]

I've noticed pulumi refresh completely ignores the ignoreChanges option. For example, when ignoreChanges on a kubernetes resource is set to ["status"], a pulumi refresh still shows an update operation where the diff adds "status : {}"

[lukehoban]

pulumi refresh completely ignores the ignoreChanges option

I believe that is currently by-design. The "changes" in refresh are not changes that are being made to the cloud - they are changes that are being made to the Pulumi state to align with the cloud. So its not immediately clear to me whether it's more "right" to ignore these changes or not during refresh. Though if ignoreChanges is there, it should ensure that after the refresh, the next update does not propose making changes - which is really the goal. It seems at first blush that that's a reasonable net experience - you still get your state brought up to date, but you avoid making any change to your cloud provider related to those properties.

Curious what workflow you have is where this doesn't meet your needs?

[benswinburne]

@gj are you able to share the code you used to get this to work please? I implemented the following but I can't seem to get more granularity than containerDefinitions. I only really want to ignore taskDefinitionArgs.containers.myservice.image. I'd possibly want to sort the task definition arn as you described too but I haven't worked that far ahead yet.

const service = new awsx.ecs.FargateService(
  'fgservice',
  {
    taskDefinitionArgs: {
      containers: {
        myservice: {
          image: image,
          memory: 256,
          portMappings: [loadbalancer],
        },
      },
    },
    desiredCount: 1,
  },
  {
    // ignoreChanges: ['taskDefinitionArgs'],
    transformations: [
      (args) => {
        if (args.type === 'aws:ecs/taskDefinition:TaskDefinition') {
          return {
            props: args.props,
            opts: pulumi.mergeOptions(args.opts, {
              ignoreChanges: ['containerDefinitions'],
            }),
          };
        }

        return undefined;
      },
    ],
  }
);

[gj]

@benswinburne try targeting the Service instead of the TaskDefinition. Haven't looked at this code in months, but this is what I ended up with:

    transformations: [
      args => {
        if (args.type === 'aws:ecs/service:Service') {
          return {
            props: args.props,
            opts: pulumi.mergeOptions(args.opts, {
              ignoreChanges: [
                'desiredCount',
                'loadBalancers',
                'taskDefinition',
              ],
            }),
          };
        }
        return undefined;
      },
    ],

[benswinburne]

@gj thanks very much for this.

How have you solved being able to make changes to your task definitions etc with this setup?

If we don't ignore any changes/implement any transformations, Pulumi allows me to create a task definition with say nginx as the image, use say codepipeline/codedeploy to put a new image into the service on a new task definition revision with no changes noticed by Pulumi. That said, as soon as I want to make any changes to the task definition I can't because if I change the memory for example then the task definition has changed and it reverts the image back to nginx.

In the above example combined with the transformation, I can't update the memory because the whole definition is ignored.

I guess just finding out the image name currently applied to the service and putting it into the definition at the same time as changing the memory would resolve this but it seems a bit convoluted, prone to human error and also not even guaranteed to work (if the service changes between getting the image name and applying it to the Pulumi program you'd end up rolling back a version).

[gj]

@benswinburne I "solved" it by simply not making those changes through Pulumi 🙂

I hit the same frustrating wall that you've run up against now, and I'm not sure if much has changed since I was playing around with this stuff last autumn. I ended up managing changes to task definitions through the AWS CLI instead of through Pulumi.

[benswinburne]

Ah, that's frustrating. I think I'm effectively aiming for the same solution as you and coming up with lots of the same issues!

@stack72 are you able to shed any light on this?

[Adam-Burke]

Is there a way to not update the task definition if it hasn't changed.

[bryceAebi]

+1 on this general issue.

The transformation is no longer needed, ignore_changes parameter is now working. But yeah, if I ignore the taskDefinition to get around the task revision issue, I run into the same problems as @benswinburne

Additionally, if I just set the taskDefinition to task-definition-family-name instead of task-definition-family-name:revision (the former in AWS is supposed to default to the latest task definition revision) I still get diff issues, even if my tasks in fargate are running the latest revision.

Diff looks like this:

~ taskDefinition: "fargate-web-task-definition:95" => "fargate-web-task-definition"

where the latest revision is indeed 95

Thankfully, executing the Pulumi update does not actually "do" anything. So it seems like this is a partial solution to the revision issue, in that "things work". But it's annoying that the diff appears every single time. It's likely to confuse anyone who has not dug into this issue.

[d4nyll]

Is there any updates on this? Specifically, whether ignoreChanges of awsx resources can be propagated to the corresponding underlying aws resources?

[simwak]

I also have trouble using it with a Helm Chart which changes it's certs each deployment. Or is it a completly different problem?

    └─ kubernetes:helm.sh/v3:Chart   cilium                                       
 +-     ├─ kubernetes:core/v1:Secret  kube-system/hubble-ca-secret     replace     [diff: ~data]
 +-     └─ kubernetes:core/v1:Secret  kube-system/hubble-server-certs  replace     [diff: ~data]

[sivanovnr]

I am assuming this is still an issue, no?

We create a Fargate services with b/g deployment via codedeploy.
After a deployment, running pulumi up does not ignore loadBalancers, but it tries to switch back to TargetGroupA.

The code:

    return new FargateService("test-service-pulumi",
        fargateServiceArgs, {ignoreChanges: ["loadBalancers"]}
    )

I've tried to use transformations but again no luck. This is the code and it still does not ignore the Load balancer changes. I've used the same transformation for the ListenerRule and it successfully ignores it, so code should be working. Any ideas or advices are welcome!

    const fargateServiceArgs: FargateServiceArgs = {
        continueBeforeSteadyState: true,
        cluster: conf.require("ecs_cluster_arn"),
        desiredCount: 1,
        networkConfiguration: {
            assignPublicIp: publicIpFlag,
            securityGroups: [securityGroup.id],
            subnets: loadBalancer.subnets
        },
        deploymentController: {
            type: "CODE_DEPLOY"
        },
        loadBalancers: [serviceLoadBalancer],
        taskDefinition: taskDefinition.arn,
    }
    

    return new FargateService("test-service-pulumi",
        fargateServiceArgs, {
            transformations: [
                args => {
                  if (args.type === 'aws:ecs/service:Service') {
                    return {
                      props: args.props,
                      opts: pulumi.mergeOptions(args.opts, {
                        ignoreChanges: [
                          "loadBalancers",
                        ],
                      }),
                    };
                  }
                  return undefined;
                },
              ],
        }
    )
}

[bob-bins]

transformations doesn't seem to target child resources anymore (I'm on version 2.4.0 of the package). For example, this transformation on a new Vpc

      transformations: [
        (args) => {
          console.log(args.type)
          return undefined
        }
      ]

only prints out awsx:ec2:Vpc.

I don't think there's a working workaround anymore :/

[dan-cooke]

I would just like to +1 this, as others have pointed out the main issue here is that the Crosswalk package for ecs does not propagate ignoreChanges to its child resources

A workaround in the meantime is to not use the crosswalk service and provision aws.ecs.Service directly.

Or if you want to use cross walk, you can just hardcode your taskDefinition image name

const service = new awsx.ecs.FargateService(
  'api',
  {
    cluster: baseInfra.requireOutput('clusterArn'),
    enableExecuteCommand: true,

    deploymentController: {
      type: 'CODE_DEPLOY',
    },
    networkConfiguration: {
      securityGroups: [apiSecurityGroup.id],
      assignPublicIp: true,
      subnets: baseInfra.requireOutput('vpcPublicSubnetIds'),
    },
    continueBeforeSteadyState: true,

    // TODO: this is hardcoded as nginx due to a bug with pulumi not being able
    // to ignore changes on the task definition
    // As our deployments are handled by code deploy, we should ignore changes on the task definition
    taskDefinition: 'nginx:latest',
    loadBalancers: [
      {
        containerPort: 80,
        containerName: 'api',
        targetGroupArn: blueTargetGroup.arn,
      },
    ],
  },
  {
    ignoreChanges: ['taskDefinition'],
  }
);

[t0yv0]

There have been some updates to Pulumi platform that might be helpful here. There is now a transforms resource option https://www.pulumi.com/docs/concepts/options/transforms/ that is able to work with awsx.* resources.

I am confirming with the following TypeScript program - I've selected a fast to provision awsx resource for illustration purposes. The program is to be activated with:

pulumi config set step 0
pulumi up --yes
pulumi config set step 1
pulumi up --yes

import * as awsx from "@pulumi/awsx";
import * as pulumi from "@pulumi/pulumi";

const cfg = new pulumi.Config();

const step = cfg.requireNumber("step");

const myVpc = new awsx.ec2.Vpc("awsx-nodejs-subnets-with-tags", {
    tags: {
        isoverridden: "false",
    },
    subnetSpecs: [
        {
            type: awsx.ec2.SubnetType.Public,
            cidrMask: 22,
            tags: {
                isoverridden: "true",
                custom_tag_subnet_type: "subnet_public",
                custom_tag_one: "1"
            }
        },
        {
            type: awsx.ec2.SubnetType.Private,
            cidrMask: 21,
            tags: {
                custom_tag_subnet_type: "subnet_private",
                custom_tag_two: step === 0 ? "zero" : "one",
                custom_tag_three: "3"
            }
        },
    ],
}, { 
    // Using ignoreChanges does not work.
    // ignoreChanges: ["tags"]

    // However, using the transforms option works to suppress the update.
    // More information at https://www.pulumi.com/docs/concepts/options/transforms/
    transforms: [args => {
        return {
            props: args.props,
            opts: pulumi.mergeOptions(args.opts, { ignoreChanges: ["tags"] })
        }
    }],
});

export const vpcId = myVpc.vpcId;
export const publicSubnetIds = myVpc.publicSubnetIds;
export const privateSubnetIds = myVpc.privateSubnetIds;

Note that ignoreChanges option is a no-op for the awsx.ec2.Vpc resource but using transforms to add an ignoreChanges option for every child resource achieves the desired effect of ignoring the change.

Please let us know if this unblocks any of the scenarios here 🙏

[t0yv0]

The provider should probably emit a user-friendly warning to users who attempt to set ignoreChanges when this setting is actually a no-op. Unfortunately this is not yet technically possible until the feature in pulumi/pulumi#12154 is completed.

[flostadler]

To simplify the Code Deploy usecase we could expose an input property that automatically sets ignore changes on the correct underlying resources. Something like a enableCodeDeploy flag