Move ECS deployment workflows to this repo

[dhruvkb]

Currently both the template for the ECS deployment workflows and the code to compile it to 4 output YAML files are in the WordPress/openverse-infrastructure repository. This has the following main drawbacks:

• We need to define the workflows outside of where they’re actually being run. It's not intuitive and even fooled me once.
• It takes multiple contrived steps to get a change implemented including reviewing both the original change and then a sync PR and introduces many delays.
• We need to learn a new templating language that's a similar to Jinja, but different, and is only used in one piece of our code.
• This is problematic for contributors who can't see the private repo and so can't improve the workflows (see updated python scripts in ci workflows with python files #1152, for example).
• These templates are compiled with Terraform, which does a bunch of state checks and is very slow. Also Terraform's handling of local file providers is abysmal.

These problems go away if we move the template here and use this repos' automation tools to compile the template (like we do for pre-commit and Prettier configs).

[AetherUnbound]

The more I think about it, the more I think this move would definitely be worth it. We've had at least 2 if not 3 instances of someone editing the produced file and it not being caught in review (because GitHub doesn't automatically show the headers in diffs). I'm also frequently frustrated with how Terraform handles the generation of local files when working on multiple pieces of the infrastructure at once; there end up being no-op applies which muddy other plans because local files have changed across branches.

If we wanted to have this templated in the main repo, we'd need to define the following (per the definition):

• Service name
• Environment(s)
• AWS region (perhaps this could be defaulted?)
• ECS cluster name
• Container definition attributes:
  • Container name
  • Image name
  • Package name
• Deployment timeout in minutes

Many of the latter attributes we already have defined in the CI/CD workflow because they're published there.

I'm really interested to hear what you might think @sarayourfriend 🙂

[sarayourfriend]

I certainly understand why this would be more convenient, but having these templates as a direct result of the infrastructure settings is useful. Adding a new service? You don't have to remember to update the template. With the sync workflow Dhruv added, moving them happens for you now.

It takes multiple contrived steps to get a change implemented including reviewing both the original change and then a sync PR and introduces many delays.

This change would require re-writing the templates, moving those variables into hard-coded ones in a different repository, reminding people to sync them if they change. All for templates that very often do not change. We've had to change them for the monorepo recently, but that's an exception. And for the most complex changes listed above, it's even easier to mess up and pointedly annoying to have to copy the settings from one repository to another. It wouldn't get rid of needing to open two PRs for the most important changes not to make a blunder on: adding or changing service infrastructure configuration. You'd still need to make the changes in the infrastructure repository and then remember to update the template(s) and variables in this one. We still have two places to maintain the change. Using the sync workflow, at least you cannot forget to make that change or make a mistake in the copy/pasting that causes an issue.

These problems go away if we move the template here and use this repos' automation tools to compile the template (like we do for pre-commit and Prettier configs).

The templates for pre-commit and prettier configs are nice and solve a problem we have, but I will be eternally grateful for when they go away. Iterating on changes to any of those files is a pain and requires an unintuitive workflow that I don't think is measurably better than the Terraform templates except for maybe being slightly faster if you actually know how to iterate on them, which is obscure, despite my having documented the process.

We need to learn a new templating language that's a similar to Jinja, but different, and is only used in one piece of our code.

The template language is used in several places in our code. Grep for templatefile in the infrastructure repo. I have a similar feeling about MDX (I, in fact, unfortunately loath MDX), but it hasn't been an issue to use MDX.

I don't think it is worth re-writing them templates for what is not, to me, a net benefit but would incur a cost of development and testing and not actually remove the complexity, just slightly change it. This is just my personal opinion. If others think that this change is a significant improvement, then I won't stop it, despite my disagreement.

I would like to propose an alternative and strongly recommend it: dispatch the workflows in the infrastructure repository itself. This would hide the deployment logs from people who do not have access to that repository, but it would remove the need to sync them anywhere else. It would actually remove the complexity of iterating on the templates and needing to sync changes. Workflows in the monorepo that need to dispatch them can do so exactly the same way they're dispatching them now via gh just by adding the --repo flag:

$ gh workflow run ... --repo=WordPress/openverse-infrastructure

To me this is probably the most natural solution to the problem. It doesn't make it possible for contributors to make changes to the templates, but that isn't a big loss to me considering we have heaps of other places to contribute similar types but probably more meaningful and impactful changes.

[AetherUnbound]

Ooo, I like that proposal! It would also prevent people from accidentally editing those templates (like in #1152) since they simply won't be there in the repo in the first place 😮 You bring up a great point Sara, if we move it to the monorepo we're kind of moving the problem around, not fixing it at its core. I really like the idea of having them in one place and then dispatching them from the main repo.

[sarayourfriend]

Cool beans. Issue for this is here: https://github.com/WordPress/openverse-infrastructure/issues/462

[sarayourfriend]

Closing this discussion now with the solution agreed upon recorded in https://github.com/WordPress/openverse-infrastructure/issues/462