rewrite nixpkgs's create-amis script using Terraform

[AmineChikhaoui]

This rewrites create-amis.sh using terraform.
The create-amis.sh was pretty slow and switching to Terraform makes things more declarative and easier to read.
There are few things I don't like in the Terraform code, since we're copying the AMI to a set of regions (which require a different aws provider per region), the module system of terraform doesn't really work for this. The easy solution was to just use jq and make to generate the copy ami resources in JSON spec instead of HCL.
Even using Terraform, this still takes 15minutes which isn't that fast either but at least ami copying is done in parallel.

A note about usage of Terraform workspaces, since we don't want to trigger terraform replacements i.e delete and recreate of resources as we want to keep AMIs being used (and perhaps use a lifecycle to purge them periodically), I went with a workspace (statefile) per {release}.{arch}.{buildId}.

Things I'm planning to explore now that the building blocks are here:

• Add an example terraform+flake ec2 instance config using the generated AMIs as a quick way to test.
• Use Github Actions for running the Terraform applies. Explore OIDC integration with AWS: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services.
• Setup some lifecycle policies to purge older AMIs, which will allow scheduling this in cron without worrying about accumulated costs from snapshots, objects, etc ..

[AmineChikhaoui]

@zimbatm @grahamc if you can do a quick review or if there are any suggestions.

AWS OIDC setup is pointing to my fork/branch but can switch it before merging to nixos/nixos-org-configurations. I can also terraform the IAM role config for visibility.

Here is an example of the workflow runs: https://github.com/AmineChikhaoui/nixos-org-configurations/actions/runs/5172652543

[cole-h]

Something that I just discovered is the aws_dlm_lifecycle_policy resource, which has an example for cross-region copy: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dlm_lifecycle_policy#example-cross-region-snapshot-copy-usage. Supposedly, setting policy_type to IMAGE_MANAGEMENT and resource_types to INSTANCE might work? Might be worth looking into, if it means we might be able to get rid of the jq and scripted processing.

Though I don't know if the cross-region copying means you'd need a provider for each region (as is currently necessary). I also don't know if the fact that it's for managing a lifecycle makes it unsuitable for this, though (i.e. if it requires you to specify something that would "clean up" images / etc and not just use it for cross-region copying).

EDIT: But the fact the documentation says "for the copied snapshot", "cross-Region snapshot copies", etc makes me think that this might not work exactly as I originally thought 🥲

EDIT2: There is also this aws_ami_copy resource that might be useful. I think it still requires a separate provider for each region we want to copy to, but...

[AmineChikhaoui]

EDIT2: There is also this aws_ami_copy resource that might be useful. I think it still requires a separate provider for each region we want to copy to, but...

That's the one I'm using in regions.jq right ?

[cole-h]

......yes. 🤦

If only there was some easier way to do this :')

[AmineChikhaoui]

@cole-h probably CDK but not sure if it's worth the effort compared to a few lines of jq file 😄

I'll look into DLM, I was planning to explore it for managing the deprecation and removal of AMIs. Didn't see copy AMI handling specifically from a quick scroll over docs but can look a bit more.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/ami-for-nixos-23-11/36860/4

[delroth]

Trying to understand the status of this PR - what is the relation between this work and the (currently operational, I think) https://github.com/NixOS/amis maintained by @arianvp? Are these complementary / parallel implementations / does one deprecate the other / ...?

(Bear with me, I have very little context, but I'd like to clean up this repo and make sure we have a clear set of pending items to look at :p)

[AmineChikhaoui]

@delroth I didn't read through https://github.com/NixOS/amis yet but my understanding it will deprecate this process and this PR won't be needed anymore in that case.