From c1dcb7820f164a8840ad1df5ff95b11a4232d0e0 Mon Sep 17 00:00:00 2001 From: Khai Do Date: Fri, 17 Jan 2025 08:59:06 -0800 Subject: [PATCH 1/6] [IT-3831] Update AMI and docs Update agora-data-manager-github-runner AMI and general project documentation. depends on https://github.com/Sage-Bionetworks/aws-infra/pull/421 --- README.md | 59 +++++++++++++------ .../agora-data-manager-github-runner.yaml | 4 +- 2 files changed, 44 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index c22953b..229f8c0 100644 --- a/README.md +++ b/README.md @@ -7,21 +7,46 @@ Request using PRs provide history, gating, reviewing and an approval process. ### Add Images -Step by step instructions to add a new image to the image builder: - -1. Add image builder cloudformation templates should be added to -[Sage-Bionetworks/aws-infra](https://github.com/Sage-Bionetworks/aws-infra) -repo in the templates/ImageBuilder directory. -2. Add a new Sceptre config in `config/prod` directory referencing -a versioned (or tagged) instance of the cloudformation template. -3. Create a PR to merge the new config into the `main` branch. -4. Once the PR is merged, the image builder will be triggered to -to deploy to the AWS org-sagebase-imagecentral account which will then -trigger a build of the AMI. -5. The AMI will be shared to all accounts in our AWS organization which -means that it is searchable using the AWS console (EC2-Instances-AMIs or -Image Builder->Images) or the AWS CLI from any AWS account in our organization. - +Step by step instructions to create a new image: + +The first step is to create the definition template: +1. Create an Image Builder definition (cloudformation) template and add it to +the [Sage-Bionetworks/aws-infra](https://github.com/Sage-Bionetworks/aws-infra) +repo in the `templates/ImageBuilder` directory. +2. Create a PR with this change to aws-infra repo +3. Review, approve and Merge the PR +4. Tag the repo with a [version number](https://github.com/Sage-Bionetworks/aws-infra/tags) + +Next step is to use the definition template to provision an AMI: +1. In this repo, add a new Sceptre config in `config/prod` directory referencing +a versioned (or tagged) instance of the cloudformation template. Set the +the `ImageVersion` to the same number as the aws-infra repo tag (i.e. 0.9.2 without the `v`). +It must match a [semantic version](https://semver.org/) number. +2. Create a PR to merge the new config into the `main` branch. +3. Once the PR is merged, the cloudformation template will be deploy to the +AWS org-sagebase-imagecentral account which will in turn trigger a build +of the image which will generate an AMI. +4. The AMI will be shared to all accounts in our AWS organization which +means that it is searchable from any of our AWS accounts using either the +AWS console EC2 Instances->AMIs or Image Builder->Images pages. The AMI is +also searchable using the AWS CLI +[describe-images command](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-images.html). + + +### Updating Images +To update an image the cloudformation template we must update two repos + +Update the cloudformation template in aws-infra: +1. Create a PR to update the cloudformation template in the aws-infra repo +2. Review, approve and Merge the PR +3. Tag the repo with a version number + +Update the Sceptre config in this repo: +1. Create a PR to update the url reference to the cloudformation template in + the Sceptre config in the `config/prod` directory +2. An update to the image definition (or cloudformation template) may require + an `ImageVersion` update as well otherwise cloudformation will fail with a + message like `The following resource XXXXX already exists..`. ### Removing Images Important info when removing image builder resources and the generated @@ -43,8 +68,8 @@ __NOTE__: Steps above can also be done using the `sceptre delete` command or the To test an image we recommend that you manually provision an EC2 instance from the AMI image then connect to it using the [AWS session manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) or [SSH](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-linux-inst-ssh.html). -Validate that the instance contains everything that was specified from the image builder and -that it's generally running as expected. +Validate that the image meets the specifications defined in the cloudformation template and +that image is generally running as expected. ## Contributions diff --git a/config/prod/agora-data-manager-github-runner.yaml b/config/prod/agora-data-manager-github-runner.yaml index 4c166e3..4bdae37 100644 --- a/config/prod/agora-data-manager-github-runner.yaml +++ b/config/prod/agora-data-manager-github-runner.yaml @@ -1,9 +1,9 @@ template: type: "http" - url: "https://raw.githubusercontent.com/Sage-Bionetworks/aws-infra/v0.9.3/templates/ImageBuilder/amazon-linux-2023-agora-bastian.yaml" + url: "https://raw.githubusercontent.com/Sage-Bionetworks/aws-infra/v0.9.4/templates/ImageBuilder/amazon-linux-2023-agora-bastian.yaml" stack_name: "agora-data-manager-github-runner" stack_tags: OwnerEmail: "it@sagebase.org" CostCenter: "Platform Infrastructure / 990300" parameters: - ImageVersion: "0.0.1" + ImageVersion: "0.9.4" From 6ad758aa8f03b85613039bfb8694ea479e73ab83 Mon Sep 17 00:00:00 2001 From: Khai Do Date: Fri, 17 Jan 2025 09:24:04 -0800 Subject: [PATCH 2/6] refine docs --- README.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 229f8c0..783617f 100644 --- a/README.md +++ b/README.md @@ -34,19 +34,25 @@ also searchable using the AWS CLI ### Updating Images -To update an image the cloudformation template we must update two repos +To update an image the we must first update aws-infra then this repo. Update the cloudformation template in aws-infra: -1. Create a PR to update the cloudformation template in the aws-infra repo +1. Create a PR to update the ImageBuilder cloudformation template in the aws-infra repo 2. Review, approve and Merge the PR 3. Tag the repo with a version number Update the Sceptre config in this repo: -1. Create a PR to update the url reference to the cloudformation template in - the Sceptre config in the `config/prod` directory -2. An update to the image definition (or cloudformation template) may require - an `ImageVersion` update as well otherwise cloudformation will fail with a - message like `The following resource XXXXX already exists..`. +1. Create a PR to update the template `url` reference and `ImageVersion` number +in the Sceptre config file. +2. Review, approve and Merge the PR +3. Once merged the cloudformation template will be deploy which will trigger +an update to the AMI. AMIs are immutable therefore AWS will create a new AMI +on every change. The updated AMIs will retain the same name, only the version +number is updated. + +__Note__: An update to the image definition (or cloudformation template) requires +an `ImageVersion` update otherwise cloudformation may fail with a message similar to +`The following resource XXXXX already exists..`. ### Removing Images Important info when removing image builder resources and the generated @@ -62,7 +68,7 @@ situations where it is appropriate to delete AMIs, for example when testing AMIs To delete AMIs go into the AWS console EC2 -> Images -> AMIs, search for the AMIs then delete (or de-register) them. -__NOTE__: Steps above can also be done using the `sceptre delete` command or the AWS CLI. +__NOTE__: Step #2 above can also be done using the `sceptre delete` command or the AWS CLI. ### Testing Images To test an image we recommend that you manually provision an EC2 instance from the From 9fc373778ee0485bcea87c5e51be2cb9c1e8f1c7 Mon Sep 17 00:00:00 2001 From: Khai Do Date: Tue, 21 Jan 2025 14:19:11 -0800 Subject: [PATCH 3/6] update cost center --- config/prod/agora-data-manager-github-runner.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/config/prod/agora-data-manager-github-runner.yaml b/config/prod/agora-data-manager-github-runner.yaml index 4bdae37..366f25f 100644 --- a/config/prod/agora-data-manager-github-runner.yaml +++ b/config/prod/agora-data-manager-github-runner.yaml @@ -3,7 +3,7 @@ template: url: "https://raw.githubusercontent.com/Sage-Bionetworks/aws-infra/v0.9.4/templates/ImageBuilder/amazon-linux-2023-agora-bastian.yaml" stack_name: "agora-data-manager-github-runner" stack_tags: - OwnerEmail: "it@sagebase.org" - CostCenter: "Platform Infrastructure / 990300" + OwnerEmail: "jessica.britton@sagebase.org" + CostCenter: "Agora / 112300" parameters: ImageVersion: "0.9.4" From d3e021563707620484ebc3671560e3f766ee6e3f Mon Sep 17 00:00:00 2001 From: Khai Do Date: Tue, 21 Jan 2025 14:20:43 -0800 Subject: [PATCH 4/6] update doc --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 783617f..70c68ce 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ a versioned (or tagged) instance of the cloudformation template. Set the the `ImageVersion` to the same number as the aws-infra repo tag (i.e. 0.9.2 without the `v`). It must match a [semantic version](https://semver.org/) number. 2. Create a PR to merge the new config into the `main` branch. -3. Once the PR is merged, the cloudformation template will be deploy to the +3. Once the PR is merged, the cloudformation template will be deployed to the AWS org-sagebase-imagecentral account which will in turn trigger a build of the image which will generate an AMI. 4. The AMI will be shared to all accounts in our AWS organization which From 374e4026551affa1bf5d5ce9b654f15f583540b1 Mon Sep 17 00:00:00 2001 From: Khai Do Date: Tue, 21 Jan 2025 14:22:14 -0800 Subject: [PATCH 5/6] fix --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 70c68ce..51819b3 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ Update the Sceptre config in this repo: 1. Create a PR to update the template `url` reference and `ImageVersion` number in the Sceptre config file. 2. Review, approve and Merge the PR -3. Once merged the cloudformation template will be deploy which will trigger +3. Once merged the cloudformation template will be deployed which will trigger an update to the AMI. AMIs are immutable therefore AWS will create a new AMI on every change. The updated AMIs will retain the same name, only the version number is updated. From 2fde7f82a90e11c0e951e852f0b74422fdba20c4 Mon Sep 17 00:00:00 2001 From: Khai Do Date: Wed, 22 Jan 2025 07:14:06 -0800 Subject: [PATCH 6/6] set VolumeSize --- config/prod/agora-data-manager-github-runner.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/config/prod/agora-data-manager-github-runner.yaml b/config/prod/agora-data-manager-github-runner.yaml index 366f25f..c6ef9a4 100644 --- a/config/prod/agora-data-manager-github-runner.yaml +++ b/config/prod/agora-data-manager-github-runner.yaml @@ -7,3 +7,4 @@ stack_tags: CostCenter: "Agora / 112300" parameters: ImageVersion: "0.9.4" + VolumeSize: "50"