Merge pull request #48 from marshall7m/pr-display-tf-plans

Comment Terraform plans within PR page

README.md

@@ -167,12 +167,14 @@ Terraform module that provisions an AWS serverless CI/CD pipeline used for manag
 | <a name="input_ecs_task_logs_retention_in_days"></a> [ecs\_task\_logs\_retention\_in\_days](#input\_ecs\_task\_logs\_retention\_in\_days) | Number of days the ECS task logs will be retained | `number` | `14` | no |
 | <a name="input_ecs_tasks_common_env_vars"></a> [ecs\_tasks\_common\_env\_vars](#input\_ecs\_tasks\_common\_env\_vars) | Common env vars defined within all ECS tasks. Useful for setting Terragrunt specific env vars required to run Terragrunt commands. | <pre>list(object({<br>    name  = string<br>    value = string<br>  }))</pre> | `[]` | no |
 | <a name="input_enable_branch_protection"></a> [enable\_branch\_protection](#input\_enable\_branch\_protection) | Determines if the branch protection rule is created. If the repository is private (most likely), the GitHub account associated with<br>the GitHub provider must be registered as a GitHub Pro, GitHub Team, GitHub Enterprise Cloud, or GitHub Enterprise Server account. See here for details: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/about-protected-branches | `bool` | `true` | no |
+| <a name="input_enable_gh_comment_approval"></a> [enable\_gh\_comment\_approval](#input\_enable\_gh\_comment\_approval) | Determines if execution approval votes can be sent via GitHub comments.<br>This will also enable Terraform plans to be commented within merged PR page | `bool` | `false` | no |
+| <a name="input_enable_gh_comment_pr_plan"></a> [enable\_gh\_comment\_pr\_plan](#input\_enable\_gh\_comment\_pr\_plan) | Determines if Terraform plans will be commented within open PR page | `bool` | `false` | no |
 | <a name="input_enforce_admin_branch_protection"></a> [enforce\_admin\_branch\_protection](#input\_enforce\_admin\_branch\_protection) | Determines if the branch protection rule is enforced for the GitHub repository's admins. <br>  This essentially gives admins permission to force push to the trunk branch and can allow their infrastructure-related commits to bypass the CI pipeline. | `bool` | `false` | no |
 | <a name="input_file_path_pattern"></a> [file\_path\_pattern](#input\_file\_path\_pattern) | Regex pattern to match webhook modified/new files to. Defaults to any file with `.hcl` or `.tf` extension. | `string` | `".+\\.(hcl|tf)$\n"` | no |
 | <a name="input_github_token_ssm_description"></a> [github\_token\_ssm\_description](#input\_github\_token\_ssm\_description) | Github token SSM parameter description | `string` | `"Github token used by Merge Lock Lambda Function"` | no |
 | <a name="input_github_token_ssm_key"></a> [github\_token\_ssm\_key](#input\_github\_token\_ssm\_key) | AWS SSM Parameter Store key for sensitive Github personal token used by the Merge Lock Lambda Function | `string` | `null` | no |
 | <a name="input_github_token_ssm_tags"></a> [github\_token\_ssm\_tags](#input\_github\_token\_ssm\_tags) | Tags for Github token SSM parameter | `map(string)` | `{}` | no |
-| <a name="input_github_token_ssm_value"></a> [github\_token\_ssm\_value](#input\_github\_token\_ssm\_value) | Registered Github webhook token associated with the Github provider. The token will be used by the Merge Lock Lambda Function.<br>If not provided, module looks for pre-existing SSM parameter via `var.github_token_ssm_key`".<br>GitHub token needs the `repo` permission to send commit statuses for private repos. (see more about OAuth scopes here: https://docs.github.com/en/developers/apps/building-oauth-apps/scopes-for-oauth-apps) | `string` | `""` | no |
+| <a name="input_github_token_ssm_value"></a> [github\_token\_ssm\_value](#input\_github\_token\_ssm\_value) | Registered Github token associated with the Github provider. If not provided, <br>module looks for pre-existing SSM parameter via `var.github_token_ssm_key`".<br>GitHub token needs the `repo` permission to send commit statuses and write comments <br>for private repos (see more about OAuth scopes here: <br>https://docs.github.com/en/developers/apps/building-oauth-apps/scopes-for-oauth-apps) | `string` | `""` | no |
 | <a name="input_lambda_approval_request_vpc_config"></a> [lambda\_approval\_request\_vpc\_config](#input\_lambda\_approval\_request\_vpc\_config) | VPC configuration for Lambda approval request function.<br>Ensure that the configuration allows for outgoing HTTPS traffic. | <pre>object({<br>    subnet_ids         = list(string)<br>    security_group_ids = list(string)<br>  })</pre> | `null` | no |
 | <a name="input_lambda_approval_response_vpc_config"></a> [lambda\_approval\_response\_vpc\_config](#input\_lambda\_approval\_response\_vpc\_config) | VPC configuration for Lambda approval response function.<br>Ensure that the configuration allows for outgoing HTTPS traffic. | <pre>object({<br>    subnet_ids         = list(string)<br>    security_group_ids = list(string)<br>  })</pre> | `null` | no |
 | <a name="input_lambda_trigger_sf_vpc_config"></a> [lambda\_trigger\_sf\_vpc\_config](#input\_lambda\_trigger\_sf\_vpc\_config) | VPC configuration for Lambda trigger\_sf function.<br>Ensure that the configuration allows for outgoing HTTPS traffic. | <pre>object({<br>    subnet_ids         = list(string)<br>    security_group_ids = list(string)<br>  })</pre> | `null` | no |

docker/src/common/utils.py

@@ -82,3 +82,37 @@ def send_commit_status(state: str, target_url: str):
         context=os.environ["STATUS_CHECK_NAME"],
         target_url=target_url,
     )
+
+
+def tf_to_diff(matchobj) -> str:
+    """Replaces Terraform plan syntax with GitHub markdown diff syntax"""
+    if matchobj["add"]:
+        return matchobj["add"].replace("+", " ").replace(" ", "+", 1)
+
+    elif matchobj["minus"]:
+        return matchobj["minus"].replace("-", " ").replace(" ", "-", 1)
+
+    elif matchobj["update"]:
+        # replace ~ with ! to highlight with orange
+        return matchobj["update"].replace("~", " ").replace(" ", "!", 1)
+
+
+def get_diff_block(plan) -> str:
+    """
+    Returns Terraform plan as a markdown diff code block
+
+    Arguments:
+        plan: Terraform Plan stdout without color formatting (use -no-color flag for plan cmd)
+    """
+    diff = re.sub(
+        r"((?P<add>^\s*\+)|(?P<minus>^\s*\-)|(?P<update>^\s*\~))",
+        tf_to_diff,
+        plan,
+        flags=re.MULTILINE,
+    )
+
+    return f"""
+``` diff
+{diff}
+```
+"""

docker/src/pr_plan/plan.py

@@ -1,11 +1,14 @@
 import os
 import logging
+import json
 from pprint import pformat
 import subprocess
 import sys
+
 import github
-import json
-from common.utils import get_task_log_url
+
+sys.path.append(os.path.dirname(__file__) + "/..")
+from common.utils import get_task_log_url, get_diff_block
 
 log = logging.getLogger(__name__)
 stream = logging.StreamHandler(sys.stdout)
@@ -14,18 +17,42 @@
 log.setLevel(logging.DEBUG)
 
 
+def comment_pr_plan(plan: str) -> str:
+    plan_block = get_diff_block(plan)
+    comment = f"""
+## Open PR Infrastructure Changes
+### Directory: {os.environ["CFG_PATH"]}
+<details open>
+<summary>Plan</summary>
+<br>
+{plan_block}
+</details>
+"""
+    pr = (
+        github.Github(os.environ["GITHUB_TOKEN"], retry=3)
+        .get_repo(os.environ["REPO_FULL_NAME"])
+        .get_pull(int(os.environ["PR_ID"]))
+    )
+    pr.create_issue_comment(comment)
+
+    return comment
+
+
 def main() -> None:
     """
     Runs Terragrunt plan command on Terragrunt directory that has been modified
     and send a commit status if enabled.
     """
 
-    cmd = f'terragrunt plan --terragrunt-working-dir {os.environ["CFG_PATH"]} --terragrunt-iam-role {os.environ["ROLE_ARN"]}'
+    cmd = f'terragrunt plan --terragrunt-working-dir {os.environ["CFG_PATH"]} --terragrunt-iam-role {os.environ["ROLE_ARN"]} -no-color'
     log.debug(f"Command: {cmd}")
     try:
         run = subprocess.run(cmd.split(" "), capture_output=True, text=True, check=True)
         log.info(run.stdout)
         state = "success"
+        if os.environ.get("COMMENT_PLAN"):
+            comment_pr_plan(run.stdout)
+
     except subprocess.CalledProcessError as e:
         log.info(e.stderr)
         log.info(e)

docker/src/terra_run/run.py

@@ -3,16 +3,19 @@
 import subprocess
 import sys
 import json
+import ast
 from typing import List
+
 import aurora_data_api
-import ast
+import github
 import boto3
 
 sys.path.append(os.path.dirname(__file__) + "/..")
 from common.utils import (
     subprocess_run,
     send_commit_status,
     get_task_log_url,
+    get_diff_block,
 )
 
 log = logging.getLogger(__name__)
@@ -89,6 +92,29 @@ def update_new_resources() -> None:
         log.info("New provider resources were not created -- skipping")
 
 
+def comment_terra_run_plan(plan) -> str:
+    """Sends a GitHub PR comment for the run's Terraform plan"""
+    plan_block = get_diff_block(plan)
+    comment = f"""
+## Deployment Infrastructure Changes
+### Directory: {os.environ["CFG_PATH"]}
+### Execution ID: {os.environ["EXECUTION_ID"]}
+<details open>
+<summary>Plan</summary>
+<br>
+{plan_block}
+</details>
+"""
+    pr = (
+        github.Github(os.environ["GITHUB_TOKEN"], retry=3)
+        .get_repo(os.environ["REPO_FULL_NAME"])
+        .get_pull(int(os.environ["PR_ID"]))
+    )
+    pr.create_issue_comment(comment)
+
+    return comment
+
+
 def main() -> None:
     """
     Primarily this function prints the results of the Terragrunt command. If the
@@ -106,11 +132,10 @@ def main() -> None:
             text=True,
             check=True,
         )
-        print(run.stdout)
+        log.info(run.stdout)
         state = "success"
     except subprocess.CalledProcessError as e:
-        print(e.stderr)
-        print(e)
+        log.error(e)
         state = "failure"
 
     log_url = get_task_log_url()
@@ -122,6 +147,9 @@ def main() -> None:
             )
             # send ECS task log url with task token to allow Request Approval state to use log url
             # within approval email
+            if os.environ.get("COMMENT_PLAN"):
+                log.info("Commenting Terraform plan results")
+                comment_terra_run_plan(run.stdout)
             if state == "success":
                 output = json.dumps({"LogsUrl": log_url})
                 sf.send_task_success(taskToken=os.environ["TASK_TOKEN"], output=output)

fargate.tf

@@ -247,6 +247,10 @@ resource "aws_ecs_task_definition" "pr_plan" {
           {
             name  = "LOG_STREAM_PREFIX"
             value = local.pr_plan_log_stream_prefix
+          },
+          {
+            name  = "COMMENT_PLAN"
+            value = var.enable_gh_comment_pr_plan ? "true" : ""
           }
         ],
         local.ecs_tasks_base_env_vars,

functions/webhook_receiver/invoker.py

@@ -56,6 +56,7 @@ def trigger_pr_plan(
     base_ref: str,
     head_ref: str,
     head_sha: str,
+    pr_id: int,
     logs_url: str,
     send_commit_status: bool,
 ) -> None:
@@ -140,6 +141,7 @@ def trigger_pr_plan(
                                             "name": "COMMIT_ID",
                                             "value": head_sha,
                                         },
+                                        {"name": "PR_ID", "value": str(pr_id)},
                                         {"name": "CFG_PATH", "value": path},
                                         {
                                             "name": "ROLE_ARN",

functions/webhook_receiver/lambda_function.py

@@ -30,12 +30,13 @@ def open_pr(request: Request):
     )
 
     trigger_pr_plan(
-        event.body.repository.full_name,
-        event.body.pull_request.base.ref,
-        event.body.pull_request.head.ref,
-        event.body.pull_request.head.sha,
-        context.logs_url,
-        event.body.commit_status_config.get("PrPlan"),
+        repo_full_name=event.body.repository.full_name,
+        base_ref=event.body.pull_request.base.ref,
+        head_ref=event.body.pull_request.head.ref,
+        head_sha=event.body.pull_request.head.sha,
+        pr_id=event.body.pull_request.number,
+        logs_url=context.logs_url,
+        send_commit_status=event.body.commit_status_config.get("PrPlan"),
     )
 
     return JSONResponse(

main.tf

@@ -57,6 +57,18 @@ resource "aws_sfn_state_machine" "this" {
                         "Name"    = "TASK_TOKEN"
                         "Value.$" = "$$.Task.Token"
                       },
+                      {
+                        "Name"    = "CFG_PATH"
+                        "Value.$" = "$.cfg_path"
+                      },
+                      {
+                        "Name"    = "PR_ID"
+                        "Value.$" = "States.Format('{}', $.pr_id)"
+                      },
+                      {
+                        "Name"  = "COMMENT_PLAN"
+                        "Value" = var.enable_gh_comment_approval ? "true" : ""
+                      }
                     ]
                   )
                 }

tests/e2e/base_e2e.py

@@ -94,8 +94,14 @@ def pr_plan_pending_statuses(self, request, mut_output, pr, repo):
         )
         log.debug(f"Expected count: {expected_count}")
         wait = 10
+        attempts = 0
+        max_attempts = 12
         statuses = []
         while len(statuses) != expected_count:
+            if attempts == max_attempts:
+                pytest.fail(
+                    "Max attempt reached -- Lambda Function might have failed beforehand"
+                )
             log.debug(f"Waiting {wait} seconds")
             time.sleep(wait)
             statuses = [
@@ -104,15 +110,23 @@ def pr_plan_pending_statuses(self, request, mut_output, pr, repo):
                 if status.context != mut_output["merge_lock_status_check_name"]
             ]
 
+            attempts += 1
+
         return statuses
 
     @pytest.fixture(scope="class")
     def pr_plan_finished_statuses(self, pr_plan_pending_statuses, mut_output, pr, repo):
         """Returns list of PR plan tasks' finished commit statuses"""
         log.info("Waiting for all PR plan commit statuses to be updated")
         wait = 15
+        attempts = 0
+        max_attempts = 12
         statuses = []
         while len(statuses) != len(pr_plan_pending_statuses):
+            if attempts == max_attempts:
+                pytest.fail(
+                    "Max attempt reached -- ECS task might have failed beforehand"
+                )
             log.debug(f"Waiting {wait} seconds")
             time.sleep(wait)
             statuses = [
@@ -123,6 +137,7 @@ def pr_plan_finished_statuses(self, pr_plan_pending_statuses, mut_output, pr, re
             ]
 
             log.debug(f"Finished count: {len(statuses)}")
+            attempts += 1
 
         return statuses
 

tests/fixtures/terraform/mut/basic/main.tf

@@ -22,7 +22,9 @@ module "mut_infrastructure_live_ci" {
 
   enforce_admin_branch_protection = var.enforce_admin_branch_protection
 
-  commit_status_config = var.commit_status_config
+  enable_gh_comment_pr_plan  = true
+  enable_gh_comment_approval = true
+  commit_status_config       = var.commit_status_config
 
   metadb_name         = var.metadb_name
   metadb_username     = var.metadb_username

tests/unit/docker/test_pr_plan.py

@@ -0,0 +1,64 @@
+import os
+import logging
+from unittest.mock import patch
+
+from docker.src.pr_plan.plan import comment_pr_plan
+
+log = logging.getLogger(__name__)
+log.setLevel(logging.DEBUG)
+
+
+@patch("github.Github")
+@patch.dict(
+    os.environ,
+    {"CFG_PATH": "terraform/cfg", "REPO_FULL_NAME": "user/repo", "PR_ID": "1"},
+)
+def test_comment_pr_plan(mock_gh):
+    """Ensures comment_pr_plan() formats the comment's diff block properly and returns the expected comment"""
+    plan = """
+
+Changes to Outputs:
+  - bar = "old" -> null
+  + baz = "new"
+  ~ foo = "old" -> "new"
+
+You can apply this plan to save these new output values to the Terraform
+state, without changing any real infrastructure.
+
+─────────────────────────────────────────────────────────────────────────────
+
+Note: You didn't use the -out option to save this plan, so Terraform can't
+guarantee to take exactly these actions if you run "terraform apply" now.
+
+"""
+    expected = """
+## Open PR Infrastructure Changes
+### Directory: terraform/cfg
+<details open>
+<summary>Plan</summary>
+<br>
+
+``` diff
+
+
+Changes to Outputs:
+-   bar = "old" -> null
++   baz = "new"
+!   foo = "old" -> "new"
+
+You can apply this plan to save these new output values to the Terraform
+state, without changing any real infrastructure.
+
+─────────────────────────────────────────────────────────────────────────────
+
+Note: You didn't use the -out option to save this plan, so Terraform can't
+guarantee to take exactly these actions if you run "terraform apply" now.
+
+
+```
+
+</details>
+"""
+    actual = comment_pr_plan(plan)
+
+    assert actual == expected