Wait till AWS ECS task ends (#243)

An additional loop was added to the `run_cli_task` script that repeatedly checks the status of an AWS ECS task, waiting for it to stop. The current solution should prevent the `Waiter TasksStopped failed: Max attempts exceeded` error that occurred during the execution of the `post_deployment` task. --------- Co-authored-by: mazur <[email protected]>
opensupplyhub · Jun 14, 2024 · 1afbf44 · 1afbf44
1 parent 294eb4f
commit 1afbf44
Show file tree

Hide file tree

Showing 2 changed files with 11 additions and 2 deletions.
diff --git a/deployment/run_cli_task b/deployment/run_cli_task
@@ -1,5 +1,4 @@
 #!/bin/bash
-set -e
 
 vpc_id="$(aws ec2 describe-vpcs --filters=Name=tag:Name,Values=vpcOpenSupplyHub$1 --query 'Vpcs[*].VpcId' --output text)"
 echo "VPC ID: $vpc_id"
@@ -30,7 +29,16 @@ id=$(echo $task_arn | awk -F'/' '{print $NF}')
 echo "Log Stream ID: $id"
 
 echo "Task URL: https://eu-west-1.console.aws.amazon.com/ecs/v2/clusters/ecsOpenSupplyHub${1}Cluster/tasks/$id"
-aws ecs wait tasks-stopped --cluster ecsOpenSupplyHub${1}Cluster --tasks ${task_arn}
+
+echo "Waiting for task to stop..."
+res=1
+while [ $res -ne 0 ]; do
+    echo "Continue waiting: $(date +%T)"
+    aws ecs wait tasks-stopped --cluster "ecsOpenSupplyHub${1}Cluster" --tasks "${task_arn}"
+    res=$?
+done
+
+echo "Stop time: $(date +%T)"
 
 echo "Task stopped reason:"
 aws ecs describe-tasks --cluster ecsOpenSupplyHub${1}Cluster --tasks ${task_arn} --query 'tasks[0].[stoppedReason]'

diff --git a/doc/release/RELEASE-NOTES.md b/doc/release/RELEASE-NOTES.md
@@ -40,6 +40,7 @@ This project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html
 * Removed calling command `clean_facilitylistitems` from the `post_deployment` command.
 * Added calling command `reindex_database` from the `post_deployment` command.
 * Added calling command `index_facilities_new` from the `post_deployment` command.
+* An additional loop was added to the `run_cli_task` script that repeatedly checks the status of an AWS ECS task, waiting for it to stop.
 
 ### Bugfix
 * [OSDEV-1019](https://opensupplyhub.atlassian.net/browse/OSDEV-1019) - Fixed an error message to 'Your account is not verified. Check your email for a confirmation link.' when a user tries to log in with an uppercase letter in the email address and their account has not been activated through the confirmation link.