Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ecs_taskdefinition: use aws_retry to avoid throttling exception #2124

Conversation

eacherkan-aternity
Copy link
Contributor

@eacherkan-aternity eacherkan-aternity commented Jul 16, 2024

SUMMARY

Fixes #2123 by adding aws_retry=True to the API calls.

ISSUE TYPE
  • Bugfix Pull Request
COMPONENT NAME

ecs_taskdefinition

ADDITIONAL INFORMATION

We observed that ecs_taskdefinition intermittently causes a ThrottlingException when running on a task definition with a large number of revisions. Looking at the code, it appears that describe_task_definitions loops over the revisions without using the retry mechanism. This PR attempts to solve the problem by adding aws_retry=True to the API calls.

Due to the nature of the problem (intermittent throttling by AWS), I couldn't devise automated tests that validate the fix.

Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/9521e98fcd9f408db8add5bda6b9e8e1

ansible-galaxy-importer FAILURE in 4m 58s (non-voting)
✔️ build-ansible-collection SUCCESS in 13m 07s
✔️ ansible-test-splitter SUCCESS in 5m 06s
✔️ integration-community.aws-1 SUCCESS in 16m 11s
Skipped 21 jobs

@alinabuzachis alinabuzachis added backport-7 PR should be backported to the stable-7 branch backport-8 labels Jul 18, 2024
Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/f7735d4a4d1c402d8666ff3248745785

✔️ ansible-galaxy-importer SUCCESS in 8m 03s (non-voting)
✔️ build-ansible-collection SUCCESS in 12m 14s
✔️ ansible-test-splitter SUCCESS in 5m 09s
✔️ integration-community.aws-1 SUCCESS in 17m 52s
Skipped 21 jobs

@tremble tremble added the mergeit Merge the PR (SoftwareFactory) label Jul 23, 2024
Copy link
Contributor

Build succeeded (gate pipeline).
https://ansible.softwarefactory-project.io/zuul/buildset/a5dbb568a6544ab6a4fe63039062b1b1

ansible-galaxy-importer FAILURE in 4m 49s (non-voting)
✔️ build-ansible-collection SUCCESS in 12m 20s
✔️ ansible-test-splitter SUCCESS in 4m 56s
✔️ integration-community.aws-1 SUCCESS in 15m 54s
Skipped 21 jobs

@softwarefactory-project-zuul softwarefactory-project-zuul bot merged commit 97131ec into ansible-collections:main Jul 23, 2024
38 checks passed
Copy link

patchback bot commented Jul 23, 2024

Backport to stable-7: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-7/97131eca035974f41ad17357ef9bef8a0a61f675/pr-2124

Backported as #2128

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback bot pushed a commit that referenced this pull request Jul 23, 2024
)

SUMMARY

Fixes #2123 by adding aws_retry=True to the API calls.

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

ecs_taskdefinition
ADDITIONAL INFORMATION

We observed that ecs_taskdefinition intermittently causes a ThrottlingException when running on a task definition with a large number of revisions. Looking at the code, it appears that describe_task_definitions loops over the revisions without using the retry mechanism. This PR attempts to solve the problem by adding aws_retry=True to the API calls.
Due to the nature of the problem (intermittent throttling by AWS), I couldn't devise automated tests that validate the fix.

Reviewed-by: Alina Buzachis
Reviewed-by: Mark Chappell
Reviewed-by: Eli Acherkan
(cherry picked from commit 97131ec)
Copy link

patchback bot commented Jul 23, 2024

Backport to stable-8: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-8/97131eca035974f41ad17357ef9bef8a0a61f675/pr-2124

Backported as #2129

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback bot pushed a commit that referenced this pull request Jul 23, 2024
)

SUMMARY

Fixes #2123 by adding aws_retry=True to the API calls.

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

ecs_taskdefinition
ADDITIONAL INFORMATION

We observed that ecs_taskdefinition intermittently causes a ThrottlingException when running on a task definition with a large number of revisions. Looking at the code, it appears that describe_task_definitions loops over the revisions without using the retry mechanism. This PR attempts to solve the problem by adding aws_retry=True to the API calls.
Due to the nature of the problem (intermittent throttling by AWS), I couldn't devise automated tests that validate the fix.

Reviewed-by: Alina Buzachis
Reviewed-by: Mark Chappell
Reviewed-by: Eli Acherkan
(cherry picked from commit 97131ec)
softwarefactory-project-zuul bot pushed a commit that referenced this pull request Aug 1, 2024
) (#2129)

This is a backport of PR #2124 as merged into main (97131ec).
SUMMARY

Fixes #2123 by adding aws_retry=True to the API calls.

ISSUE TYPE


Bugfix Pull Request

COMPONENT NAME

ecs_taskdefinition
ADDITIONAL INFORMATION

We observed that ecs_taskdefinition intermittently causes a ThrottlingException when running on a task definition with a large number of revisions. Looking at the code, it appears that describe_task_definitions loops over the revisions without using the retry mechanism. This PR attempts to solve the problem by adding aws_retry=True to the API calls.
Due to the nature of the problem (intermittent throttling by AWS), I couldn't devise automated tests that validate the fix.

Reviewed-by: Markus Bergholz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-7 PR should be backported to the stable-7 branch backport-8 mergeit Merge the PR (SoftwareFactory)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ecs_taskdefinition throws a ThrottlingException when the task definition has a large number of revisions
3 participants