Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log number of images in ECR repository for CI #1998

Merged
merged 1 commit into from
Apr 11, 2024

Conversation

torredil
Copy link
Member

@torredil torredil commented Apr 9, 2024

CI is down:

ERROR: failed to solve: failed to push 209411653980.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:fc52f7d8597340d52f263923768024b8-windows-amd64-ltsc2022: failed commit on ref "manifest-sha256:cf82f3a2999248be710bf71eebfced344c2ceeccdb1a3f412b1f051ae9aebda0": unexpected status from PUT request to https://209411653980.dkr.ecr.us-west-2.amazonaws.com/v2/aws-ebs-csi-driver/manifests/fc52f7d8597340d52f263923768024b8-windows-amd64-ltsc2022: 403 Forbidden

The root cause is that we've hit a limit for images per repository quota. Specifically, the aws-ebs-csi-driver ECR repository had 10,000 images. See https://docs.aws.amazon.com/AmazonECR/latest/userguide/service-quotas.html

This PR adds a helpful log to help debug future occurrences of this issue which may happen once aws-janitor is updated to clean up images but there is a regression which results in hitting the quota.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 9, 2024
@k8s-ci-robot k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Apr 9, 2024
@torredil
Copy link
Member Author

torredil commented Apr 9, 2024

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 9, 2024
Copy link

github-actions bot commented Apr 9, 2024

Code Coverage Diff

This PR does not change the code coverage

@torredil torredil force-pushed the master branch 2 times, most recently from b2b63e9 to 8d36893 Compare April 9, 2024 21:13
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Apr 9, 2024
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 9, 2024
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 9, 2024
@torredil torredil force-pushed the master branch 6 times, most recently from d2f1a10 to deb6e8a Compare April 9, 2024 22:31
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 9, 2024
@torredil torredil changed the title Do not merge - CI debugging Log number of images in ECR repository for CI Apr 9, 2024
@torredil
Copy link
Member Author

/retest

1 similar comment
@torredil
Copy link
Member Author

/retest

@k8s-ci-robot k8s-ci-robot removed the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Apr 10, 2024
@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 10, 2024
@torredil torredil force-pushed the master branch 2 times, most recently from f0e071a to 5c355d7 Compare April 10, 2024 19:03
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 10, 2024
@torredil torredil force-pushed the master branch 3 times, most recently from 6fc84ee to 3ce471a Compare April 10, 2024 20:20
@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 10, 2024
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 10, 2024
hack/e2e/ecr.sh Outdated Show resolved Hide resolved
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 10, 2024
Copy link
Contributor

@ConnorJC3 ConnorJC3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 10, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ConnorJC3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 10, 2024
@torredil
Copy link
Member Author

/unhold

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 11, 2024
@k8s-ci-robot k8s-ci-robot merged commit e92b955 into kubernetes-sigs:master Apr 11, 2024
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants