Change golang base image to public ECR to avoid rate limit

[Axelcouty]

For issue #69

I believe the best way to avoid that would be to have the action written in a natively supported github action language such as Javascript, but that would do the trick in the meantime 👍

What do you think ?

[rytswd]

Thanks for this! The change itself is probably safe to go in, but I would like to make sure we are fully clear on the implication of using "non-official Docker registry" -- although AWS copy may also be considered as an official repository, I myself am not familiar with it and would love to get some clarification on that! 🙏

[lxe]

A gentle bump on this one. Docker is really limiting public unauthenticated image pulls, especially on persistent self-hosted runners. I'm guessing they are using IP-based origin detection or something similar.

Check out this AWS help article for more guidance:

https://repost.aws/knowledge-center/ecs-pull-container-error-rate-limit

Use Amazon ECR public registry for public container images
Identify the public images that you're using in the Docker file. Use the appropriate search filters to search for these images on the Amazon ECR Public Gallery. You don't need to authenticate to browse the public repositories and pull images. The Amazon ECR Public contains popular base images, including operating systems, AWS-published images, Kubernetes add-ons, and artifacts. Pull images from the Amazon ECR public registry to avoid reaching the Docker Hub's rate limit.

[rytswd]

@lxe Thanks for the input here! It's certainly not our intention to get blocked by the rate limit, and the public gallery from AWS seems to be managed based on the docker's repo-info: https://github.com/docker-library/repo-info/tree/master/repos/golang

Amazon ECR Rate Limit

However, the Amazon ECR side also has some rate limit -- theirs is more to do with the bandwidth, 50GB per month (https://aws.amazon.com/about-aws/whats-new/2020/12/announcing-amazon-ecr-public-and-amazon-ecr-public-gallery/ -- it also mentions 500 GB, and I'm not sure which one's correct).

Authenticated AWS account will get the limit bumped to 5 TB.

Docker Hub Rate Limit

Speaking of authentication, with Docker Hub, you can also get some rate limit bump to 200 pulls per 6 hours, or 5,000 per day for a paid subscription.

Authentication using docker/login-action

Authentication to either Docker repository can be done with an additional step in the GitHub Actions config, such as adding the following:

AWS (ref: https://github.com/docker/login-action#aws-public-elastic-container-registry-ecr)

      - name: Log in to Amazon ECR
        uses: docker/login-action@v2
        with:
          registry: public.ecr.aws
          username: ${{ secrets.AWS_ACCESS_KEY_ID }}
          password: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        env:
          AWS_REGION: <region>

Docker Hub (ref: https://github.com/docker/login-action#docker-hub)

      - name: Log in to Amazon ECR
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

Consideration

I suppose the best way forward is to allow users to choose which Go base image to use based on the release reference, such as merge-gatekeeper@v1-ecr for Amazon ECR, etc. As Go image is 100+MB, that would mean roughly 50,000 pulls / mon would exceed the limit of 5TB for Amazon ECR with simple authentication.

If you have a paid Docker subscription, the rate limit there would actually be much higher in this Go context, at 5,000 pulls / day.

I'm not sure how exactly to manage the release versions -- it may need to be something rather manual, but it should certainly be possible to do. Any thoughts? 🤔

[Axelcouty]

Hey guys thanks for your replies,

Amazon ECR Rate Limit

True ! I does exists too there even if the form is not the same

Authentication using docker/login-action

Thanks for your additional notes, and that's true by paying one can pull a lot more from the public docker repository

I suppose the best way forward is to allow users to choose which Go base image to use based on the release reference, such as merge-gatekeeper@v1-ecr for Amazon ECR, etc. As Go image is 100+MB, that would mean roughly 50,000 pulls / mon would exceed the limit of 5TB for Amazon ECR with simple authentication.

If you have a paid Docker subscription, the rate limit there would actually be much higher in this Go context, at 5,000 pulls / day.

I didn't think about that and it's interesting, while I don't have an answer I agree with your idea to allow users to change the base image if they wish to.

I'm not sure to understand the which Go base image to use based on the release reference part.
If you mean adding some logic that guesses which repository to use based on the release version I'm not sure that would be ideal to maintain. I mean if we just keep the idea of allowing to override the base image I believe you can also leaves users autonomous when they override the base image / repository.

What do you think ?

[rytswd]

If you mean adding some logic that guesses which repository to use based on the release version I'm not sure that would be ideal to maintain. I mean if we just keep the idea of allowing to override the base image I believe you can also leaves users autonomous when they override the base image / repository.

Precisely, what I suggested is not the best solution for sure due to its maintenance overhead. If there is a way to override, that would be ideal -- but I'm personally not sure how one could do that, especially when it's about updating Dockerfile before the step starts executing. I'm open to ideas, just don't have a better one myself... 🫠

[Axelcouty]

but I'm personally not sure how one could do that, especially when it's about updating Dockerfile before the step starts executing

Hmmm 🤔 . Does that mean we don't have the possibility to override the ARGS when building the image when defining merge-gatekeeper's actions ?

merge-gatekeeper:
    runs-on: ubuntu-latest
    permissions:
      checks: read
      statuses: read
    steps:
      - name: Run Merge Gatekeeper
        uses: upsidr/merge-gatekeeper@v1
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          docker-arg-base-image: ?

# ARG BASE_REPOSITORY=golang
ARG GO_VERSION=1.16.7
# Add version flavour
ARG GO_SPECIFIC_TAG=-alpine

FROM ${BASE_REPOSITORY}:${GO_VERSION}${GO_SPECIFIC_TAG}

[ooaklee]

If anyone is interested, we kept encountering the Docker rate limit and decided to fork this action. We have updated it to be a native action by creating a wrapper that targets the appropriate binary depending on the runner architecture. To give it a go, please check it out here: https://github.com/surgeventures/merge-gatekeeper.