Docker Hub Rate Limits

[cgreene]

The new Docker Hub rate limits are causing some builds to fail. I especially notice this with appveyor. Here's an example https://ci.appveyor.com/project/cgreene/annorxiver-manuscript/builds/36473656

It might be nice to examine our dependencies and to see which ones are already on the GitHub Container Registry ( https://github.blog/2020-09-01-introducing-github-container-registry/ ).

[agitter]

We could consider falling back to WeasyPrint when the athenapdf Docker pull fails. Having a reliable way to pull the container would be the better solution though.

athenapdf has their Dockerfile available if we need it.

[cgreene]

In the case of Athena, we could probably file a PR to also push to the github container registry at these lines and then pull from there:
https://github.com/arachnys/athenapdf/blob/dc33a8ac7f06ed6054f847aff291e489f981ec9a/.travis.yml#L16-L19

It would require the athena folks setup the access token to allow pushing: https://docs.github.com/en/free-pro-team@latest/packages/getting-started-with-github-container-registry/migrating-to-github-container-registry-for-docker-images#authenticating-with-the-container-registry

I guess, alternatively, we could setup something in the manubot org to push docker containers that we depend on to our own GCR account, but that seems more brittle.

[dhimmel]

Here is the error:

docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.

I think the issue is that other AppVeyor users ran jobs that from the same IP address, exceeding the docker pull rate limit:

The rate limits of 100 container image requests per six hours for anonymous usage, and 200 container image requests per six hours for free Docker accounts are now in effect. Image requests exceeding these limits will be denied until the six hour window elapses.

There's a chance CI providers find a solution.

GitHub Container Registry is not a great solution because it requires authentication even to pull (described here), so would be annoying to configure for desktop users and on AppVeyor.

So might just be easier to give users the option to authenticate with Docker hub.

Caching is a also possibility.

Also switching away from athenapdf to pagedjs.

[cgreene]

I agree that it is because of the new Docker Hub limits. I think we might expect to see these becoming pervasive as manubot use increases.

Are you sure that the GitHub Container Registry requires logging in to pull? That link seems to be referring to the GitHub Package Registry. The Container Registry docs say:

Public images allow anonymous access and can be pulled without authentication or signing in via the CLI.

Also, the naming of this product from GitHub, given the prior one, is just terrible.

[cgreene]

pagedjs looks nice too! is there a ticket for that already?

[dhimmel]

Are you sure that the GitHub Container Registry requires logging in to pull? That link seems to be referring to the GitHub Package Registry.

Ah I wasn't aware of the distinction between "GitHub Packages Docker registry" (scoped to a repo) and the GitHub Packages Docker registry (scoped to an account).

That's great that GHCR will allow anonymous access. On the flip side, publishing images to GHCR from GitHub Actions will require custom credential configuration.

pagedjs looks nice too! is there a ticket for that already?

It comes up in #299, but we should have a dedicated issue. Opened #394.

[agitter]

This excerpt from the GitHub page @dhimmel linked above is worth noting:

Note: GitHub Container Registry is currently in public beta and subject to change. During the beta, storage and bandwidth are free.

There's a chance that post-beta GitHub Container Registry could have the same limitations as Docker Hub.

[cgreene]

Near that it says:

After the beta, the same billing and storage rates that other GitHub Packages registries use will apply to the container registry. For more information, see "About billing for GitHub Packages."

I do agree there's no strong commitment to unlimited, free pulls, but I think that clause is referring to the changes around private ones.

[agitter]

Good catch @cgreene.

The (former) athenapdf maintainer recommended https://thecodingmachine.github.io/gotenberg/ as another alternative. It looks like it is also distributed as a Docker image.