Run a scalable mirror of conda-forge on GitHub packages.

[jaimergp]

📌 Summary

Create and maintain a mirror of conda-forge and bioconda on Github packages

📝 Background

Using the OCI capabilities of Github Packages, we can store conda artifacts.
This can be used as a mirror for archival purposes, at the very least.

🚀 Tasks / Deliverables

Beta Give feedback

1. CEP for the OCI storage of conda packages & repodata conda/ceps#70
   vote +1
2. Staging mechanism
3. Repodata indexing mechanism
4. Some packages in conda-forge/osx-64 return 404 channel-mirrors/mirrormirror#11
   +0
5. conda-forge/noarch errors: "Remote end closed connection without response" channel-mirrors/mirrormirror#12
   +0
6. bioconda/noarch: "Max retries exceeded" channel-mirrors/mirrormirror#13
   +0

📅 Estimated completion

This task should be finished in the first 12 months.

ℹ️ References

• Initial prototype
• Github Action to run the sync
• Packages upload to ghcr conda-forge/conda-forge-ci-setup-feedstock#208

[jaimergp]

About direct uploads to ghcr.io, see this document.

[jaimergp]

xref conda-forge/conda-forge-ci-setup-feedstock#208

[jaimergp]

Posted one more comment with my thoughts in the issue above. See conda-forge/conda-forge-ci-setup-feedstock#208 (comment)

[jaimergp]

Current status:

• Cronjob GHA workflow keeps running at https://github.com/channel-mirrors/mirrormirror. We recently fixed a problem with idle workflows that should be resolved now.
• Small 404 problems with some marked-as-broken artifacts (see Something wrong with https://conda.anaconda.org/conda-forge/osx-64/arpack-3.6.1-blas_openblash1f444ea_0.tar.bz2 conda/infrastructure#797 and Mark arpack 3.6.1 as not broken conda-forge/admin-requests#937)

What we need next:

• CEP to document the OCI layering and naming conventions used at https://github.com/orgs/channel-mirrors/packages
• A mechanism to stage and publish recently uploaded packages without Anaconda.org (see comments above for discussions)

[jaimergp]

Important CEP: https://github.com/conda-incubator/ceps/blob/main/cep-15.md

This allows the repodata and packages to live in separate locations.

[jaimergp]

There are some extra errors in osx-arm64 and bioconda caused by connection errors of different types. @goanpeca can you look into this?