From 99ba0423d95c23f61b204024cb0fa7a2d464469b Mon Sep 17 00:00:00 2001 From: Laurie O Date: Fri, 3 Feb 2023 15:46:44 +1000 Subject: [PATCH 01/10] Document more package index host/cache/mirror options --- source/guides/index-mirrors-and-caches.rst | 157 +++++++++++++++++++-- 1 file changed, 147 insertions(+), 10 deletions(-) diff --git a/source/guides/index-mirrors-and-caches.rst b/source/guides/index-mirrors-and-caches.rst index 18dc3463e..1223c420e 100644 --- a/source/guides/index-mirrors-and-caches.rst +++ b/source/guides/index-mirrors-and-caches.rst @@ -11,16 +11,20 @@ Package index mirrors and caches :local: -Mirroring or caching of PyPI can be used to speed up local package installation, -allow offline work, handle corporate firewalls or just plain Internet flakiness. +Mirroring or caching of PyPI (and other package indexes) can be used to speed +up local package installation, allow offline work, work with corporate +firewalls or handle just plain internet flakiness. -Three options are available in this area: +There are multiple classes of options in this area: -1. pip provides local caching options, -2. devpi provides higher-level caching option, potentially shared amongst - many users or machines, and -3. bandersnatch provides a local complete mirror of all PyPI :term:`packages - `. +1. local/hosted caching of package indexes. + +2. local/hosted mirroring of a package index. A mirror is a (whole or + partial) copy of a package index, which can be used in place of the + original index. + +3. private package index with fall-through to public package indexes (for + example, to mitigate dependency confusion attacks). Caching with pip @@ -50,17 +54,150 @@ documentation for getting started`__. __ https://devpi.net/docs/devpi/devpi/latest/+d/quickstart-pypimirror.html +devpi has additional funcionality, such as mirroring package indexes, running +multiple indexes with a concept of inheritance, syncing between multiple +servers, index replication and fail-over, and package upload. + +* `devpi on PyPI `_ +* `devpi source `_ + Complete mirror with bandersnatch ---------------------------------- -bandersnatch will set up a complete local mirror of all PyPI :term:`packages +bandersnatch will set up a complete local (or `AWS S3`_) mirror of all PyPI +:term:`packages ` (externally-hosted packages are not mirrored). See the `bandersnatch documentation for getting that going`__. -__ https://github.com/pypa/bandersnatch/ +__ https://bandersnatch.readthedocs.io/en/latest/ A benefit of devpi is that it will create a mirror which includes :term:`packages ` that are external to PyPI, unlike bandersnatch which will only cache :term:`packages ` hosted on PyPI. + +* `bandersnatch on PyPI `_ +* `bandersnatch source `_ + + +Other package index servers +--------------------------- + +In the following + +simpleindex +^^^^^^^^^^^ + +Routes URLs to multiple package indexes (including PyPI), serves local (or +`AWS S3`_, with a plugin) directory of packages, no caching without custom +plugins, no mirroring. + +* `simpleindex on PyPI `_ +* `simpleindex source / documentation + `_ + +pypiserver +^^^^^^^^^^ + +Serves local directory of packages, no fall-through to package indexes +(including PyPI), supports package upload. + +* `pypiserver on PyPI `_ +* `pypiserver source / documentation + `_ + +pypiprivate +^^^^^^^^^^^ + +Serves local (or `AWS S3`_-hosted) directory of packages, no fall-through to +package indexes (including PyPI). + +* `pypiprivate on PyPI `_ +* `pypiprivate source / documentation + `_ + +Python package index plugin for Pulp +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Supports local/`AWS S3`_ mirrors, package upload, proxying to multiple indexes, no +caching. + +* `pulp_python on PyPI `_ +* `pulp_python documentation `_ +* `pulp_python source `_ + +pip2pi +^^^^^^ + +Manual syncing of specific packages, no proxy. + +* `pip2pi on PyPI `_ +* `pip2pi source / documenation `_ + +proxpi +^^^^^^ + +Package index caching proxy, supports multiple indexes, no mirroring. + +* `proxpi on PyPI `_ +* `proxpi source `_ + +Flask-Pypi-Proxy +^^^^^^^^^^^^^^^^ + +.. warning:: Not maintained, project archived + +Caches PyPI. No cache size limit, no caching index pages. + +* `Flask-Pypi-Proxy on PyPI `_ +* `Flask-Pypi-Proxy documentation + `_ +* `Flask-Pypi-Proxy source `_ + +http.server +^^^^^^^^^^^ + +Standard-library, hosts directory exactly as laid out, no proxy to package +indexes (eg PyPI). See more in :ref:`Hosting your Own Simple Repository`. + +* `http.server documentation + `_ + +Apache +^^^^^^ + +Using +`mod_rewrite `_ and +`mod_cache_disk +`_, +you can cache requests to package indexes through an Apache server. + +Gemfury +^^^^^^^ + +Hosted and managed solution. Private indexes are not free, documentation +doesn't say anything about fall-through. + +* `Host Python packages on Gemfury `_ +* `Gemfure PyPI documentation `_ + +GitLab Package Registry +^^^^^^^^^^^^^^^^^^^^^^^ + +Hosted and managed solution. Private and public package index with +optional fall-through, permissioning. + +* `GitLab documentation + `_ + +Azure Artifacts +^^^^^^^^^^^^^^^ + +Hosted and managed solution. Private package index with optional +fall-through. + +* `Azure documentation + `_ + +.. _`AWS S3`: https://aws.amazon.com/s3/ From 928c14f51227a679c060925dedbfe90e4bce0ca8 Mon Sep 17 00:00:00 2001 From: Laurie O Date: Tue, 7 Feb 2023 15:52:26 +1000 Subject: [PATCH 02/10] Add more hosted package indexes Artifactory, Nexus, AWS CodeArtifact, Google Artifact Registry --- source/guides/index-mirrors-and-caches.rst | 49 ++++++++++++++++++++-- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/source/guides/index-mirrors-and-caches.rst b/source/guides/index-mirrors-and-caches.rst index 1223c420e..04c0b8a34 100644 --- a/source/guides/index-mirrors-and-caches.rst +++ b/source/guides/index-mirrors-and-caches.rst @@ -182,6 +182,28 @@ doesn't say anything about fall-through. * `Host Python packages on Gemfury `_ * `Gemfure PyPI documentation `_ +Artifactory +^^^^^^^^^^^ + +Hosted and managed solution. Proxy (with caching) multiple package indexes, and +host a new package index (supporting upload) with fall-through. Can be +self-hosted (not for free). + +* `JFrog Artifactory `_ +* `PyPI Repositories on Artifactory documentation + `_ + +Nexus Repository Manager +^^^^^^^^^^^^^^^^^^^^^^^^ + +Hosted and managed solution. Proxy (with caching) multiple package indexes, and +host a new package index (supporting upload) with fall-through. + +* `Sonatype Nexus repository + `_ +* `PyPI documentation for Nexus + `_ + GitLab Package Registry ^^^^^^^^^^^^^^^^^^^^^^^ @@ -191,13 +213,34 @@ optional fall-through, permissioning. * `GitLab documentation `_ +AWS CodeArtifact +^^^^^^^^^^^^^^^^ + +Hosted and managed solution. Private package index with optional cached +fall-through to PyPI. + +* `AWS CodeArtifact `_ +* `Python packages on CodeArtifact documentation + `_ + Azure Artifacts ^^^^^^^^^^^^^^^ -Hosted and managed solution. Private package index with optional -fall-through. +Hosted and managed solution. Private package index with optional fall-through. -* `Azure documentation +* `Azure Artifacts + `_ +* `Python packages on Azure Artifacts documentation `_ +Google Artifact Registry +^^^^^^^^^^^^^^^^^^^^^^^^ + +Hosted and managed solution. Private package index with no fall-through nor +mirroring. + +* `Google Artifact Registry `_ +* `Python packages on Artifact Registry documentation + `_ + .. _`AWS S3`: https://aws.amazon.com/s3/ From cd4e1e75dcf4a39639ef0510c7eea61382c23c35 Mon Sep 17 00:00:00 2001 From: Laurie O Date: Tue, 7 Feb 2023 15:52:35 +1000 Subject: [PATCH 03/10] Define proxy --- source/guides/index-mirrors-and-caches.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/source/guides/index-mirrors-and-caches.rst b/source/guides/index-mirrors-and-caches.rst index 04c0b8a34..1e3be4f7a 100644 --- a/source/guides/index-mirrors-and-caches.rst +++ b/source/guides/index-mirrors-and-caches.rst @@ -24,7 +24,8 @@ There are multiple classes of options in this area: original index. 3. private package index with fall-through to public package indexes (for - example, to mitigate dependency confusion attacks). + example, to mitigate dependency confusion attacks), also known as a + proxy. Caching with pip From 60fa5675a59eaa476dc28c1cfcd0b03351afdb79 Mon Sep 17 00:00:00 2001 From: Laurie O Date: Tue, 7 Feb 2023 15:53:31 +1000 Subject: [PATCH 04/10] Remove errant paragraph --- source/guides/index-mirrors-and-caches.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/source/guides/index-mirrors-and-caches.rst b/source/guides/index-mirrors-and-caches.rst index 1e3be4f7a..a204b56ea 100644 --- a/source/guides/index-mirrors-and-caches.rst +++ b/source/guides/index-mirrors-and-caches.rst @@ -85,8 +85,6 @@ hosted on PyPI. Other package index servers --------------------------- -In the following - simpleindex ^^^^^^^^^^^ From 54b8ae50a5d21f256617b017b5e64d26ba9d80a2 Mon Sep 17 00:00:00 2001 From: Laurie O Date: Fri, 24 Feb 2023 12:46:14 +1000 Subject: [PATCH 05/10] Add 3 more options --- source/guides/index-mirrors-and-caches.rst | 29 ++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/source/guides/index-mirrors-and-caches.rst b/source/guides/index-mirrors-and-caches.rst index a204b56ea..3d6a0890c 100644 --- a/source/guides/index-mirrors-and-caches.rst +++ b/source/guides/index-mirrors-and-caches.rst @@ -116,6 +116,27 @@ package indexes (including PyPI). * `pypiprivate source / documentation `_ +PyPI Cloud +^^^^^^^^^^ + +PyPI server, backed by `AWS S3`_, another cloud storage service, or local +files. Supports redirect/cached proxying, authentication and authorisation, no +mirroring. + +* `PyPI Cloud on PyPI `_ +* `PyPI Cloud source `_ +* `PyPI Cloud documentation `_ + +pywharf +^^^^^^^ + +.. warning:: Not maintained, project archived + +PyPI server, backed by GitHub or local files. No proxy or mirror. + +* `pywharf on PyPI `_ +* `pywharf source `_ + Python package index plugin for Pulp ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -203,6 +224,14 @@ host a new package index (supporting upload) with fall-through. * `PyPI documentation for Nexus `_ +Coherent Minds PyPI Filter +^^^^^^^^^^^^^^^^^^^^^^^^ + +Hosted and managed solution. Only filters requests, redirecting to PyPI if not +filtered, and blocking requests otherwise. + +* `Coherent Minds PyPI filter `_ + GitLab Package Registry ^^^^^^^^^^^^^^^^^^^^^^^ From 347671202f41ffe9c276ed905c7c4842a2e67253 Mon Sep 17 00:00:00 2001 From: Laurie O Date: Fri, 24 Feb 2023 13:29:48 +1000 Subject: [PATCH 06/10] Fix RST --- source/guides/index-mirrors-and-caches.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/guides/index-mirrors-and-caches.rst b/source/guides/index-mirrors-and-caches.rst index 3d6a0890c..18ba3f545 100644 --- a/source/guides/index-mirrors-and-caches.rst +++ b/source/guides/index-mirrors-and-caches.rst @@ -140,8 +140,8 @@ PyPI server, backed by GitHub or local files. No proxy or mirror. Python package index plugin for Pulp ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Supports local/`AWS S3`_ mirrors, package upload, proxying to multiple indexes, no -caching. +Supports local/`AWS S3`_ mirrors, package upload, proxying to multiple indexes, +no caching. * `pulp_python on PyPI `_ * `pulp_python documentation `_ @@ -225,7 +225,7 @@ host a new package index (supporting upload) with fall-through. `_ Coherent Minds PyPI Filter -^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^^^^^ Hosted and managed solution. Only filters requests, redirecting to PyPI if not filtered, and blocking requests otherwise. From 70bee6145fc3d9db9d5be92460718dde59379e44 Mon Sep 17 00:00:00 2001 From: Laurie O Date: Mon, 27 Feb 2023 10:52:24 +1000 Subject: [PATCH 07/10] Fix typo --- source/guides/index-mirrors-and-caches.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/guides/index-mirrors-and-caches.rst b/source/guides/index-mirrors-and-caches.rst index 18ba3f545..9ddbbf09e 100644 --- a/source/guides/index-mirrors-and-caches.rst +++ b/source/guides/index-mirrors-and-caches.rst @@ -55,7 +55,7 @@ documentation for getting started`__. __ https://devpi.net/docs/devpi/devpi/latest/+d/quickstart-pypimirror.html -devpi has additional funcionality, such as mirroring package indexes, running +devpi has additional functionality, such as mirroring package indexes, running multiple indexes with a concept of inheritance, syncing between multiple servers, index replication and fail-over, and package upload. From 7a3e2f7a81f84ac2b770898c994b53cf78ad2829 Mon Sep 17 00:00:00 2001 From: Laurie O Date: Mon, 27 Feb 2023 10:55:35 +1000 Subject: [PATCH 08/10] Better describe 'simpleindex' S3 support --- source/guides/index-mirrors-and-caches.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/guides/index-mirrors-and-caches.rst b/source/guides/index-mirrors-and-caches.rst index 9ddbbf09e..2fe6ab001 100644 --- a/source/guides/index-mirrors-and-caches.rst +++ b/source/guides/index-mirrors-and-caches.rst @@ -89,8 +89,8 @@ simpleindex ^^^^^^^^^^^ Routes URLs to multiple package indexes (including PyPI), serves local (or -`AWS S3`_, with a plugin) directory of packages, no caching without custom -plugins, no mirroring. +cloud-hosted, for example `AWS S3`_, with a custom plugin) directory of +packages, no caching without custom plugins, no mirroring. * `simpleindex on PyPI `_ * `simpleindex source / documentation From bf319bbd99f8f30c04f2a702d0a53ca8fcc3a1c3 Mon Sep 17 00:00:00 2001 From: Laurie O Date: Wed, 8 Nov 2023 11:31:01 +1000 Subject: [PATCH 09/10] Update update-date --- source/guides/index-mirrors-and-caches.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/guides/index-mirrors-and-caches.rst b/source/guides/index-mirrors-and-caches.rst index 257fd7d17..4958d5b55 100644 --- a/source/guides/index-mirrors-and-caches.rst +++ b/source/guides/index-mirrors-and-caches.rst @@ -4,8 +4,8 @@ Package index mirrors and caches ================================ -:Page Status: Incomplete -:Last Reviewed: 2014-12-24 +:Page Status: Ongoing +:Last Reviewed: 2023-02-27 Mirroring or caching of PyPI (and other package indexes) can be used to speed From 2f619e13e22f72dfdf2ea472975aa60768d3653f Mon Sep 17 00:00:00 2001 From: Laurie O Date: Wed, 8 Nov 2023 11:33:53 +1000 Subject: [PATCH 10/10] Add nginx_pypi_cache --- source/guides/index-mirrors-and-caches.rst | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/source/guides/index-mirrors-and-caches.rst b/source/guides/index-mirrors-and-caches.rst index 4958d5b55..45eadcb30 100644 --- a/source/guides/index-mirrors-and-caches.rst +++ b/source/guides/index-mirrors-and-caches.rst @@ -5,7 +5,7 @@ Package index mirrors and caches ================================ :Page Status: Ongoing -:Last Reviewed: 2023-02-27 +:Last Reviewed: 2023-11-08 Mirroring or caching of PyPI (and other package indexes) can be used to speed @@ -160,6 +160,13 @@ Package index caching proxy, supports multiple indexes, no mirroring. * `proxpi on PyPI `_ * `proxpi source `_ +nginx_pypi_cache +^^^^^^^^^^^^^^^^ + +Package index caching proxy using `nginx `_ + +* `nginx_pypi_cache source `_ + Flask-Pypi-Proxy ^^^^^^^^^^^^^^^^