diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index e000d3934b1..db44eec3c69 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -672,6 +672,7 @@ peps/pep-0791.rst @vstinner peps/pep-0792.rst @dstufft peps/pep-0793.rst @encukou peps/pep-0794.rst @brettcannon +peps/pep-0796.rst @ncoghlan # ... peps/pep-0801.rst @warsaw # ... diff --git a/peps/pep-0796.rst b/peps/pep-0796.rst new file mode 100644 index 00000000000..3cc10fe838e --- /dev/null +++ b/peps/pep-0796.rst @@ -0,0 +1,263 @@ +PEP: 796 +Title: Relative Virtual Environment Home +Author: Richard Levasseur +Sponsor: Alyssa Coghlan +Discussions-To: Pending +Status: Draft +Type: Standards Track +Created: 26-Jun-2025 +Python-Version: 3.15 + +.. highlight:: rst + + +Abstract +======== + +This PEP describes how a relative path for ``home`` in a Python virtual +environment's ``pyvenv.cfg`` is understood by the Python startup process. +Specifically, how it is canonicalized into an absolute path later used +by the runtime. This small detail is a fundamental building block for +virtual environments to be more portable. + +Motivation +========== + +There are two main motivations for allowing relative paths in ``pyvenv.cfg``. + +First, it is currently prescribed that the ``home`` value in ``pyvenv.cfg`` be +an absolute path. The behavior of relative paths is unspecified. While +techniques exist to work around this for every other sub-part of a virtual +environment, the one remaining part without a tenable solution is how the +Python runtime itself finds ``PYTHONHOME``. This is because, currently, the +startup process requires absolute paths be used for the ``home`` key in +``pyvenv.cfg``. If a relative path is used, behavior is unspecified (the +current implementation ends up making it relative to the process's current +working directory, making it untenable to use). + +This requirement is overly proscriptive and restrictive because, given a known +anchor point, it's easy to transform a relative path to an absolute path and +still retain predictable and reliable behavior. Thus, the absolute path +requirement should be relaxed and relative path behavior allowed and defined. + +Second, such relative paths are a building block to enable portable virtual +environments, i.e. copying a virtual environment as-is between hosts of +compatible platforms. Portable venvs are appealing because virtual environments +are a popular mechanism for running Python applications. This provides several +benefits: + +* The closer the development environment is to the non-development environment, + the more reliable software can be achieved, and the easier it is to reproduce + issues +* The simpler the process of re-creating the environment, the more reliable + software can be achieved, and the faster the process can be. + +Making it simpler to copy a virtual environment from one host to another +mitigates these categories of problems. Additionally, the development tools to +create a virtual environment and install its dependencies aren't needed on the +host that intends to run the program. + +When the virtual environment doesn't require modifications to be usable, it +also allows more advanced deployment mechanisms, e.g. remote mounting and +caching of artifacts. While this PEP on its own isn't sufficient to enable +that, it allows tools like ``bazel`` or ``venvstacks`` to more easily prepare +constrained environments that allow for such use cases. + +Rationale +========= + +The reason support for relative virtual environments needs to be +in the interpreter itself is because locating ``PYTHONHOME`` happens +very early in the interpreter startup process, which limits the options for +customizing how it's computed. Without the ability to specify where the +supporting Python runtime files are, the interpreter can't finish startup, +so other hook points (e.g. ``site`` initialization) never trigger. + +Tools that currently look to enable virtual environment portability across +machines do so either by relying on undocumented interpreter behaviour +(``bazel``, omitting the ``home`` key entirely to trigger an implementation +dependent fallback to resolving via a symlinked interpreter binary on +non-Windows systems) or by requiring a post-installation script to be executed +after the environment is placed in its target location (``venvstacks``). + +Specification +============= + +The ``home`` value in ``pyvenv.cfg`` is permitted to use a relative path value. +These may contain up-references outside of the virtual environment root +directory. Examples: + +* ``subdir/whatever/bin`` (a directory within the virtual environment). +* ``./subdir/whatever/bin`` (same as above) +* ``../../../../../elsewhere/runtime/bin`` (a directory outside the virtual + environment). + +Relative paths are relative to the directory containing ``pyvenv.cfg``. During +interpreter startup (i.e. ``getpath.py``), the relative path is joined to the +directory to form an absolute path. Up-references (``../``) and current +directory references (``./``) are resolved syntactically (i.e. not resolving +symlinks). Symlinks are *not* resolved prior to construction of the absolute +path to ensure semantics between a relative path and absolute path remain the +same. + +For example, given +``/home/user/venv/bin/pyvenv.cfg`` with +``home = ../../runtime/./bin``, the result is ``home = /home/user/runtime/bin``, +i.e. it's equivalent to using that value verbatim in ``pyvenv.cfg``. + + +CPython Runtime Changes +======================= + +The CPython runtime itself *almost* already supports relative paths. The +primitives are there, so the only change needed is to define how it resolves +relative paths for ``home`` in ``pyvenv.cfg``. + +Currently, relative paths resolve relative to the process's current working +directory. Because CWD isn't knowable in advance, it makes relative paths today +effectively impossible. + +Instead, the paths should be relative to the location of the ``pyvenv.cfg`` +file. This file is chosen as the anchor point because the tool that creates the +file also has to know where the Python runtime is, so can easily calculate the +correct relative path. For tools that read the ``pyvenv.cfg``, it is also easy +to simply join the directory name of where ``pyvenv.cfg`` was found with the +path in the config file. When a person reads the config file, they can do +something similar, which results in a lower cognitive burden and helps avoid +the question of "relative to what?" + +This change is only a couple of lines in the startup code. Specifically, when +parsing the ``pyvenv.cfg`` file and finding the ``home`` value, it just needs +to be checked if it's already absolute. If not, then join it to the directory +name of the ``pyvenv.cfg`` file. The code already knows the directory and has +helpers already present for checking if a path is absolute and joining two +paths. + +A proof-of-concept of this is implemented in +`rickeylev/feat.relative.pyvenv.home `__ + +Backwards Compatibility +======================= + + +Tools that work around the absolute ``home`` key limitation the way ``bazel`` +and ``venvstacks`` currently do (omitting the ``home`` key, or editing it after +moving the environment) will be unaffected. + +While the PEP author and sponsor aren't aware of any projects that work around +the limitation by carefully controlling the current working directory used to +launch the deployed Python environments on target systems, any such projects +would be unaffected if they already ensured the working directory was set to +the folder containing ``pyvenv.cfg`` (which seems like a plausible choice, +since that is typically the root directory of the virtual environment). In the +even more unlikely case where that assumption doesn't hold, tools generating +relative virtual environment paths will typically be aware of the underlying +base runtime Python version, and hence able to update the emitted relative path +accordingly. + + +How to Teach This +================= + +Teaching this should be simple: if you use a relative path in ``pyvenv.cfg``, +then it's relative to the directory containing the ``pyvenv.cfg`` file. This +is simple to explain and understand. + + +Reference Implementation +======================== + +A reference implementation is available by using the combination of: + +* Python runtime from `rickeylev/feat.relative.pyvenv.home `__ +* Relative venv from `rickeylev/relvenv `__ + +And following the +`relvenv README `__. + +Open Issues +=========== + +This PEP does not specify how to create a ``pyvenv.cfg`` with a relative path, +nor how downstream tools (e.g. installers) should identify them or process +them. These questions are best addressed separately by tool owners. + +References +========== + +* `rules_python `__: implements + host-relocatable virtual environments. +* `rules_py `__: implements + host-relocatable virtual environments. +* `python-build-standalone `__: +* `venvstacks `__: a tool for creating reproducible distribution artifacts from virtual environments + A relocatable Python runtime. +* `PoC for relative home in Python startup `__ +* `Python Ideas "Making venvs relocatable friendly" discussion `__ +* `GH-136051: relative pyvenv.cfg home `__ + +Rejected Ideas +===================== + +Relative to virtual env root +---------------------------- + +Having the ``home`` value in ``pyvenv.cfg`` relative to the virtual +environment's root directory would work just as well, but this idea is rejected +because it requires additional effort to compute the virtual env root. + +Unspecified home means to dynamically compute home +---------------------------------------------------- + +Today, if a ``pyvenv.cfg`` file doesn't set ``home``, the runtime will try to +dynamically compute it by checking if the current executable (which is +typically the venv's ``bin/python3`` symlink) is a symlink and, if so, use +where that points as ``PYTHONHOME``. + +While currently used as a workaround by some tools, *standardising* this +behavior is undesirable for a couple reasons: + +1. It presents platform-specific issues, namely with Windows. Windows does + support symlinks, but not by default, and it can require special + permissions to do so. +2. It *requires* that a symlink be used, which precludes using otherwise + equivalent mechanisms for creating an executable (e.g. a wrapper script, + hard links, etc). + +In general, symlinks work best when they aren't special cased by consumers. + +Using the term "relocatable" +---------------------------- + +Discussions pointed out that the term "relocatable" is somewhat ambiguous and +misleading for a couple reasons. + +First, absolute paths make a venv arbitrarily relocatable *within* a host, but +not between hosts, so "relocatable" requires *some* qualification for +clarity. + +Second, when using relative paths that point outside the venv, the venv is only +relocatable insofar as those external artifacts are also relocated. This is an +additional nuance that requires qualification of the term. + +To better avoid this confusion, "relative" is chosen, which more naturally +invites the question *"Relative to what?"*. + + +Using PYTHONHOME at runtime to specify home +------------------------------------------- + +Using the ``PYTHONHOME`` environment variable (or any environment variable) is +problematic because it's difficult to know and control when an environment +variable should or shouldn't be inherited by subprocesses. In some cases, it's +not feasible because of how layers of programs calling programs interact. + +Code generally assumes that any virtual environment will be +automatically detected and activated by the presence of ``pyvenv.cfg``, so +things work better when alterations to the environment aren't a concern. + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.