plain invocation of python subprocess doesn't inherit sys.path for bootstrap_impl=script #2169
Comments
|
At $dayjob I had a similar usecase where I need to have python interpreter with all dependencies set up and with the way the new thing is setup, I am not sure how I could achieve that using the new bootstrap. I had a py_binary that was using sys.executable and just forward the args to the interpreter and using pythonpath env var would just work, but with the new method, I would need to also setup the sys.path myself before invoking the interpreter. Maybe at the very least there is a way to workaround this where sys.executable could be set to something that sets up the sys.path. |
|
I was reading some Python docs (venv or site, i can't remember), and they gave me the following idea:
|
@rickeylev could you provide more details on how this can be implemented, we are having issues with prefect library as well. When you say binary do you mean a py_binary or just a script (bash/python) |
|
Okay I tried the following approach and it seems to work Create sitecustomize.py import os
import sys
def find_site_packages_dirs(root_dir):
# Store all paths with the name 'site-packages'
site_packages_dirs = []
# Walk through the directory tree
for dirpath, dirnames, _ in os.walk(root_dir):
# Check if 'site-packages' is in the list of directories at this level
if "site-packages" in dirnames:
site_packages_path = os.path.join(dirpath, "site-packages")
site_packages_dirs.append(site_packages_path)
# Optionally, remove 'site-packages' from dirnames to skip its subtree
dirnames.remove(
"site-packages"
) # Skip subdirs within 'site-packages' for efficiency
return site_packages_dirs
root_directory = os.environ["RUNFILES_DIR"]
site_packages_paths = find_site_packages_dirs(root_directory)
sys.path.extend(site_packages_paths)I can't think of any situation where this fails. |
|
I've been poking this off and on over the last couple days and it looks pretty promising. It looks approximately like this:
It all looks to be working, but its still prototype quality, so needs cleanup and to see what happens when all the tests are run.
Some problematic cases I can think of are:
EDIT: Just to clarify, thank you for hacking away on this :). I didn't meant to be overly critical. |
Not at all. I love to learn I did think of some of the points you mentioned, with potential ways to address them, but that is not relevant since you are working on a potentially more robust solution. Thanks. |
When `--bootstrap_impl=script` is used, `PYTHONPATH` is no longer used to set the import paths, which means subprocesses no longer inherit the Bazel paths. This is generally a good thing, but breaks when `sys.executable` is used to directly invoke the interpreter. Such an invocation assumes the interpreter will have the same packages available and works with the system_python bootstrap. To fix, have the script bootstrap use a basic virtual env. This allows it to intercept interpreter startup even when the Bazel executable isn't invoked. Under the hood, there's two pieces to make this work. The first is a binary uses `declare_symlink()` to write a relative-path based symlink that points to the underlying Python interpreter. The second piece is a site init hook (triggered by a `.pth` file using an `import` line) performs sys.path setup as part of site (`import site`) initialization. Fixes #2169 --------- Co-authored-by: Ignas Anikevicius <[email protected]>
|
#2409 might have broken bootstrap=script for windows using wsl @ewianda reported on the PR:
Unfortunately, I don't have a windows machine for development. Last time I tried to set one up for bazel dev it took days and I never got it actually working. Our CI doesn't cover windows+wsl. It might take me a bit of time to repro this and fix it. Elvis, are you able to determine what the correct relative path should be? It kinda looks like there should 4, not 5, Building with |
|
#2409 also seems to have broken Trivial reproduction:
I get: Where the symlink points to: I haven't had the time to look into the venv implementation yet so I'm not sure what the problem is.
|
|
Sorry @rickeylev I was quick to say that it happens only on Ubuntu WSL, it also happens on regular linux VM on gcp |
|
changing to I assume there is no automated test for |
Windows + wsl2 is pretty mature. It is indistinguishable from native Linux installation. So need need to cover this case in CI. |
|
Ok - so I don't quite understand the original logic - but I think the correct logic for this: rules_python/python/private/py_executable_bazel.bzl Lines 370 to 374 in 4ee7e25 is supposed to be: EDIT: ignore the below - i'm wrong and stupid interpreter = ctx.actions.declare_symlink("{}/bin/{}".format(venv, py_exe_basename))
interpreter_actual_path = runtime.interpreter.short_path # Always relative to .runfiles/_main
escapes = 2 # To escape out of <target>.venv/bin
escapes += executable.short_path.count("/") # To escape into .runfiles/_main
parent = "/".join([".."] * escapes)
rel_path = parent + "/" + interpreter_actual_path
ctx.actions.symlink(output = interpreter, target_path = rel_path)( Let me raise a PR in a moment so we can see this better. |
If read this correctly it translates to |
|
They're not equivalent, no - can you try these changes out please? |
|
Thanks @chowder , bazel test is now passing, |
|
OHH okay, I think see how this is supposed to work now. @ewianda I've updated the PR The only case it doesn't cover is an external rule using a Python interpreter stored in the main repository (but I reckon that's a bit of an unusual setup). |
…`--bootstrap_impl=script` (#2439) Computing the relative path from the venv interpreter to the underlying interpreter was incorrectly using the actual interpreter's directory depth, not the venv interpreter's directory depth, when computing the distance from the venv interpreter to the runfiles root. The net effect is the correct relative path would only be computed for binaries with the same directory depth as the actual interpreter (e.g. 2). This went undetected in CI because the tests for this logic just happen to have the same directory depth as the actual interpreter used. To fix, compute the relative path to the runfiles root using the venv interpreter directory. Also added a test in a more nested directory to test this case. Along the way: * Change relative path computation to compute a minimum relative path. * Fix the internals to pass a runfiles-root relative path, not main-repo relative path, for the actual interpreter, as intended. Fixes #2169 --------- Co-authored-by: Richard Levasseur <[email protected]> Co-authored-by: Richard Levasseur <[email protected]> Co-authored-by: Ignas Anikevicius <[email protected]>
A side-effect of no longer propagating import paths using the
PYTHONPATHenvvar is that subprocesses don't inherit the paths. This is usually a good thing, but ends up breaking plain calls to python that assume they're going to inherit the current python's settings.An example is pre-commit and its invocation of virtual env:
Eventually, it'll run:
[sys.executable, '-mvirtualenv', ...], its sys.path will be just the stdlib, and fail to importvirtualenvThis is sort of WAI. Part of the purpose of bootstrap_impl=script is to no longer use the envvar so that PYTHONPATH doesn't get too long and bleed into subprocesses.
I'm not sure how to work around this. I guess a legacy option to set the env var?
I'm not sure how this is supposed to work outside of bazel, either. It must assume that it's invoked in a venv or something? The surrounding code seems to indicate it's setting up a venv for pre-commit itself...or something. This all seems odd -- I would have to create a venv with virtualenv in it to run pre-commit so pre-commit can create its own venv? That doesn't sound right.
The text was updated successfully, but these errors were encountered: