python: remove LD_LIBRARY_PATH hack from running python environment
#1562
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vlaci
added a commit
to onekey-sec/unblob
that referenced
this pull request
Oct 30, 2024
Flake lock file updates:
• Updated input 'devenv':
'github:vlaci/devenv/5186fecbe2835ac45018ea4940388f8523bf1624' (2024-10-04)
→ 'github:vlaci/devenv/953a526754c5ff5e64ab6925c4388b26ae2c45c6' (2024-10-30)
From the related PR[^1] in `devenv`:
Injecting `LD_LIBRARY_PATH` to the Python runtime environment is great
to bypass the need of having to patch non-nix binaries loaded into
that environment, however it breaks down, when Python executes any
other program not compiled for the given Nix system, e.g. a shell
script via `subprocess`.
To work this around, `devenv` will inject a `pth`[^2] file to the virtual
environment it creates, which mangles the `LD_LIBRARY_PATH` variable,
undoing any changes to it made by `devenv` but preserving changes from
other sources.
[^1]: cachix/devenv#1562
[^2]: https://docs.python.org/3/library/site.html
vlaci
added a commit
to onekey-sec/unblob
that referenced
this pull request
Oct 30, 2024
Flake lock file updates:
• Updated input 'devenv':
'github:vlaci/devenv/5186fecbe2835ac45018ea4940388f8523bf1624' (2024-10-04)
→ 'github:vlaci/devenv/385d38cfb6872e7f95cae30d94f7fc4afd23fd76' (2024-10-30)
From the related PR[^1] in `devenv`:
Injecting `LD_LIBRARY_PATH` to the Python runtime environment is great
to bypass the need of having to patch non-nix binaries loaded into
that environment, however it breaks down, when Python executes any
other program not compiled for the given Nix system, e.g. a shell
script via `subprocess`.
To work this around, `devenv` will inject a `pth`[^2] file to the virtual
environment it creates, which mangles the `LD_LIBRARY_PATH` variable,
undoing any changes to it made by `devenv` but preserving changes from
other sources.
[^1]: cachix/devenv#1562
[^2]: https://docs.python.org/3/library/site.html
src/modules/languages/python.nix
Outdated
| pth_file = | ||
| let | ||
| exec_content = '' | ||
| ld_library_path_var = "DYLD_LIBRARY_PATH" if sys.platform == "darwin" else "LD_LIBRARY_PATH" |
There was a problem hiding this comment.
I fairly sure DYLD_LIBRARY_PATH doesn't propagate to subprocesses. So it is probably not needed to support it here.
Maybe the pth file only needs to be added on Linux machines?
There was a problem hiding this comment.
I have no idea about it. I am okay to remove it
| os.environ[ld_library_path_var] = ld_library_path.removeprefix(ld_library_path_prefix + ":") | ||
| ''; | ||
| in | ||
| pkgs.writeText "devenv.pth" ''import os; exec("""${builtins.replaceStrings [ "\n" ] [ "\\n" ] exec_content}""")''; |
There was a problem hiding this comment.
Does this need exec? I thought it might be possible to be executed in pth as-is.
There was a problem hiding this comment.
Pth is executed line by line, so this is the only way to use multi-line constructs like if conditions
|
I was thinking, since this will probably be needed just for Linux, that it might be nice to eventually have an LD_PRELOAD that removed the path from LD_LIBRARY_PATH. That way this exact same trick can be applied to ruby, nodejs and more. Still useful to have this pth to see how it fares in the python world. |
Injecting `LD_LIBRARY_PATH` to the Python runtime environment is great to bypass the need of having to patch non-nix binaries loaded into that environment, however it breaks down, when Python executes any other program not compiled for the given Nix system, e.g. a shell script via `subprocess`. To work this around, `devenv` will inject a `pth`[^1] file to the virtual environment it creates, which mangles the `LD_LIBRARY_PATH` variable, undoing any changes to it made by `devenv` but preserving changes from other sources. Fixes cachix#1111 [^1]: https://docs.python.org/3/library/site.html
|
Can we get some tests for this? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Injecting
LD_LIBRARY_PATHto the Python runtime environment is greatto bypass the need of having to patch non-nix binaries loaded into
that environment, however it breaks down, when Python executes any
other program not compiled for the given Nix system, e.g. a shell
script via
subprocess.To work this around,
devenvwill inject apth1 file to the virtualenvironment it creates, which mangles the
LD_LIBRARY_PATHvariable,undoing any changes to it made by
devenvbut preserving changes fromother sources.
I am unsure what the best way would be to integrate this, bit I think the approach in general is sane. I am tested only the poetry case currently.
Fixes #1111
Footnotes
https://docs.python.org/3/library/site.html ↩