writers.writePython3: created script fails to run on MacOS

[adrian-gierakowski]

Describe the bug
Executables produced by writers.writePython3 (and probably also writePython2 etc) fail to run on MacOS

To Reproduce
Steps to reproduce the behavior:

1. clone https://github.com/adrian-gierakowski/nix-writePython3-broken-on-darwin
2. run $(nix-build) from top level the repo (assuming POSIX shell)
3. see it fail

Expected behavior
The command from step 2 should print success

Additional context

This seems to be caused by the fact that ${python3.withPackages}/bin/python which is used for shebang line
in the produced script is itself a "script text executable" which does not seem to work on MacOS.

The repo mentioned above contains a tentative fix, which prepend ${pkgs.bash}/bin/bash to the shebang line in the produced script.

NOTE: the repo above uses my own fork of nixpkgs since dash is currently broken on darwin (and writePython3 depends on dash)

Notify maintainers
@Lassulus

Metadata

• system: "x86_64-darwin"
• host os: Darwin 18.7.0, macOS 10.14.6
• multi-user?: no
• sandbox: yes
• version: nix-env (Nix) 2.3.6pre7027_6f162d8
• channels(adriangierakowski): ""
• nixpkgs: /nix/store/ciqd3l5bznq5ms389b4w7yjv7vk9vpm0-source

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
- writers.writePython2
- writers.writePython3
- writers.writePython2Bin
- writers.writePython3Bin

[Lassulus]

that seems quite sad. Sadly I'm not able to reproduce it since I have to working MacOS installation. Did this work in the past for you? Or is it generally broken?
I guess there is also no harm in switching from a shebang to a shell wrapper script, I'm just not sure how to test it :D

[veprbl]

Related to #65351 and #11133

[adrian-gierakowski]

thanks @veprbl

@Lassulus as described in the issues liked above this is generally broken since writePython3 uses ${python3.withPackages}/bin/python. which is a bash script, as interpreter which does not work on darwin (and BSD in general).

I believe a good way to fix it is to use "${self.bash}/bin/bash ${py}/bin/python" as interpreter (but only if sdenv.isDarwin == true). This should not introduce any extra run time dependencies since bash is already a dependency of python3.withPackages.

I'm happy to submit a PR

[adrian-gierakowski]

additionally when libraries == [] we could simply use "${python3}/bin/python" as interpreter which would avoid the dependency on bash

[adrian-gierakowski]

this is the fix I'd go for:

{
  interpreter =
    if libraries == []
    then "${super.python3}/bin/python"
    else if super.stdenv.isDarwin
      then "${self.bash}/bin/bash ${py}/bin/python"
      else "${py}/bin/python"
  ;
}

[bergkvist]

This is still an issue for me. How to reproduce:

1. Create shell.nix

let
  pkgs = import <nixpkgs> {};
  python39 = pkgs.python39.withPackages(ps: [ ps.pip ]);
in pkgs.mkShell {
  buildInputs = [ python39 ];
  shellHook = ''
    export PIP_PREFIX="$(pwd)/.build/pip"
    export PYTHONPATH="$(pwd):$(pwd)/.build/pip/lib/python3.9/site-packages:$PYTHONPATH"
    export PATH="$(pwd)/.build/pip/bin:$PATH"
  '';
}

2. Try to install and run ipython in the nix-shell (or any other executable python library)

[nix-shell:~/my-project]$ pip install ipython

# Now, after installing ipython, let's try to execute it
[nix-shell:~/my-project]$ ipython

# /Users/tobias/my-project/.build/pip/bin/ipython: line 3: import: command not found
# /Users/tobias/my-project/.build/pip/bin/ipython: line 4: import: command not found
# from: can't read /var/mail/IPython
# /Users/tobias/my-project/.build/pip/bin/ipython: line 7: syntax error near unexpected token `('
# /Users/tobias/my-project/.build/pip/bin/ipython: line 7: `    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])'

Looking inside .build/pip/bin/ipython, I see this:

#!/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

[veprbl]

@bergkvist Please file a new issue.