Python script packaged with (python3.withPackages ...) as build input not working on Darwin (shebang issue?)

[anderslundstedt]

Describe the bug
If I package a Python script with python3.withPackages ... as build input then the script will build but not run on Darwin.

To Reproduce
Steps to reproduce the behavior:

1. Use MacOS.
2. Create default.nix:

let
  pkgs = import <nixpkgs> { };
  python = pkgs.python3.withPackages (_: []);
in
  pkgs.stdenv.mkDerivation rec {
    name = "py-test";
    buildInputs = [ python ];
    unpackPhase = "true";
    installPhase = ''
      mkdir -p $out/bin
      touch $out/bin/py-test
      chmod +x $out/bin/py-test
      echo "#!${python}/bin/python" >> $out/bin/py-test
      echo "print(0)" >> $out/bin/py-test
    '';
  }

3. Build and run:

nix-build
./result/bin/py-test

4. Observe unexpected behaviour.

• In bash:

./result/bin/py-test: line 2: syntax error near unexpected token `0'
./result/bin/py-test: line 2: `print(0)'

• In zsh:

zsh: exec format error: ./result/bin/py-test

Expected behavior
Output:

0

Additional context
Everything works as expected if we replace pkgs.python3.withPackages ... with pkgs.python3. The only difference in the result is the shebang.

• ./result/bin/py-test with pkgs.python3.withPackages ...:

#!/nix/store/jhbikg6nir868g3wfypi7vppn5mbjhp5-python3-3.7.4-env/bin/python
print(0)

• ./result/bin/py-test with pkgs.python3:

#!/nix/store/8r52cdbn9g3sl9pz8vm4xcf7rhrqjqin-python3-3.7.4/bin/python
print(0)

Executing the python executable in the shebang works, as does executing py-test with the same executable, that is

/nix/store/jhbikg6nir868g3wfypi7vppn5mbjhp5-python3-3.7.4-env/bin/python result/bin/py-test

gives the expected output.

Existing Darwin shebang-related issues:

• buildPerlModule broken on Darwin due to shebang character limit #35353
• ack installs, but reports "use: command not found" #47302

However, it does not seem like a problem with the character limit in this case?

Metadata
Please run nix run nixpkgs.nix-info -c nix-info -m and paste the result.

 - system: `"x86_64-darwin"`
 - host os: `Darwin 16.7.0, macOS 10.12.6`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.2.2`
 - channels(anders): `""`
 - nixpkgs: `/Users/anders/nixpkgs`

[FRidh]

Nothing was changed in the Python infra that could have caused this so I think it's Darwin.

[jonringer]

if you used echo "#!/usr/bin/env python" >> $out/bin/py-test does it still fail?

If you're hitting the character limit, then this is one solution.

Possibly related article as to why this works on linux here https://lwn.net/Articles/779997/

[anderslundstedt]

Nothing was changed in the Python infra that could have caused this so I think it's Darwin.

Yes, everything works as expected on NixOS.

if you used echo "#!/usr/bin/env python" >> $out/bin/py-test does it still fail?

Yes, that shebang is (correctly) replaced by

#!/nix/store/jhbikg6nir868g3wfypi7vppn5mbjhp5-python3-3.7.4-env/bin/python

Possibly related article as to why this works on linux here https://lwn.net/Articles/779997/

I do not think we exceed the character limit on Linux. The shebang is 74 characters long. Neither should this exceed the character limit on Darwin?

[FRidh]

How does #!/nix/store/jhbikg6nir868g3wfypi7vppn5mbjhp5-python3-3.7.4-env/bin/python look? It should be a wrapper. Maybe it is generated incorrectly?.

[anderslundstedt]

How does #!/nix/store/jhbikg6nir868g3wfypi7vppn5mbjhp5-python3-3.7.4-env/bin/python look? It should be a wrapper. Maybe it is generated incorrectly?.

#! /nix/store/bbw1rwzgb43dda7gw9qjxbbpzb847n4z-bash-4.4-p23/bin/bash -e
export PYTHONHOME='/nix/store/jhbikg6nir868g3wfypi7vppn5mbjhp5-python3-3.7.4-env'
export PYTHONNOUSERSITE='true'
exec "/nix/store/8r52cdbn9g3sl9pz8vm4xcf7rhrqjqin-python3-3.7.4/bin/python"  "${extraFlagsArray[@]}" "$@"

[FRidh]

I think I recall the issue. On Darwin it is not possible to use a wrapper as a shebang.

[anderslundstedt]

#23018 (comment) shows that in some cases wrapper-as-a-shebang works?

Relevant discussions:

• wrapProgram & friends depend on linux kernel shebang feature #11133

• Wrapper machinery should produce a tiny compiled binary #23018

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.