Issue with deployment of pip installed python packages with command line tools

[rbeucher]

The med-utils package which provides the ilamb-tree-generator command line tool is currently installed using pip in our access-med environment:

- pip:
  - railroad-diagrams ### Unlisted dependency of pip and pyparsing
  - git+https://github.com/ACCESS-NRI/MED-utils

The problem is that during install/deploy a shebang is created:

#!/jobfs/109596945.gadi-pbs/squashfs-root/opt/conda/access-med-0.7/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from access_med_utils.ilamb import tree_generator
if __name__ == '__main__':
   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
   sys.exit(tree_generator())

This leads to the following error:

bash: /opt/conda/access-med-0.7/bin/ilamb-tree-generator: /jobfs/109596945.gadi-pbs/squashfs-root/opt/conda/access-med-0.7/bin/python: bad interpreter: No such file or directory

@rhaegar325 I think that's the cause of the issue but I could be wrong.
@dsroberts do you have a suggestion on how to fix this?

[dsroberts]

Interesting. The --relocate-prefix argument in this line is supposed to handle that. I guess it doesn't work for pip packages installed that way? I notice you have MED-utils on your conda channel, why not install from there?

[rbeucher]

Yes that is what I thought we should do.

[dsroberts]

I had other ideas in case --relocate-prefix didn't work out. My first thought was to maintain a fork of micromamba that doesn't resolve path symlinks, or to run sed on every script in the bin directory. Neither of these ideas are good, so but if --relocate-prefix doesn't work one of those might be the only option without reverting back to mamba and a base environment.

[rbeucher]

Hi @dsroberts ,

I am having a similar problem.
I have tried to deploy a new version of our ESMValTool environment

module use /g/data/xp65/public/modules
module load conda/esmvaltool-0.4

Build and deployment went OK. I can also import the python packages from a notebook on ARE.

However, I cannot run the esmvaltool command from the CLI. I get:

esmvaltool version  
/g/data/xp65/public/apps/med_conda_scripts/esmvaltool-0.4.d/bin/esmvaltool: /g/data/xp65/public/apps/med_conda/envs/esmvaltool-0.4/bin/esmvaltool: /jobfs/120579340.gadi-pbs/squashfs-root/opt/conda/esmvaltool-0.4/bin/python: bad interpreter: No such file or directory
/g/data/xp65/public/apps/med_conda_scripts/esmvaltool-0.4.d/bin/esmvaltool: line 121: /g/data/xp65/public/apps/med_conda/envs/esmvaltool-0.4/bin/esmvaltool: Success

I was initially installing ESMValTool with pip in my environment file, I moved it to get ESMValTool from conda-forge but I still have the same issue.
See environment conda/esmvatool-0.4b which returns the same error.

I think the MED-condaenv is in sync with your branch. I don't understand what is going on.
Would you be able to help?

Thanks!

[rbeucher]

I have added the logs in /scratch/hh5/rb5533 if that helps