How should I be setting up my environment?

[gordonwatts]

My code requires a fair number of dependencies. So each time a container is created, I have to do a pip install -e .[test], basically. And when the container is torn down, everything is removed. While not impossible, this is a bit of a pain for two reasons:

• My environment is not stable. Each time I do pip install, I get whatever is latest on pypi.
• On Chicago's AF my coffee-casa's containers are restarted with some frequency (I do not yet understand the cadence). Each time I have to go through this setup process.
• You can't work on more than one project, unless the set of python packages are compatible.

Python virtual environments were created to solve this problem. And the containers user space is persistent (thank goodness). However, the jupyter server only starts in the global context. So a virtual env is only useful if you are doing everything by command line.

I'm wondering how others shape their workflow?

[robrwg]

Fengping has a development ver of coffea which uses your local unix id (and persistent home directory). I don't know if that's going to solve your problem, however, but maybe a step in that direction.

[gordonwatts]

This is certainly a help - especially accessing our large data disks. But I think the thing this misses is the environment that the Jupyter server is running in.

[fengpinghu]

You can maintain a python venv in your home directory and add that as a python kernel. I tested the following and it works:

1. Activate your venv -- $ source your-venv/bin/activate
2. Install jupyter in the virtualenv -- pip install jupyter
3. Add the virtualenv as a jupyter kernel -- ipython kernel install --name "local-venv" --user
4. Find the kernelspec -- jupyter kernelspec list
5. Copy kernelspec to the conda dir -- cp -r $pathyour find on step 4 ( kernels/*) /opt/conda/share/jupyter/kernels/
6. Select new notebook and you should see your own kernel in the list of kernels you can choose from.

[gordonwatts]

Oh, that is wonderful, thank you for that! I'll try that out!

[matthewfeickert]

@fengpinghu's instructions got my code that was used for the statistical inference with pyhf and cabinetry examples at the IRIS-HEP Analysis Grand Challenge Tools 2022 Workshop working in a Jupyter notebook. 👍 Thanks!

Though it is also worth remembering that this is only affecting the notebook kerenl and as soon as you escape out to the shell you're not in your virtual environment anymore as seen in this screenshot

So this is different behavior form Binder (where you only get 1 environment: The one that you built). That shouldn't be too surprising, and isn't a problem is all you're doing is basic Linux CLI operations, but this can catch people off guard. The following example of black isn't a super good one, but you could imagine that there is a library you've installed in your virtual environment that has CLI API behavior that you wanted to use in the shell from inside your notebook using IPython magics

This isn't meant as criticism, as I don't think there's much you can do to improve the situation, but I think given that this requires 5 steps just to get a kernel working that most people will view that as too much of a hassle and just treat the pod's environment as a short lived one to install directly into without any virtual environment. To be clear though, I'm still happy to have this workflow for the time being. 👍