[Bug]: Some packages rely on site-packages layout

[alexeagle]

What happened?

rules_py doesn't really create a standard pip install-style site-packages folder currently, rather it creates a .pth file and expects packages to "do the right thing".

This plays badly with loading shared objects in particular, such as

Error: /main.runfiles/pip_deps_nvidia_cudnn_cu11/site-packages/nvidia/cublas/lib/libcublas.so.11: cannot open shared object file: No such file or directory
# But that file libcublass.so.11 exists in a different external package:
./main.runfiles/pip_deps_nvidia_cublas_cu11/site-packages/nvidia/cublas/lib/libcublas.so.11

Version

HEAD

How to reproduce

No response

Any other information?

No response

[siddharthab]

I gave some thought to it, and I think making a symlink forest inside the venv will be the right way to do this.

[alexeagle]

Yes, and it's very similar to what @thesayyn has been working on in rules_oci where those symlinks point outside of that folder to other entries in bazel-out - we'll get his help to fix this.

[siddharthab]

I threw together something in a PR because my team needed it sooner. Please feel free to reject it.

[alexeagle]

@mattem could you take a look at that PR?