[Tracking] Handling multiple versions of CUDA libraries in the same closure

[ConnorBaker]

Describe the bug

When multiple versions of CUDA libraries are in the same closure, the order they are discovered and loaded by different packages is unclear.

This is issue is meant to track progress toward documentation, tools, and tests to communicate and ensure that libraries are loaded in a predictable and deterministic fashion.

As an example (courtesy of @SomeoneSerge), we should have tests which ensure both of the following work:

# test1
import torch
torch.randn(10, 10, device="cuda").sum().item()
import cv2
# do something with cv2 and cuda

# test2
import cv2
# do something with cv2 and cuda
import torch
torch.randn(10, 10, device="cuda").sum().item()

Related Links

PRs

• testers.testLibs: init #341471

Issues

• opencv: misc cleanups; fix CUDA build #339619

Notify maintainers

@NixOS/cuda-maintainers

---

Add a 👍 reaction to issues you find important.

[samuela]

This seems like a Hard Problem, and I'm not aware of any package managers that actually solve it. Is there a need to fix this?

[Princemachiavelli]

When I need to manually debug these kinds of issues, I use LD_DEBUG=libs but it's output isn't easy to parse. You might find using the LD_AUDIT interface easier and more informative. Unfortunately the only user space tool that implements it, latrace, appears to be unmaintained and not functional. Luckily the LD_AUDIT interface is only a few functions.

Is the thought that CUDA libraries are loading in a non-predictable fashion due to the use of dlopen or some source of non-determinism effecting the dynamic linker /lib/ld-linux.so.2?

[samuela]

Is the thought that CUDA libraries are loading in a non-predictable fashion due to the use of dlopen or some source of non-determinism effecting the dynamic linker /lib/ld-linux.so.2?

That's my understanding. Other uses should be explicit in RUNPATH/RPATH.

[SomeoneSerge]

You might find using the LD_AUDIT interface easier and more informative

Oh! We were thinking of parsing LD_DEBUG but you're making a good point! FWIW there are more examples of using LD_AUDIT, e.g. in guix pack, flox, and now cachix/devenv#773 (comment)

[SomeoneSerge]

However, note that for this particular test we don't even have to interact with ld.so in any particular way, we just verify that the program does not crash when using two potentially conflicting modules