Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show CUDA Driver Version in the output #164

Closed
LincolnYe opened this issue Oct 21, 2023 · 2 comments
Closed

Show CUDA Driver Version in the output #164

LincolnYe opened this issue Oct 21, 2023 · 2 comments

Comments

@LincolnYe
Copy link

LincolnYe commented Oct 21, 2023

Is your feature request related to a problem? Please describe.
Since some CUDA-based applications (like PyTorch, TensorFlow, etc.) have released new versions built on CUDA Toolkit 12.x, which require CUDA Driver 12.0+ to run, it's important to display the current CUDA driver version on each of my GPU servers, just like nvidia-smi does. This way, I can determine whether the CUDA driver is new enough.

Describe the solution you'd like
Display CUDA Driver Version along with the NVIDIA Driver Version.

Describe alternatives you've considered
None.

Additional context
The referrence nvidia-smi header:
image

@wookayin
Copy link
Owner

wookayin commented Oct 22, 2023

You are getting something wrong -- the CUDA version shown in nvidia-smi output has nothing to do with the actual CUDA runtime used in your system, it's just the latest CUDA version that the current NVIDIA Driver is compatible with.

So in your case, nvidia driver 535.113 can support CUDA 12.2 and below (as long as it's supported). But your ML environments might be using cuda 11.x.

To know the actual cuda runtime version on the current environment:

$ ldconfig -p | grep cudart
	libcudart.so.11.0 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudart.so.11.0
	libcudart.so.10.2 (libc6,x86-64) => /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2
	libcudart.so (libc6,x86-64) => /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so
	libcudart.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudart.so

Or from a specific DL framework, for instance:

print(torch.version.cuda)

@wookayin wookayin closed this as not planned Won't fix, can't repro, duplicate, stale Oct 22, 2023
@wookayin
Copy link
Owner

Discussion continued in #165

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants