Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[k8s] CUDA envvars don't work in ssh #2453

Closed
romilbhardwaj opened this issue Aug 24, 2023 · 0 comments · Fixed by #2500
Closed

[k8s] CUDA envvars don't work in ssh #2453

romilbhardwaj opened this issue Aug 24, 2023 · 0 comments · Fixed by #2500
Labels
Milestone

Comments

@romilbhardwaj
Copy link
Collaborator

CUDA-specific environment variables are dynamically set by the nvidia kubernetes operator and the node's kubelet when a command in a pod or kubectl exec -it ... is run. This allows the user to access the GPU in their python programs and through nvidia-smi.

Currently, these environment variables are currently not set when we run ssh mycluster. As a result, the GPU is unavailable when the user ssh's into the cluster for debugging.

This is related to #2287 and should be closed by #2435.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant