[ci] CUDA CI jobs are broken: "driver/library version mismatch"

[jameslamb]

Description

The CUDA CI jobs for this project are all failing, with the following error.

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown.
Error: Process completed with exit code 125.

Reproducible example

• build from [tests][dask] fix workers without data test (fixes #5537) #5544 (build link)
• build from [python-package] prefix several internal functions with _ #5545 (build link)

References

Here is the line where these jobs are failing.

LightGBM/.github/workflows/cuda.yml

Line 109 in 0c0eb2a

docker run --env-file docker.env -v "$GITHUB_WORKSPACE":"$ROOT_DOCKER_FOLDER" --rm --gpus all "$docker_img" /bin/bash $ROOT_DOCKER_FOLDER/docker-script.sh

The references at https://stackoverflow.com/questions/43022843/nvidia-nvml-driver-library-version-mismatch suggest that this issue could be resolved by rebooting.

[jameslamb]

@shiyu1994 since you are the only person with administrative access to the machine the CUDA jobs runs on, can you please try rebooting that machine and investigating other fixes for this?

I'm happy to help do other research however I can, but you are the only person who can reboot the machine.

[jameslamb]

I just re-triggered a CUDA job...this is still broken.

https://github.com/microsoft/LightGBM/actions/runs/3281397887/jobs/5452120895

@shiyu1994 is there any way I can help you resolve this?

[jameslamb]

I just triggered another run and this is still happening.

https://github.com/microsoft/LightGBM/actions/runs/3350110051/jobs/5563225006

@shiyu1994 I really hope you're able to get to this soon. @ me any time if there's any way I can help.

[shiyu1994]

@jameslamb Sorry for the long delay.

I've fixed the virtual machine. And now the CI tests should be able to run.

[jameslamb]

Excellent, thanks @shiyu1994 ! I'll try re-running the checks from #5545 right now. If they work, I'll work on merging some of the approved PRs today.

[jameslamb]

🎉 🎉 🎉

that worked!

https://github.com/microsoft/LightGBM/actions/runs/3281397887/jobs/5609753729

Thank you so much for the help @shiyu1994 !

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.