Issues with Nvidia docker container

[timmarkhuff]

I've gotten the NVIDIA docker container to work several times, but I am periodically faced with this error:

tim@tim:~/blue$ docker compose -f ./.docker/compose/nvidia-desktop.yaml up -d
[+] Building 0.0s (0/0)                                                                                            docker:desktop-linux
[+] Running 0/1
 ⠸ Container compose-blue-1  Starting                                                                                              0.3s 
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown

It seems docker is not able to find the NVIDIA container toolkit, but I don't know why because it is installed. I have resolved this issue a few times by blindly reinstalling Docker and NVIDIA container toollkit and repeated trying docker compose, but I can't figure out what is the root cause of this issue, or what the correct way to fix it is.

@evan-palmer , do you have any insight here?

[amarburg]

@timmarkhuff what base OS and platform are you testing on?

[evan-palmer]

@timmarkhuff Could you also share how you have installed Docker?

[timmarkhuff]

I am testing on Ubuntu 22.04 on a ASUS laptop with an nvidia geforce rtx gpu. I have tried installing docker two ways: 1) docker engine only (https://docs.docker.com/engine/install/ubuntu/) 2) docker desktop (https://docs.docker.com/desktop/install/ubuntu/). I get similar results with both ways of installing Docker.

[evan-palmer]

I haven't personally experienced this issue before; however, it seems related to this issue. Folks there indicate that the issue is most prominent when using Docker via the Docker Desktop installation. They also mention that the issue seems to be with Docker not getting access to the GPU and running the container with sudo seems to help

[evan-palmer]

There are a couple of other solutions documented in that thread, though it's an open issue

[timmarkhuff]

Thanks Evan. I encountered this issue before (and after) using Docker Desktop. I agree the issue does seem to be with permissions, but so far, I have been unable to put my finger on what the issue is. Simply adding sudo isn't enough. I will keep at it.

[evan-palmer]

Have you tried the devcontainer by chance? If the issue persists there then we can isolate the problem to either a container or local issue. If it does work then that would indicate that there’s an issue with the docker-compose script

[timmarkhuff]

I got it working last night without devcontainer.

If I run into the issue again, I might explore devcontainer

[evan-palmer]

Great! I'm glad to hear that. What ended up working for you?

[timmarkhuff]

I ended up doing a fresh reinstall of Docker. The odd thing is that with each system reboot, the problem re-emerges. I put together this script to quickly reinstall Docker. Obviously not ideal, but it's working for me now. Hopefully I will be able to pinpoint the problem eventually.

#!/bin/bash
# This script is a bandaid to fix the issue of docker commands not working with the nvidia-desktop container


# Uninstall Docker Packages:
# First, remove any installed Docker packages. 
# The names might vary depending on how Docker was installed (e.g., docker, docker-engine, docker.io, docker-ce, docker-ce-cli, docker-ce-rootless-extras, docker-scan-plugin):
sudo apt-get purge -y docker docker-engine docker.io docker-ce docker-ce-cli docker-ce-rootless-extras docker-scan-plugin

# Remove Docker Dependencies:
# To remove dependencies used by Docker which are no longer needed:
sudo apt-get autoremove -y

# Delete Docker Data and Configuration Files:
# This includes images, containers, volumes, and custom configuration files. 
# Be cautious with this step as it will delete all your Docker containers and images!
sudo rm -rf /var/lib/docker
sudo rm -rf /var/lib/containerd


# Additionally, remove any custom configuration files and directories you may have created
# including Docker's daemon.json file if it exists:
sudo rm -rf /etc/docker

# Reinstall Docker
sudo apt-get update -y

sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository -y "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt-get update -y
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

sudo systemctl start docker
sudo systemctl enable docker
sudo systemctl status docker

# Add the current user to the docker group so that you don't have to use sudo
sudo usermod -aG docker $USER