nvidia-smi executable file not found

Question

I have went through 3 different issues in the nvidia-docker repo about this exact problem but actually couldn't figure out what's wrong.

I'm a heavy docker user but I don't understand much of the terminology and solution used in those issues.

When I run nvidia-smi as sudo or not, everything works great and I get the standard output. My nvidia-docker-plugin is up and running, and I get these messages when I run nvidia-docker run --rm nvidia/cuda nvidia-smi:

nvidia-docker-plugin | 2017/11/04 09:14:18 Received mount request for volume 'nvidia_driver_387.22' Blockquote nvidia-docker-plugin | 2017/11/04 09:14:18 Received unmount request for volume 'nvidia_driver_387.22'

I also tried to run the deepo repository, can't get it to work as all my containers exit upon starting, and the nvidia-docker run --rm nvidia/cuda nvidia-smi outputs the error:

container_linux.go:247: starting container process caused "exec: \"nvidia-smi\": executable file not found in $PATH" /usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"nvidia-smi\": executable file not found in $PATH".

What am I doing wrong?

I run Fedora 26, if it makes any difference

Answer 1

On Ubuntu, you should install nvidia-modprobe package. I understand that also exists in Fedora. For some reason, this dependency isn't required either documented.

Answer 2

I've just solved this.

Removing the volume related to nvidia-docker-plugin solved the issue.

For future readers, just read out the log messages on your nvidia-docker-plugin, look for the mount/unmount logged lines, and use the following command to remove the volume

docker volume rm -f <volume_to_remove> where volume_to_remove should be something like nvidia_driver_387.22 (which matched my case)

Seems like the issue is that the mapping to the nvidia-smi call is made upon the volume creation and removing and reattaching the volume fixes this