In the past, I've successfully configured Ubuntu systems containing NVIDIA GPUs to launch nvidia-smi in the background from within /etc/rc.local during the system boot to create the requisite /dev/nvidia* files. After recently trying out NVIDIA's 346.46 drivers on Ubuntu 14.04 with CUDA 6.5-19 and Linux kernel 3.13.0-48, I noticed that nvidia-smi would hang indefinitely (and never create any /dev/nvidia* files) during bootup. I was able to work around this by inserting a delay (i.e., sleep 30) in /etc/rc.local immediately before launching nvidia-smi. Is there some way to check whether the GPUs are somehow "accessible" (for lack of a better word) before starting nvidia-smi?
I should add that the problem only seems to affect Tesla GPUs (Fermi generation); I haven't had to add any delays to /etc/rc.local on systems with GeForce (Fermi) GPUs.
