theanorc for multiple GPUs

Question

I have a aws machine with 4 GPUs:

00:03.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)
00:04.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)
00:05.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)
00:06.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)

and my theanorc file looks like this:

[global]
floatX = float32
device = gpu0

[lib]
cnmem = 1

When I open one jupyter notebook and import theano I get the following (which I assume is only using one GPU):

Using Theano backend.
Using gpu device 0: GRID K520 (CNMeM is enabled with initial size: 95.0% of memory, cuDNN 5105)
/home/sabeywardana/anaconda3/lib/python3.5/site-packages/theano/sandbox/cuda/__init__.py:600: UserWarning: Your cuDNN version is more recent than the one Theano officially supports. If you see any problems, try updating Theano or downgrading cuDNN to version 5.

However, if I open a second jupyter notebook on the same machine at the same time. Then I get the error:

ERROR (theano.sandbox.cuda): ERROR: Not using GPU. Initialisation of device 0 failed:
initCnmem: cnmemInit call failed! Reason=CNMEM_STATUS_OUT_OF_MEMORY. numdev=1

ERROR (theano.sandbox.cuda): ERROR: Not using GPU. Initialisation of device gpu failed:
initCnmem: cnmemInit call failed! Reason=CNMEM_STATUS_OUT_OF_MEMORY. numdev=1

If I manually change my .theanorc to use gpu1 then the second jupyter notebook works fine. So the question is: Is there a way to configure .theanorc to just get the available GPU?

Not sure I am. Though trying. Try to replace gpu0 with gpu only. Let me know if it works. — Avijit Dasgupta, Nov 25 '16 at 11:11

score 2 · Accepted Answer · answered Nov 25 '16 at 16:03

You can use device=gpu, which will select the first available GPU. However, in your case, GPU 0 will still be considered "available" (it does not have much memory left, but execution is still possible). You can use nvidia-smi to set the compute mode of your GPUs to "Exclusive Thread", so that the first notebook "blocks" the first GPU for its exclusive use, and the second notebook will use another one.

Another option is to change the THEANO_FLAGS environment variable from inside the notebook, before importing theano. Something like:

import os

os.environ['THEANO_FLAGS'] = os.environ.get('THEANO_FLAGS', '') + ',' + 'device=gpu1'

import theano

I think instead of changing the env variable from inside notebook, its better that he can start the notebook by doing $THEANO_FLAGS=device=gpuN jupyter notebook. — Sentient07, Nov 25 '16 at 20:33

Avijit Dasgupta · Answer 2 · 2016-11-25T13:07:05.160

It is not possible to change the gpu device after importing theano.

May be you can try this-

import os
os.system("THEANO_FLAGS='device=gpu0' python script_1.py")
os.system("THEANO_FLAGS='device=gpu1' python script_2.py")
os.system("THEANO_FLAGS='device=gpu1' python script_3.py")
os.system("THEANO_FLAGS='device=gpu1' python script_4.py")

If you want to do it from inside the notebook (more programmatical) you may use following snippet :-

import theano.sandbox.cuda
theano.sandbox.cuda.use("gpu0")

Paste this to every notebook and change the gpu id. It will work.

theanorc for multiple GPUs

2 Answers2