Possible causes of CUDA get device properties error with Python3 / Theano?

Question

I'm trying to use multiple GPUs with multiprocessing in Python3. I can run a simple test case, like the following:

import theano
import theano.tensor as T
import multiprocessing as mp
import time
# import lasagne

def target():
    import theano.sandbox.cuda
    print("target about to use")
    theano.sandbox.cuda.use('gpu1')
    print("target is using")
    import lasagne
    time.sleep(15)
    print("target is exiting")

x = T.scalar('x', dtype='float32')

p = mp.Process(target=target)

p.start()

time.sleep(1)
import theano.sandbox.cuda
print("master about to use")
theano.sandbox.cuda.use('gpu0')
print("master is using")
import lasagne
time.sleep(4)
print("master will join")

p.join()
print("master is exiting")

When I run this, I get the master and the spawned process each using a GPU successfully:

>> target about to use
>> master about to use
>> Using gpu device 1: GeForce GTX 1080 (CNMeM is enabled with initial size: 50.0% of memory, cuDNN 5105)
>> target is using
>> Using gpu device 0: GeForce GTX 1080 (CNMeM is enabled with initial size: 50.0% of memory, cuDNN 5105)
>> master is using
>> master will join
>> target is exiting
>> master is exiting

But in a more complex code-base, when I try to set up the same scheme, the spawned worker fails with:

ERROR (theano.sandbox.cuda): ERROR: Not using GPU. Initialisation of device 1 failed:
Unable to get properties of gpu 1: initialization error
ERROR (theano.sandbox.cuda): ERROR: Not using GPU. Initialisation of device gpu failed:
Not able to select available GPU from 2 cards (initialization error).

And I'm having a hard time chasing down what's causing this. In the code snippet above, the problem is recreated if lasagne is imported at the top, before forking. But I've managed to prevent my code from importing lasagne until after forking and trying to use a GPU (I checked sys.modules.keys()), and still the problem persists. I don't see anything Theano related except for theano itself and theano.tensor being imported before forking, but in the example above that's fine.

Has anyone else chased down anything similar?

May be the 'bleeding edge' part could be the problem. Did you try with the stable version? — Ébe Isaac, Nov 18 '16 at 09:53

score 0 · Answer 1 · answered Nov 18 '16 at 04:48

0

I've been through a similar problem before when trying to configure Theano with Python3 in a Windows PC with the GTX-980. It worked fine with the CPU, but it just doesn't use the GPU.

After which, I tried configuring it with Python2/Theano, and the problem was resolved. I suppose it could be something wrong with the CUDA version. You could give Python2/Theano a try (with a virtual environment if needed).

answered Nov 18 '16 at 04:48

Ébe Isaac

11,563
17
64
97

Good to know I'm not alone! We just migrated everything to Python3, and there's no going back, but I like your suggestion to set up virtual envs and try different versions. I feel like, since the example snippet works, there must be something else getting imported or configured that is causing the problem, and I just need to find it and put it off until after forking processes? – Adam S. Nov 18 '16 at 07:17

score 0 · Accepted Answer · answered Nov 23 '16 at 01:28

OK this turned out to be very simple... I had a stray import theano.sandbox.cuda in a pre-fork location, but this needs to happen only after forking. It was still necessary to also move lasagne imports to after the fork, in case that helps anyone else.

(In my case, I actually need information from lasagne-based code before the fork, so I have to spawn a throw-away process which loads that and gives the relevant values back to the master thread. The master can then build shared objects accordingly, fork, and subsequently each process builds its own lasagne-based objects which work on its own GPU.)

Possible causes of CUDA get device properties error with Python3 / Theano?

2 Answers2