Theano is creating additional threads on devices not specified in configuration

Question

So I have the following situation:

computer with 3 graphic cards
python script using Keras with Theano backend and multiple threads

I specified the device to use in .theanorc as described in the documentation.

The python script is of this form (still working on a standalone example):

import theano
from threading import Thread
...
class Test(Thread):

    def run(self):
        #calculations with Keras

test = Test()
test.start()
test.join()

Starting the script Theano uses the specified device but after some time a second python thread appears on one of the other graphic cards (and uses up resources).

The second thread seems to ignore the config as its running on the wrong GPU and isn't allocating ram as specified by the CNEM flag.

This should not be possible according to the documentation as everything that forks from the thread that started the Theano calculation should be running on the same device (ensured by importing Theano right at the beginning).

After some poking around I found out that this behavior stops when I don't run my Keras code in a separate thread.

So before I start creating Github issues I would like some pointers what's most likely:

Is this a bug in Theano?
Is this a bug in Keras?
Is this a bug in my own code?

@3. My whole project doesn't create separate Python processes (confirmed over process list) and doesn't change any Theano configuration.

Any idea what could even cause this kind of behavior?

As you don't share any code it's hard to say ... I can see that you're running a thread ... but nothing more ... — PyNico, Mar 14 '17 at 16:33
That's why I'm asking how this can even happen (I'm still working on a standalone example as I currently have no access to a computer with more than one graphic card). — Fabian N., Mar 14 '17 at 16:41
Just like that I would say it's probably your code, but Theano could have that kind of issue, for Keras it would be weird but why not. — PyNico, Mar 14 '17 at 16:44
Can you point me in a direction for what to look for? I think the only thing I do directly with Theano is to define a custom loss function for Keras. — Fabian N., Mar 14 '17 at 16:48

score 2 · Answer 1 · edited Feb 01 '18 at 13:04

Device(gpu) setting of a thread is independent with other threads in the same process. Look this for more details.

I haven't found a way to set device for current thread in Theano. I use obsoleted cuda_ndarray back end, there is no way to do this, but I don't know if there is a way to do this in gpuarray back end.

I do some workaround:

import numpy as np
import theano
from theano import Apply
from theano import tensor as T
from theano.scalar import Scalar
from theano.sandbox.cuda import GpuOp, nvcc_compiler

class SetGpu(GpuOp):
    '''
    Set device(gpu) for current thread.
    '''

    def c_compiler(self):
        return nvcc_compiler.NVCC_compiler

    def make_node(self, gpu_id):
        dummy_out = Scalar("int32")()
        return Apply(self, [gpu_id], [dummy_out])

    def __str__(self):
        return "SetGpu"

    def c_support_code_apply(self, node, nodename):
        return ""

    def c_code(self, node, nodename, inps, outs, sub):
        gpu_id, = inps
        dummy_out, = outs
        return """
        int _gpu_id = *((int*)PyArray_DATA(%(gpu_id)s));
        %(dummy_out)s = _gpu_id;
        cudaError_t err = cudaSetDevice(_gpu_id);
        if(err != cudaSuccess){
            PyErr_Format(PyExc_RuntimeError, "Cuda err:\\"%%s\\" when calling cudaSetDevice(%%d).", cudaGetErrorString(err), _gpu_id);
            return 0;
        }
    """ % locals()

def set_gpu(gpu_id):
    if not hasattr(set_gpu, "f"):
        set_gpu_op = SetGpu()
        gpu_id_var = T.iscalar()
        dummy_out = set_gpu_op(gpu_id_var)
        set_gpu.f = theano.function([gpu_id_var], [dummy_out])
    _dummy_out = set_gpu.f(gpu_id)

if __name__ == "__main__":
    def test():
        set_gpu(5)
        print "Test thread is using gpu %d." % theano.sandbox.cuda.active_device_number()
    print "Main thread is using gpu %d." % theano.sandbox.cuda.active_device_number()
    from threading import Thread
    thread = Thread(target=test)
    thread.start()
    thread.join()

So let's call this file set_gpu.py

here is what I get running it:

python set_gpu.py 
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: Tesla K80 (CNMeM is enabled with initial size: 95.0% of memory, cuDNN 5110)
Main thread is using gpu 0.
Test thread is using gpu 5.

Theano is creating additional threads on devices not specified in configuration

1 Answers1