0

I am trying to train a model in Keras with the Theano backend and use my GPU. If I run with 100 images or 20K I get an error from Theano(look below) saying I do not have enough memory but I have 6GB on my GPU. I have used the THEANO_FLAG from Keras docs "THEANO_FLAGS=device=gpu,floatX=float32 python my_keras_script.py" the problem I used the this one from stackoverflow (echo -e "\n[global]\nfloatX=float32\ndevice=gpu0\n[lib]\ncnmem=0\n" >> ~/.theanorc ) to set the cnmem variable so if you use the flags from Keras it will use what cnmem you set with the stack overflow one.

I have set cnmem to 0.83 (the highest it goes without erroring out right away) and to 0 and nothing has the 822 MB's that it requires but I have 6GB of video memory. I am sure I am doing something simple wrong but there isn't any information I can find on it.

I am on Ubuntu 14.04 and I have CUDA installed. And I just did the Keras MNIST example using "THEANO_FLAGS=device=gpu,floatX=float32 python mnist_transfer_cnn.py "

MemoryError: Error allocating 822083584 bytes of device memory (out of  memory).
Apply node that caused the error: GpuElemwise{add,no_inplace} (GpuDnnConv{algo='small', inplace=True}.0, GpuReshape{4}.0)
Toposort index: 375
Inputs types: [CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, (True, False, True, True))]
Inputs shapes: [(64, 64, 224, 224), (1, 64, 1, 1)]
Inputs strides: [(3211264, 50176, 224, 1), (0, 1, 0, 0)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[GpuElemwise{Composite{(i0 * (i1 + Abs(i1)))},no_inplace}(CudaNdarrayConstant{[[[[ 0.5]]]]}, GpuElemwise{add,no_inplace}.0), GpuElemwise{Composite{((i0 * i1) + (i0 * i1 * sgn(i2)))}}[(0, 1)](CudaNdarrayConstant{[[[[ 0.5]]]]}, GpuDnnPoolGrad{mode='max'}.0, GpuElemwise{add,no_inplace}.0)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
Community
  • 1
  • 1
user18101
  • 626
  • 3
  • 13
  • 22
  • Are you sure that you are using the gpu with 6GB memory? There may be multiple gpu available and theano chooses the wrong one. – sytrus Jul 27 '16 at 14:17
  • I only have one gpu, I think the pre trained model (VGG-16) was taking up almost all my memory and that is why I could only use very small batch sizes. – user18101 Jul 27 '16 at 14:19
  • How small batch sizes? I can use vgg-16 with batch size 19 on a 4GB memory gpu. – ginge Jan 11 '17 at 16:32

0 Answers0