GPU out of memory when neural network is called from python function

Question

I asked a similar question very recently: I want to use a caffe convolutional neural network in python for pixel-wise classification on an image sequence. Previously, I determined the maximum image size that I can process with my gpu (450x450 pixels). To process larger images, I want a nested loop like so:

function process_image_stack(image_stack, parameters):
    for all images
       take image apart into segments
       for all image_segments
           process image segment with network
           stitch together output
return processed_image_stack

My problem now is the following: When I call net.forward() from a function like so:

output = processing_function(parameters)

CUDA throws an error that it is out of memory

Check failed: error == cudaSuccess (2 vs. 0) out of memory

I also get this error when I use a much smaller image as input

The problem seems to be the loop. When I "manually" loop through the images, i.e. setting variables to 0,1,2... and then calling the net on the individual ones, it works fine.

I would like to understand what the problem actually is.

Thank you very much!

The image stack has images of the same size. When the individual images are cut into smaller segments, these segments may have unequal size. I accommodate this by using net.blobs['data'].reshape — warped, Nov 03 '16 at 12:45
The only way to do that was to delete the net after processing the images. This was suggested to me in the previous question I asked. I do not know how to do free the memory without deleting the net. — warped, Nov 03 '16 at 13:03
"What about using a command del your_net to free the net object in python?" I commented there in the previous question. — Dale, Nov 04 '16 at 00:31

GPU out of memory when neural network is called from python function

0 Answers0