0

I am testing an FCN-VGG16 network in Caffe. Instead of resizing the input images to a fixed size, I reshape the network each time to the image size. I tried this in both matcaffe and pycaffe. In both cases, it seems that it is able to run for small images (for example, 500x500). However, when I have a larger sized image (e.g. 1920 x 1080), I get this error "Check failed: error == cudaSuccess (2 vs. 0) out of memory". I am running this on a Quadro M6000 GPU with 12 GB memory. Any help/advice would be appreciated.

user6137678
  • 3
  • 2

1 Answers1

1

I know it's obvious from the message, but you do have to reduce the input size: the GPU doesn't have enough on-board memory for everything you fed it. Can you reduce your batch size? If not, do you really need the entire 2M pixels? Most models do quite well with cropped or reduced images. Do this as part of the input processing (on the CPU), and parametrize it for any sine input.

Resizing the network is expensive in memory, especially if you resize all layers, scaling with the input dimensions. In the case you gave, this will increase your memory requirements by roughly 8x.

As a check on the size you can have, watch the memory figures as you initialize the network; I believe that Caffe reports memory requirements by layer.

Prune
  • 76,765
  • 14
  • 60
  • 81
  • Thank you for the quick response. To answer your questions, I am using a batch size of 1, and I am reshaping the network using the following command: net.blobs['data'].reshape(1, *im.shape). I actually have tried resizing the images to a fixed size, and then doing the testing. However, I am not satisfied with the results, and I am wondering if it has to do with the resizing. Because I have a wide range of input sizes, I am concerned that resizing to a fixed size is affecting the performance (e.g. for some images, the aspect ratio is being significantly changed). – user6137678 Jun 01 '17 at 23:42
  • Since I have a wide range of input sizes, perhaps I should not resize all the images to a fixed size. Instead, I could resize each image to a smaller size, but preserving the aspect ratio. However, in this case, the input images would still be of variable size, so I would still have to reshape the network. – user6137678 Jun 01 '17 at 23:48
  • Right. You can reshape the network, as long as you keep the footprint within "reasonable" range -- with "reasonable" determined by your RAM size. – Prune Jun 01 '17 at 23:49
  • Thank you. I will give it a try. – user6137678 Jun 01 '17 at 23:50