23

By setting the bottom and the top blob to be the same we can tell Caffe to do "in-place" computation to preserve memory consumption.

Currently I know I can safely use in-place "BatchNorm", "Scale" and "ReLU" layers (please let me know if I'm wrong). While it seems to have some issues for other layers (this issue seems to be an example).

When to use in-place layers in Caffe?
How does it work with back-propagation?

Shai
  • 111,146
  • 38
  • 238
  • 371
dontloo
  • 10,067
  • 4
  • 29
  • 50

1 Answers1

12

As you well noted, in-place layers don't usually work "out of the box".
For some layers, it is quite trivial ("ReLU" and other neuron activation layers).
However, for others it requires special handling in code. For example, the implementation of "PReLU" layer has specific cache bottom_memory_ member variable that stores information needed for backprop.
You can see similar code for other layers that specifically test for if (top[0] == bottom[0]) to see if the layer is used in an "in-place" case.

Moreover, it makes little sense to have an in-place layer for which the input and output are of different shapes, thus layers such as "Convolution", "InnerProduct", "Pool" are not considered as candidates for "in-place" layers.

Shai
  • 111,146
  • 38
  • 238
  • 371
  • Thanks. How about `in-place=True` and `in-place=False` for BatchNorm layer? Sometimes, I saw the papers `in-place=False` for the BatchNorm layer. Do we have any benefit for that case? – John May 17 '17 at 06:25
  • @user8264 I'm not very familiar with the internal of BatchNorm layer so I can't really comment about it. Sometimes using in place requires a bit more computation (for saving up space)... you'd really need to ask the person who specified `in_place=False`... – Shai May 17 '17 at 06:30