-1

I asked a question earlier that might have been too specific so I'll ask again in more general terms. How does error get propagated backwards through a pooling layer when there are no weights to train? In the tensorflow video at 6:36 https://www.youtube.com/watch?v=Y_hzMnRXjhI there's a GlobalAveragePooling1D after Embedding, How does the error go backwards?

Shai
  • 111,146
  • 38
  • 238
  • 371
rkuang25
  • 79
  • 7
  • 2
    I’m voting to close this question because it is not about programming as defined in the [help] but about NN theory and/or methodology - please see the NOTE in https://stackoverflow.com/tags/neural-network/info – desertnaut Jan 13 '22 at 11:33

1 Answers1

1

A layer doesn't need to have weights in order to back-prop. You can compute the gradients of a global avg pool w.r.t the inputs - it's simply dividing by the number of elements pooled.
It is a bit more tricky when it comes to max pooling: in that case, you propagate gradients through the pooled indices. That is, during back-prop, the gradients are "routed" to the input elements that contributed the maximal elements, no gradient is propagated to the other elements.

Shai
  • 111,146
  • 38
  • 238
  • 371
  • If there's a fully connected layer after GlobalAveragePooling1D, does its layer error get divided by the number of elements pooled and added back to the pool from the Embedding layer element-wise? – rkuang25 Jan 13 '22 at 09:10
  • @rkuang25 the equations are quite simple. You can write them and see for yourself. This is the beauty of back-prop: it simply implements the [chain rule](https://en.wikipedia.org/wiki/Chain_rule), no "magic" or "algorithmic tricks" here. – Shai Jan 13 '22 at 09:29
  • for the GlobalAveragePooling1D example, the formula can be written as y = (x1 + x2 + ... + xn) / n and the gradient of this is simply 1/n where n is the number of elements in the pool, but what is done with this? Do I take multiply the vector of error neurons going into the average pooling layer by this 1/n scalar, then add each error neuron back to its respective index in the first layer? – rkuang25 Jan 13 '22 at 09:38
  • 1
    @rkuang25 this is basic "back prop" and SO posts/comments is not the right place to elaborate on it. Please refer to any basic course/tutorial on the subject to see how it is done. E.g., lecture number 3 [here](https://dl4cv.github.io/DL4CV_Spring21/schedule.html) there are slides and a video. – Shai Jan 13 '22 at 09:53