Intuition for back propagation

Question

Below is a forward pass and partly implemented backward pass of back propagation of a neural network :

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

X_train = np.asarray([[1,1], [0,0]]).T
Y_train = np.asarray([[1], [0]]).T

hidden_size = 2
output_size = 1
learning_rate = 0.1

    forward propagation

w1 = np.random.randn(hidden_size, 2) * 0.1
b1 = np.zeros((hidden_size, 1))
w2 = np.random.randn(output_size, hidden_size) * 0.1
b2 = np.zeros((output_size, 1))

Z1 = np.dot(w1, X_train) + b1
A1 = sigmoid(Z1)

Z2 = np.dot(w2, A1) + b2
A2 = sigmoid(Z2)

derivativeA2 = A2 * (1 - A2)
derivativeA1 = A1 * (1 - A1)

    first steps of back propagation

error = (A2 - Y_train)
dA2 = error / derivativeA2
dZ2 = np.multiply(dA2, derivativeA2)

What is the intuition behind :

error = (A2 - Y_train)
dA2 = error / derivativeA2
dZ2 = np.multiply(dA2, derivativeA2)

I understand error is the difference between the current prediction A2 and actual values Y_train.

But why divide this error by the derivative of A2 and then multiply the result of error / derivativeA2 by derivativeA2 ? What is intuition behind this ?

score 1 · Accepted Answer · answered Jan 06 '18 at 13:33

These expressions are indeed confusing:

derivativeA2 = A2 * (1 - A2)
error = (A2 - Y_train)
dA2 = error / derivativeA2

... because error doesn't have a meaning on its own. At this point, the goal is compute the derivative of the cross-entropy loss, which has this formula:

dA2 = (A2 - Y_train) / (A2 * (1 - A2))

See these lecture notes (formula 6) for the derivation. It just happens that the previous operation is sigmoid and its derivative is A2 * (1 - A2). That's why this expression is used again to compute dZ2 (formula 7).

But if you had a different loss function (say, L2) or a different squeeze layer, then A2 * (1 - A2) wouldn't be reused. These are different nodes in the computational graph.

Intuition for back propagation

1 Answers1