How does this regularization code affect loss?

Question

I have seen some learning with convolution neural network code. I do not understand the next part of this code.

loss = tf.reduce_sum(tf.nn.l2_loss(tf.subtract(train_output, train_gt)))
    for w in weights:
        loss += tf.nn.l2_loss(w)*1e-4

The first line is understandable. It compares the learned result with the label and then represents the square of the difference. And this is the definition of loss. But I do not understand the latter code: for w in weights:!!

Here w is a list of 10 weights and biases. So len(w) is 20(w10 + b10). But why does this code calculate the square of w and multiply it by 1e-4 to add to the loss?

Is it a necessary for the course of learning?

Vlad · Accepted Answer · 2019-04-11T10:27:48.463

2

This is the formula that you have:

tf.subtract(train_output, train_gt) does element-wise subtraction between two tensors train_output and train_gt.
tf.nn.l2_loss(tf.subtract(train_output, train_gt)) computes the l2-norm of the resulted tensor from (1).
tf.reduce_sum(tf.nn.l2_loss(tf.subtract(train_output, train_gt))) performs reduction sum over all dimensions (e.g. multiple samples in a batch that you have - N samples in the formula).
for w in weights: loss += tf.nn.l2_loss(w)*1e-4 adds l2-regularization term (squared sum of all l2-normalized weights in your model).

But why does this code calculate the square of w and multiply by 1e-4 to add to the loss? Is it a necessary course for learning?

It punishes the large values of your weights and limits your solution (in terms of weights) to some bounded region. Is it necessary? Sometimes yes, and sometimes no. There is no short answer. Start with reading this:

Tikhonov regularization in wikipedia
Pattern Recognition and Machine Learning by Bishop

edited Apr 11 '19 at 10:27

answered Apr 11 '19 at 10:08

Vlad

8,225
5
33
45

1

I think he wants more elaboration on tf.nn.l2_loss(w)*1e-4, perhaps in terms of the fundamental of the loss function update. – Angus Apr 11 '19 at 10:23
That's the answer I want! Thank you very very much! Is it ok to delete that code? – ddjfjfj djfiejdn Apr 11 '19 at 11:09
1

@ddjfjfjdjfiejdn, Glad to help. What code are you talking about? – Vlad Apr 11 '19 at 11:11
for w in weights: loss += tf.nn.l2_loss(w)*1e-4 <-- this :D – ddjfjfj djfiejdn Apr 11 '19 at 12:04
@ddjfjfjdjfiejdn, Try with and without. It may help or not, depends on the model, data e.t.c. Use it if it reduces error. – Vlad Apr 11 '19 at 12:11

How does this regularization code affect loss?

1 Answers1