1

I have a very basic doubt in tensorflow.

I have added a variable say 'var' in convolution layer, I want to update this variable('var') with gradient during training like our weights and bias are updated. I have added this variable to 'trainable params' but its not updated. Can someone shed light on how to train a variable?

mkar
  • 11
  • 5
  • You should post the code you've attempted, without it we are just guessing at a million possible things you could have done wrong and will just pick the most common errors people make and talk about them. You can edit the question to add the detail. Try to keep the code as focused on your particular question as possible. – David Parks May 07 '18 at 16:26
  • @DavidParks Sorry for that. I have added the code. – mkar May 07 '18 at 18:09

1 Answers1

2

The whole point of the optimization procedure is to update variables that the loss function depends on in a way to reduce the value of the loss. Neitherloss nor any variable that loss depends on does not depend on var (in other words var is not used in any computation that leads to the calculation of loss), therefore the gradient of loss with respect to var is not defined and var is not updated by the optimizer back_prop.

~~~

Post-edit answer: If you have a collection of variables param_list and you want to calculate L2 regularization term over all those variables you can do it the following way:

# get the list of variables to calculate L2 loss over
param_list = tf.get_collection('var_params')

# a list of L2 regularization terms for each variable in param_list
L2_lst = [tf.nn.l2_loss(param) for param in param_list]

# Total L2 loss
L2_loss = tf.add_n(L2_lst)

One would multiply L2_loss by a regularization parameter lambda before adding it to the main loss:

L2_loss = tf.mulpiply(lamda_param, L2_lst)
loss = tf.add(loss1, L2_loss)
abhuse
  • 1,086
  • 1
  • 11
  • 14
  • This is correct, and to add to it, in general, you should not need to use `var_list` or `add_to_collection`, that is handled by default in 99% of the cases in tensorflow. If you need to use these you either: 1) have a very special case and should understand exactly why you're using it, or 2) you've done something wrong. In this case the latter as @openmark has described. – David Parks May 07 '18 at 18:58
  • Yes Correct. Sorry that I didn't add the full code. I made the loss variable dependent of 'var' variable. But I don't see it being updated. Please look at the edited original code. – mkar May 07 '18 at 19:09
  • @DavidParks Do you mean I need not add weights and bias to collection ? If I don't mention the var_list, Is it supposed to find all dependencies of loss and optimize it by default? – mkar May 07 '18 at 19:15
  • @DavidParks Also If you look at the variable 'var', its initialized with a constant value 5.0. Does it have to do anything with it ? – mkar May 07 '18 at 19:26
  • @mkar what does `var_loss` calculate? is it L2 Regularization term? – abhuse May 07 '18 at 19:29
  • 1
    @mkar updated the answer, added information on how to calculate L2 regularization loss for a list of variables. – abhuse May 07 '18 at 20:24
  • @openmark by default variables are added to the `GraphKeys.TRAINABLE_VARIABLES` collection. If you don't specify `var_list` the optimizer will use this collection. So, yes, by default everything works as you would expect in all but the edge cases. Hence my recommendation. – David Parks May 07 '18 at 21:25
  • @openmark I don't want to calculate using loss functions. I am doing it mathematically as above. My ultimate loss tensor depends on loss1 and var_loss, where var_loss in turn depends on params_list, which is retrieved from collection and is nothing but a var tensor Can you tell why it doesn't affect var tensor in this case? – mkar May 07 '18 at 22:16