0

I am trying to write a custom loss in Tensorflow v2, for simplicity let's say that I'm using Mean Squared Error loss as follows,

loss_object = tf.keras.losses.MeanSquaredError()


def loss(model, x, y, training):
  # training=training is needed only if there are layers with different
  # behavior during training versus inference (e.g. Dropout).
  y_ = model(x, training=training)
  return loss_object(y_true=y, y_pred=y_)

Now I know that Tensorflow does automatic differentiation.

But I want to specify my custom gradient, in the BackPropagation algorithm, if we use MSE, we have to do the following

Is it possible in Keras to replace with where p is a tensor that is passed during training before applying gradients.

Rohan Mohapatra
  • 163
  • 1
  • 11
  • How is it different from MSE where ground truth (y_true) is `p` instead of `y` – mujjiga Feb 12 '20 at 03:54
  • `p` tensor will change with every epoch, consider p to be `tf.Variable()`, that case `p` must be fed into the optimizer on every epoch. – Rohan Mohapatra Feb 12 '20 at 04:19
  • @mujjiga Consider that my loss was different, for simplicity, I have used `MeanSquaredError`, I want to use stable gradients with a custom loss. – Rohan Mohapatra Feb 12 '20 at 04:57
  • If I understand correctly all you need to do is write a custom loss function that can be compatible with some `tf.keras.optimizer`. You should use `tf.keras.backend` functions to write your custom loss. Check out [this](https://stackoverflow.com/questions/43818584/custom-loss-function-in-keras) answer – Siddhant Tandon Feb 12 '20 at 10:29
  • @SiddhantTandon that's true, I need to write the custom loss, what I'm looking for is essentially writing a custom gradient for that loss and to make it more specific, during runtime (for every epoch), I want to compute a different gradient and not use the autograd provided by `tensorflow` (as mentioned in the question). – Rohan Mohapatra Feb 12 '20 at 10:42
  • i dont understand what is the need for writing custom gradient. I mean to say if you define any loss function in whichever way you want, it's derivative will be the same as computed by autograd and as computed by you in your custom gradient function. Since you said at every epoch you want a different gradient, i assume your `p` vector keeps changing at every epoch because of which you get a different loss and hence different gradients. Am i correct ? – Siddhant Tandon Feb 12 '20 at 10:57
  • If you want to write your own custom training loop i.e. not the `model.fit()`function where you dont have a control on whats happening in every epoch, you can do it as mentioned on [this](https://www.tensorflow.org/guide/keras/train_and_evaluate#part_ii_writing_your_own_training_evaluation_loops_from_scratch) page. – Siddhant Tandon Feb 12 '20 at 11:07
  • yes `p` tensor will change every epoch, because of which my gradients will change, there is no proper documentation on how to pass a custom tensor to the gradient, Hence i was looking for an alternative with custom gradients. – Rohan Mohapatra Feb 12 '20 at 11:27
  • I m using a custom training loop, but when i do tf.gradients() it is doing autograd, my understanding is that instead of using the gradients, if in future i would have to approximate the gradient, relying on autograd would be difficult, – Rohan Mohapatra Feb 12 '20 at 11:30
  • I think you just need to use a custom training loop where at each step you pass in your `p vector` to your loss function and every epoch you access the gradients using `tape.gradient` do stuff with it if you want and then simply call the optimizer. If you look at the attached link you pass your `p` vector to the `loss_fn`. And i think you can use directly the MSE loss. Just pass in your p at each step. – Siddhant Tandon Feb 12 '20 at 11:41

0 Answers0