0

I am trying to create a custom loss function for a regression problem that would minimize the number of elements that falls above a certain threshold. my code for this is:

import tensorflow as tf
epsilon = 0.000001

def custom_loss(actual, predicted):     # loss
    actual = actual * 12
    predicted = predicted * 12

    # outputs a value between 1 and 20
    vector = tf.sqrt(2 * (tf.square(predicted - actual + epsilon)) / (predicted + actual + epsilon))

    # Count number of elements above threshold value of 5
    fail_count = tf.cast(tf.size(vector[vector>5]), tf.float32)

    return fail_count

I however, run into the following error:

ValueError: No gradients provided for any variable: ...

How do I solve this problem?

keno
  • 3
  • 1

1 Answers1

1

I don't think you can use this loss function, because the loss does not vary smoothly as the model parameters vary - it will jump from one value to another different value as parameters pass a theshold point. So tensorflow can't calculate gradients, and so can't train the model.

It's the same reason that 'number of images incorrectly classified' isn't used as a loss function, and categorical cross-entropy, which does vary smoothly as parameters change, is used instead.

You may need to find a smoothly varying function that approximates what you want.

[Added after your response below...]

This might do it. It becomes closer to your function as temperature is reduced. But it may not have good training dynamics, and there could be better solutions out there. One approach might be to start training with relatively large temperature, and reduce it as training progresses.

temperature = 1.0
fail_count=tf.reduce_sum(tf.math.sigmoid((vector-5.)/temperature))
David Harris
  • 646
  • 3
  • 11
  • okay, my intentions are to generate predictions that would mostly lie within the limit set by the threshold, standard mse doesn't do this, this is why I have opted to use this custom loss. Do you have any suggestions of a loss function that implements this kind of idea? – keno Aug 10 '22 at 10:56
  • In relation to your update, that was a really neat solution and I do understand the idea. However, I get nan losses while training even after tweaking the temperature (I am guessing that is what you mean by "...not have good training dynamics". – keno Aug 11 '22 at 02:18
  • I can't see how you can get nan results, except from nan values in vector. If there are you may need to remove them before doing the calculation. tf.reduce_sum(tf.boolean_mask(vector, tf.logical_not(tf.math.is_nan(vector)))) – David Harris Aug 11 '22 at 11:45
  • ... I got that last expression a bit wrong, but hope you get the general idea – David Harris Aug 11 '22 at 11:52
  • Great! it finally works now. I used tf.reduce_sum(vector[vector>0]) to get rid of the nan values. Thank you for your time David :) – keno Aug 11 '22 at 16:13