log(1+exp(X)) in Tensorflow (avoiding under and over flows)

Question

I was debugging my program and I've realized that I my loss outputted NaN. These NaN values comes from the fact that I'm computing tf.log(1 + tf.exp(X))

where X is a 2d tensor. Indeed, When a value of X is large enough then tf.exp() returns +Inf and so tf.log(1 + exp(X)) will return +Inf. I was wondering if there exists a neat trick to avoid underflows and overflows in this case.

I have tried:

def log1exp(x):
    maxi = tf.reduce_max(x)
    return maxi + tf.log(tf.exp(x - maxi) + tf.exp(-maxi))

but it doesn't handle underflows in this case...

Also I've glanced at tf.reduce_logsumexp but it necessarily reduce the tensor along an axis... while I want to keep the same shape!

Finally I know that tf.log(1 + exp(X)) is almost equal to X for large values of X but I think that designing a function that will output X when X > threshold and log(1+exp(X)) otherwise is not very neat.

Thank you

P-Gn · Accepted Answer · 2020-01-06T22:10:09.720

4

This function is already implemented in tensorflow under the name tf.math.softplus, and takes care of overflows and underflows.

edited Jan 06 '20 at 22:10

answered Jun 04 '18 at 12:27

P-Gn

23,115
9
87
104

`tf.nn.softplus` does *not* take care of overflows, as one can easily verify by passing it a large input. – sk29910 Jan 03 '20 at 01:29
@sebastian_k `tf.math.softplus(np.finfo(np.float32).max)` does not overflow over here, and that is pretty much the biggest input I can think of. – P-Gn Jan 06 '20 at 22:07
thanks for testing that – very strange, as I regularly have to work around `tf.nn.softplus` spitting out `nan` when the inputs get too large. I wonder if there's something else to it? – sk29910 Jan 07 '20 at 01:43
@sebastian_k Might indeed be worth its own question here on SO. – P-Gn Jan 07 '20 at 09:21
I eat my words, you're right: `core/kernels/softplus_op.h` clearly branches on the input value. I wonder if my problems are caused in the gradient calculation ... I'll have to dig into that! :) – sk29910 Jan 07 '20 at 17:15
```with tf.Session() as sess:``` – DachuanZhao Jul 25 '22 at 03:55
```tf.math.softplus(tf.math.exp(10.0*10)).eval()``` – DachuanZhao Jul 25 '22 at 03:55
It returns ```Inf``` – DachuanZhao Jul 25 '22 at 03:55
@DachuanZhao That's because `exp(10.0*10)` lies outside `float32` range. – P-Gn Nov 18 '22 at 12:22

log(1+exp(X)) in Tensorflow (avoiding under and over flows)

1 Answers1