CNN algorithm predicts value of 1.0 and thus the cross-entropy cost function gives a divide by zero warning

Question

I am using CNN to do binary classification.

While the cross-entropy function is calculated by the code:

(-1 / m) * np.sum(np.multiply(Y, np.log(AL)) + np.multiply(1 - Y, np.log(1 - AL)))

when the algorithm predicts a value of 1.0, this cost function gives a divide by zero warning. How to deal with it? Or is there any way to avoid the prediction to become exactly 1.0, like any pre-processing skill?

When I use higher numerical precision, it works fine. But I am still curious about the warning.

Where is the division by zero in that code? The only division is by m. — Dr. Snoopy, Jun 03 '19 at 05:51
@MatiasValdenegro, `np.log(0)` generates the warning `RuntimeWarning: divide by zero encountered in log` — Warren Weckesser, Jun 03 '19 at 06:05
This question looks like a duplicate of https://stackoverflow.com/questions/50018625/how-to-handle-log0-when-using-cross-entropy — Warren Weckesser, Jun 03 '19 at 06:07

score 0 · Answer 1 · answered Jun 03 '19 at 04:30

This looks like try/except should handle the divide by zero error.

try:
    my_var = (-1 / m) * np.sum(np.multiply(Y, np.log(AL)) + np.multiply(1 - Y, np.log(1 - AL)))
except ZeroDivisionError:
    print("I tried to calculate the result, but I got a divide-by-zero error. Maybe you need to use higher precision.")
    exit()

score 0 · Answer 2 · answered Oct 16 '19 at 18:06

0

Try np.clip. This bounds the elements of an nd array between a min and a max value.

epsilon = 1e-10
clipped_AL = np.clip(AL, epsilon, 1. - epsilon)
(-1 / m) * np.sum(np.multiply(Y, np.log(clipped_AL)) + np.multiply(1 - Y, np.log(clipped_AL)))

answered Oct 16 '19 at 18:06

Yash Sharma

94
1
5

CNN algorithm predicts value of 1.0 and thus the cross-entropy cost function gives a divide by zero warning

2 Answers2