1

I'm currently building a NN from scratch where we want to identify based on two input variables (X_1 and X_2) what their output will be (0 or 1). I have 2 hidden layers with sigmoid activation on all neurons however I get stuck when calculating the cross-entropy. Suppose in the output layer I have the predictions [0.50, 0.57] however the true output is 0, so [1, 0]. How do I calculate the cross-entropy over this binary output example? Does anyone have any suggestions/tips?

Tijn van de Luijtgaarden
  • 21
  • 6

1 Answers1

1

Here is a function that I wrote and use to calculate the cross-entropy given a list of predictions and a list of true labels.

from math import log
# calculate the cross-entropy of predictions and true labels
def cross_entropy(y, p):
    # y[i] is list of real labels
    # p[i] is the probability of predicting 1
    m = len(y)
    sum_vals = 0
    for i in range(m):
        # first term is for label=1, second term is for label=0
        sum_vals += float(y[i]) * log(p[i]) + (1 - float(y[i])) * log(1 - p[i])
    R = -1/m * sum_vals
    return R

Here the values in the list of labels y are each either 0 or 1, and the probabilistic predictions from the network are in the list p.

kevin-robb
  • 104
  • 8
  • First of all, thank you very much! However do you know if I can interpreted the output values from the NN as probabilities? Since I only have been using the sigmoid activations in the neurons – Tijn van de Luijtgaarden Dec 15 '20 at 22:17
  • So, my function assumes the NN outputs a single variable which represents the probability of predicting 1. If I'm understanding correctly, your implementation outputs both a probability of predicting 0 and a probability of predicting 1. So, what I would recommend doing is normalizing your predictions. i.e., in your example of `[0.50,0.57]`, setting `y=0` and `p=0.57/(0.50 + 0.57)` when you create the lists to send through the function I included. – kevin-robb Dec 16 '20 at 22:10