In machine learning and information theory, the cross entropy is a measure of distance (inverse similarity) between two probability distributions over the same underlying set of events. Cross entropy is the common choice of the loss function in neural networks for classification tasks.
Questions tagged [cross-entropy]
360 questions
6
votes
0 answers
masking probabilities before cross entropy calculation in tensorflow
I am in a reinforcement learning setting, where my environments action space depends on state. As a result, I go through the following procedure when sampling behavior actions:
(1) generate probability logits for all possible actions
(2) compute…
John H
- 135
- 8
6
votes
1 answer
Why "softmax_cross_entropy_with_logits_v2" backprops into labels
I am wondering why in Tensorflow version 1.5.0 and later, softmax_cross_entropy_with_logits_v2 defaults to backpropagating into both labels and logits. What are some applications/scenarios where you would want to backprop into labels?
Safoora Yousefi
- 173
- 1
- 10
6
votes
1 answer
Tensorflow - loss starts high and does not decrease
i started writing Neuronal Networks with tensorflow and there is one Problem i seem to face in each of my example Projects.
My loss allways starts at something like 50 or higher and does not decrease or if it does, it does so slowly that after all…
J Polack
- 118
- 1
- 1
- 5
6
votes
1 answer
TensorFlow: Are my logits in the right format for cross entropy function?
Alright, so I'm getting ready to run the tf.nn.softmax_cross_entropy_with_logits() function in Tensorflow.
It's my understanding that the 'logits' should be a Tensor of probabilities, each one corresponding to a certain pixel's probability that it…
rikkitikkitumbo
- 954
- 3
- 17
- 38
5
votes
1 answer
torch.nn.CrossEntropyLoss over Multiple Batches
I am currently working with torch.nn.CrossEntropyLoss. As far as I know, it is common to compute the loss batch-wise. However, is there a possibility to compute the loss over multiple batches?
More concretely, assume we are given the data
import…
Vivian
- 335
- 2
- 10
5
votes
2 answers
tf keras SparseCategoricalCrossentropy and sparse_categorical_accuracy reporting wrong values during training
This is tf 2.3.0. During training, reported values for SparseCategoricalCrossentropy loss and sparse_categorical_accuracy seemed way off. I looked through my code but couldn't spot any errors yet. Here's the code to reproduce:
import numpy as…
kawingkelvin
- 3,649
- 2
- 30
- 50
5
votes
1 answer
How do I calculate cross-entropy from probabilities in PyTorch?
By default, PyTorch's cross_entropy takes logits (the raw outputs from the model) as the input. I know that CrossEntropyLoss combines LogSoftmax (log(softmax(x))) and NLLLoss (negative log likelihood loss) in one single class. So, I think I can use…
kHarshit
- 11,362
- 10
- 52
- 71
5
votes
1 answer
Custom cross-entropy loss in pytorch
I have done a custom implementation of the pytorch cross-entropy loss function (as I need more flexibility to be introduced later). The model I intend to train with this will need a considerable amount of time to train and the resources available…
Inder
- 3,711
- 9
- 27
- 42
5
votes
0 answers
Custom loss for deeplab
I wanted to add custom loss for Deeplab v3 that would work for NOT one hot encoding labels but for saliency prediction. So instead of Deeplab loss implementation that you see below:
label = tf.to_int32(label > 0.2)
one_hot_labels =…
Eva
- 83
- 5
5
votes
1 answer
Binary Crossentropy to penalize all components of one-hot vector
I understand that binary cross-entropy is the same as categorical cross-entropy in case of two classes.
Further, it is clear for me what softmax is.
Therefore, I see that categorical cross-entropy just penalizes the one component (probability) that…
hallo02
- 307
- 3
- 11
5
votes
1 answer
How to use weighted categorical crossentropy on FCN (U-Net) in Keras?
I have built a Keras model for image segmentation (U-Net). However in my samples some misclassifications (areas) are not that important, while other are crucial, so I want to assign higher weight in loss function to them. To complicate things…
johndodo
- 17,247
- 15
- 96
- 113
5
votes
1 answer
TensorFlow model gets loss 0
import tensorflow as tf
import numpy as np
def weight(shape):
return tf.Variable(tf.truncated_normal(shape, stddev=0.1))
def bias(shape):
return tf.Variable(tf.constant(0.1, shape=shape))
def output(input,w,b):
return tf.matmul(input,w)+b
x_columns…
yoshi
- 61
- 1
- 3
5
votes
1 answer
is Cross Entropy With Softmax proper for Multi-label Classification?
As mentioned here, cross entropy is not a proper loss function for multi-label classification. My question is "is this fact true for cross entropy with softmax too?". If it is, how it can be matched with this part of the document.
I should mention…
OmG
- 18,337
- 10
- 57
- 90
4
votes
3 answers
RuntimeError: 0D or 1D target tensor expected, multi-target not supported I was training a deep learning model but i am getting this issue
*My Training Model*
def train(model,criterion,optimizer,iters):
epoch = iters
train_loss = []
validaion_loss = []
train_acc = []
validation_acc = []
states = ['Train','Valid']
for epoch in range(epochs):
…
Helloworld8688
- 63
- 1
- 2
- 7
4
votes
1 answer
Print the validation loss in each epoch in PyTorch
I want to print the model's validation loss in each epoch, what is the right way to get and print the validation loss?
Is it like this:
criterion = nn.CrossEntropyLoss(reduction='mean')
for x, y in validation_loader:
optimizer.zero_grad()
out =…
rubiks
- 53
- 1
- 3