Stack Overflow
Stack Overflow

Questions tagged [cross-entropy]

In machine learning and information theory, the cross entropy is a measure of distance (inverse similarity) between two probability distributions over the same underlying set of events. Cross entropy is the common choice of the loss function in neural networks for classification tasks.

360 questions
14
votes
3 answers

How can I implement a weighted cross entropy loss in tensorflow using sparse_softmax_cross_entropy_with_logits

I am starting to use tensorflow (coming from Caffe), and I am using the loss sparse_softmax_cross_entropy_with_logits. The function accepts labels like 0,1,...C-1 instead of onehot encodings. Now, I want to use a weighting depending on the class…
Roger Trullo
  • 1,436
  • 2
  • 10
  • 19
14
votes
3 answers

Possible explanations for loss increasing?

I've got a 40k image dataset of images from four different countries. The images contain diverse subjects: outdoor scenes, city scenes, menus, etc. I wanted to use deep learning to geotag images. I started with a small network of 3…
JohnAllen
  • 7,317
  • 9
  • 41
  • 65
13
votes
4 answers

Why not use mean squared error for classification problems?

I am trying to solve a simple binary classification problem using LSTM. I am trying to figure out the correct loss function for the network. The issue is, when I use the binary cross-entropy as loss function, the loss value for training and testing…
Hussain Ali
  • 133
  • 1
  • 1
  • 4
13
votes
1 answer

Tensorflow, what does from_logits = True or False mean in sparse_categorical_crossentropy of Tensorflow?

In Tensorflow 2.0, there is a loss function called tf.keras.losses.sparse_categorical_crossentropy(labels, targets, from_logits = False) Can I ask you what are the differences between setting from_logits = True or False? My guess was that when…
Won Jun Son
  • 135
  • 1
  • 6
12
votes
2 answers

Pytorch: Weight in cross entropy loss

I was trying to understand how weight is in CrossEntropyLoss works by a practical example. So I first run as standard PyTorch code and then manually both. But the losses are not the same. from torch import nn import…
user3363813
  • 567
  • 1
  • 5
  • 19
11
votes
3 answers

Keras: binary_crossentropy & categorical_crossentropy confusion

After using TensorFlow for quite a while I have read some Keras tutorials and implemented some examples. I have found several tutorials for convolutional autoencoders that use keras.losses.binary_crossentropy as the loss function. I thought…
daniel451
  • 10,626
  • 19
  • 67
  • 125
10
votes
4 answers

How to do point-wise categorical crossentropy loss in Keras?

I have a network that produces a 4D output tensor where the value at each position in spatial dimensions (~pixel) is to be interpreted as the class probabilities for that position. In other words, the output is (num_batches, height, width,…
Alex I
  • 19,689
  • 9
  • 86
  • 158
9
votes
3 answers

Calculating Cross Entropy in TensorFlow

I am having a hard time with calculating cross entropy in tensorflow. In particular, I am using the function: tf.nn.softmax_cross_entropy_with_logits() Using what is seemingly simple code, I can only get it to return a zero import tensorflow as…
David Kaftan
  • 1,974
  • 1
  • 12
  • 18
8
votes
1 answer

shall I apply softmax before cross entropy?

The pytorch tutorial (https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py) trains a convolutional neural network (CNN) on a CIFAR dataset. class Net(nn.Module): def…
Liyuan Zhang
  • 91
  • 1
  • 3
8
votes
2 answers

Why does sigmoid & crossentropy of Keras/tensorflow have low precision?

I have the following simple neural network (with 1 neuron only) to test the computation precision of sigmoid activation & binary_crossentropy of Keras: model = Sequential() model.add(Dense(1, input_dim=1,…
syeh_106
  • 1,007
  • 2
  • 9
  • 15
8
votes
1 answer

Keras: Weighted Binary Crossentropy Implementation

I'm new to Keras (and ML in general) and I'm trying to train a binary classifier. I'm using weighted binary cross entropy as a loss function but I am unsure how I can test if my implementation is correct. Is this an accurate implementation of…
Stefan Iarca
  • 152
  • 1
  • 7
8
votes
1 answer

Why is my implementations of the log-loss (or cross-entropy) not producing the same results?

I was reading up on log-loss and cross-entropy, and it seems like there are 2 approaches for calculating it, based on the following equations. The first one is the following. import numpy as np from sklearn.metrics import log_loss def…
Vikash Singh
  • 13,213
  • 8
  • 40
  • 70
8
votes
1 answer

Cross entropy loss in pytorch nn.CrossEntropyLoss()

maybe someone is able to help me here. I am trying to compute the cross entropy loss of a given output of my network print output Variable containing: 1.00000e-02 * -2.2739 2.9964 -7.8353 7.4667 4.6921 0.1391 0.6118 5.2227 6.2540 …
Elias E.
  • 101
  • 1
  • 1
  • 8
8
votes
2 answers

How to implement Weighted Binary CrossEntropy on theano?

How to implement Weighted Binary CrossEntropy on theano? My Convolutional neural network only predict 0 ~~ 1 (sigmoid). I want to penalize my predictions in this way : Basically, i want to penalize MORE when the model predicts 0 but the truth was…
KenobiBastila
  • 539
  • 4
  • 16
  • 52
7
votes
2 answers

Keras apply different weight to different misclassification

I am trying to implement a classification problem with three classes: 'A','B' and 'C', where I would like to incorporate penalty for different type of misclassification in my model loss function (kind of like weighted cross entropy). Class weight is…
bambi
  • 373
  • 1
  • 3
  • 9
1
2
3
…
23 24