0

I'm now trying to use tf.losses.sigmoid_cross_entropy on an unbalanced dataset. However, I'm a little confused on the parameter weights. Here are the comments in the documentation:

weights: Optional Tensor whose rank is either 0, or the same rank as labels, and must be broadcastable to labels (i.e., all dimensions must be either 1, or the same as the corresponding losses dimension).

I know in tf.losses.softmax_cross_entropy the parameter weights can be a rank 1 tensor with weight for each sample. Why must the weights in tf.losses.sigmoid_cross_entropy have the same rank as labels?

Can anybody answer me? Better with an example.

Mika Sundland
  • 18,120
  • 16
  • 38
  • 50
Ben
  • 11
  • 3

1 Answers1

0

You want your loss to be weighted and so tensorflow expects that you will provide it weight for each of your label. Consider the following example

Labels: [0, 0, 0, 1, 0]  
possible_weights1: [1]
possible_weights2: [1, 2, 1, 1, 1]
illegal_weights1: [1, 2]
illegal_weights2: [[1], [2]]

Here your labels have rank 1 (only 1 dimension), so tensorflow expects that either you'll provide weight for each of the element in label (as demonstrated in possible_weights2) or will provide weight for each dimension (as demonstrated in possible_weights1, which is broadcasted to [1, 1, 1, 1, 1]).
But, if you have illegal_weights2 as your weights, then tensorflow does not understand how it should handle the two dimensions in the weights, since there is only one dimension in labels? So your rank should always be same.
illegal_weights1 is case where rank is same but weights are neither of the same length as labels, nor of length 1 (which can be broadcasted), but are of length 2 which cannot be broadcasted and hence is illegal.

layog
  • 4,661
  • 1
  • 28
  • 30
  • Thanks! What should the weights are if the labels are onehot? – Ben Feb 09 '18 at 06:09
  • It does not matter that your labels are `onehot` encoded. Just set the weights according to your needs. Generally, higher weights are set to the sparse labels. – layog Feb 09 '18 at 06:45
  • It is said in the help that the weights should be the same rank as the labels. For example, if the labels has the shape [10, 2], what should the weights be? Given the doc, it should have the same shape [10, 2], but actually it has shape [10, 1]. I'm confused here. – Ben Feb 09 '18 at 07:37
  • Shape of your labels is `[10, 2]` and hence it is of rank 2 and your weights are of shape `[10, 1]` and so they are also of rank 2. Now, the first dimension of your weights and labels match but second does not, so, your weights will be automatically broadcasted to `[10, 2]` by tensorflow. This is what the following statement means. `(i.e., all dimensions must be either 1, or the same as the corresponding losses dimension)` – layog Feb 09 '18 at 07:54
  • That makes sense! Thank you very much @layog! – Ben Feb 09 '18 at 09:00