Does it make sense to use not binary ground truth values for binary crossentropy? is there any formal proof?
Looks like it used in practice: for example in https://blog.keras.io/building-autoencoders-in-keras.html, i.e. mnist images are not binary, but gray images.
Here is code examples:
1.Normal case:
def test_1():
print('-'*60)
y_pred = np.array([0.5, 0.5])
y_pred = np.expand_dims(y_pred, axis=0)
y_true = np.array([0.0, 1.0])
y_true = np.expand_dims(y_true, axis=0)
loss = keras.losses.binary_crossentropy(
K.variable(y_true),
K.variable(y_pred)
)
print("K.eval(loss):", K.eval(loss))
Output:
K.eval(loss): [0.6931472]
2.Not binary ground truth values case:
def test_2():
print('-'*60)
y_pred = np.array([0.0, 1.0])
y_pred = np.expand_dims(y_pred, axis=0)
y_true = np.array([0.5, 0.5])
y_true = np.expand_dims(y_true, axis=0)
loss = keras.losses.binary_crossentropy(
K.variable(y_true),
K.variable(y_pred)
)
print("K.eval(loss):", K.eval(loss))
Output:
K.eval(loss): [8.01512]
3.Ground truth values out of [0,1] range:
def test_3():
print('-'*60)
y_pred = np.array([0.5, 0.5])
y_pred = np.expand_dims(y_pred, axis=0)
y_true = np.array([-2.0, 2.0])
y_true = np.expand_dims(y_true, axis=0)
loss = keras.losses.binary_crossentropy(
K.variable(y_true),
K.variable(y_pred)
)
print("K.eval(loss):", K.eval(loss))
Output:
K.eval(loss): [0.6931472]
For some reason loss in test_1 and test_3 is the same, maybe it's because clipping [-2, 2] to [0, 1] but I can't see clipping code in Keras code.
Also it's interesting that for test_1 and test_2 loss value has large difference but in 1st case we have [0.5, 0.5] and [0.0, 1.0] and in 2nd case we have [0.0, 1.0] and [0.5, 0.5], which is the same values but in reversed order.
In Keras binary_crossentropy defined as:
def binary_crossentropy(y_true, y_pred):
return K.mean(K.binary_crossentropy(y_true, y_pred), axis=-1)
def binary_crossentropy(target, output, from_logits=False):
"""Binary crossentropy between an output tensor and a target tensor.
# Arguments
target: A tensor with the same shape as `output`.
output: A tensor.
from_logits: Whether `output` is expected to be a logits tensor.
By default, we consider that `output`
encodes a probability distribution.
# Returns
A tensor.
"""
# Note: tf.nn.sigmoid_cross_entropy_with_logits
# expects logits, Keras expects probabilities.
if not from_logits:
# transform back to logits
_epsilon = _to_tensor(epsilon(), output.dtype.base_dtype)
output = tf.clip_by_value(output, _epsilon, 1 - _epsilon)
output = tf.log(output / (1 - output))
return tf.nn.sigmoid_cross_entropy_with_logits(labels=target,
logits=output)
