How to implement a filter in tensorflow?

Question

I have a convolutional neural network with three images as inputs:

x_anchor = tf.placeholder('float', [None, 4900], name='x_anchor')
x_positive = tf.placeholder('float', [None, 4900], name='x_positive')
x_negative = tf.placeholder('float', [None, 4900], name='x_negative')

Within a train function, I feed the placeholders with the actual images:

 input1, input2, input3 = training.next_batch(start,end)
    ....some other operations...
loss_value  = sess.run([cost], feed_dict={x_anchor:input1, x_positive:input2, x_negative:input3})

I'm using a triplet loss function on these three inputs (that's actually the cost variable above):

def triplet_loss(d_pos, d_neg):

    margin = 0.2

    loss = tf.reduce_mean(tf.maximum(0., margin + d_pos - d_neg))

    return loss

How can I filter the losses, so only the images with loss_value > 0 will be used to train the network?

How can I implement something like:

if(loss_value for input1, input2, input3 > 0)
  use inputs to train network
else
 do nothing/try another input

What I have tried so far:

I took the images one by one (input1[0], input2[0], input3[0]), calculated the loss, and if the loss was positive I would calculate (and apply) the gradients. But the problem is I use dropout in my model and I have to apply the model twice on my inputs:

First to calculate the loss and verify whether it's greater than 0
Second to run the optimizer: this is when things go wrong. As I mentioned before, I use dropout, so the results of the model on my inputs are different, so the new loss will sometimes be 0 even if the loss determined at step 1 is greater than 0.

I also tried to use tf.py_func but got stuck.

The gradient of your weights with respect to the zero-loss parts of the batch should be zero, right? So at most you should need to count them when figuring out how to normalize the loss, unless I'm missing something. You could use `tf.cond` to conditionally run the optimizer ops (in the batch size one setting), though. — Allen Lavoie, Oct 30 '17 at 17:06

score 3 · Accepted Answer · answered Jul 19 '18 at 10:57

There's a new TensorFlow feature called “AutoGraph”. AutoGraph converts Python code, including control flow, print() and other Python-native features, into pure TensorFlow graph code. For example:

@autograph.convert()
def huber_loss(a):
  if tf.abs(a) <= delta:
    loss = a * a / 2
  else:
    loss = delta * (tf.abs(a) - delta / 2)
  return loss

becomes this code at execution time due to the decorator:

def tf__huber_loss(a):
  with tf.name_scope('huber_loss'):
    def if_true():
      with tf.name_scope('if_true'):
        loss = a * a / 2
        return loss,
    def if_false():
      with tf.name_scope('if_false'):
        loss = delta * (tf.abs(a) - delta / 2)
        return loss,
    loss = ag__.utils.run_cond(tf.less_equal(tf.abs(a), delta), if_true,
        if_false)
    return loss

What you wanted to do could have been implemented before using tf.cond().

I found out about this through this medium post.

How to implement a filter in tensorflow?

1 Answers1