1

How could I scale gradients where the loss comes from sparse_softmax_cross_entropy_with_logits. For example, I was trying to divide by 128 as below, but I found error:

new_gradients = [(grad/128, var) for (grad, var) in gradients] 
TypeError: unsupported operand type(s) for /: 'IndexedSlices' and 'int'

The code I was using is below:

loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=labels)

gradients = opt.compute_gradient(loss)

new_gradients = [(grad/128, var) for (grad, var) in gradients]

train_step = opt.appy_gradients(new_gradients)
Mostofa Patwary
  • 11
  • 2

1 Answers1

0

I found a way to solve the problem as follows:

new_gradients = [(grad/128, var) for (grad, var) in gradients]

should be

new_gradients = [(tf.div(grad, 128), var) for (grad, var) in gradients]
Mostofa Patwary
  • 11
  • 2