Stack Overflow
Stack Overflow

Questions tagged [tensorflow-gradient]

The tag for questions regarding gradient computation, auto-differentiation and evaluation in tensorflow

24 questions
18
votes
1 answer

How to monitor gradient vanish and explosion in keras with tensorboard?

I would like to monitor the gradient changes in tensorboard with keras to decide whether gradient vanish or explosion. What should I do?
Joey Chia
  • 195
  • 1
  • 9
6
votes
1 answer

How to print the gradients during training in Tensorflow?

In order to debug a Tensorflow model I need to see whether the gradients get changed or if there any nans in them. Simply printing a variable in Tensorflow does not work because all you see is:
minerals
  • 6,090
  • 17
  • 62
  • 107
4
votes
1 answer

OpenAI Gradient Checkpointing with Tensorflow Eager Execution

I have recently switched to Tensorflow Eager (currently working with TF 1.8.0) and like it a lot. However, I now have quite a large model which does not fit into my GPU Memory (GTX 1080Ti, 12GB VRAM) when run with the Gradient Tape which is needed…
Nemorior
  • 111
  • 6
2
votes
0 answers

RAM Usage keeps growing while training reinforcement learning agent

The other day I started training my Atari Breakout reinforcement learning agent. But after an hour and a half or so I noticed my screen started freezing and it became very difficult to interact with the computer via mouse. So, I decided I'd rerun…
Bryan Carty
  • 83
  • 6
2
votes
1 answer

How to accumulate my loss over mini batches then calculate my gradient

My main question is; is averaging the loss the same thing as averaging the gradient and how do i accumulate my loss over mini batches then calculate my gradient? I have been trying to implement policy gradient in Tensorflow and run into the issue…
Mike Jankowiak
  • 29
  • 3
2
votes
0 answers

making a memory efficient GAN in keras: `clear_session` causing conflicting tensorflow graphs

I have written the code for a general adversarial network that will run for 4000 epochs, however, after 2000 epochs- the model compiling time and memory usage become very inefficient and the code runs extremely slowly. I would like to make my code…
xenopus
  • 78
  • 8
2
votes
1 answer

stop_gradient in tensorflow

I am wondering if tf.stop_gradient stops the gradient computation of just a given op, or stops the update of its input tf.variable ? I have the following problem - During the forward path computation in MNIST, I would like to perform a set of…
user1193538
  • 103
  • 1
  • 3
  • 8
1
vote
1 answer

TensorFlow gradient of matrix wrt a matrix is not making sense

Let's say i have two matrix tf_t (shape : 5x3 ) and tf_b ( shape : 3x3). y_tf = tf.matmul(tf_t, tf_b) and then I've computed dy/dt using tf.gradient api import tensorflow as tf mat = [[0.8363, 0.4719, 0.9783], [0.3379, 0.6548, 0.3835], …
Anish
  • 13
  • 6
1
vote
2 answers

How to assign custom gradient to TensorFlow op with multiple inputs

I'm trying to use TensorFlow's @tf.custom_gradient functionality to assign a custom gradient to a function with multiple inputs. I can put together a working setup for only one input, but not for two or more. I've based my code on TensorFlow's…
aedificatori
  • 119
  • 1
  • 11
1
vote
1 answer

Using tensorflow, how can you average parameter gradient values over a number of batches and update using that average?

Like many people developing deep learning models, I am hindered by my hardware resources, namely GPU memory. I have an audio classification problem for which I am trying out a number of RNNs. The data is very large and I am only able to use small…
michael_question_answerer
  • 946
  • 1
  • 11
  • 24
1
vote
1 answer

Can I get the gradient of a tensor with respect to the input without applying the input?

For example, I need to compute the gradient of the cross_entropy with respect to x, but I need to apply another value to the gradient function. That is: f'(x)|x = x_t I think tf.gradients() function will only give the gradient at x = x. So does…
Sharyn Hu
  • 11
  • 1
1
vote
2 answers

tensorflow: change in parameter weights is different than it should be based on gradients

EDIT: Solved -- it was the stupidity of using different training examples for the gradients vs the optimizer update. OK this has me totally stumped. I have a parameter vector, let's call it w. w = [-1.34554319, 0.86998659, 0.52366061, 2.6723526…
anon
  • 407
  • 2
  • 12
1
vote
1 answer

Tensorflow - No gradients provided for any variable

I am experimenting some code on Jupyter and keep getting stuck here. Things work actually fine if I remove the line starting with "optimizer = ..." and all references to this line. But if I put this line in the code, it gives an error. I am not…
edn
  • 1,981
  • 3
  • 26
  • 56
1
vote
1 answer

Updating a tensor in tensorflow

I have defined an unsupervised problem in tensorflow, I need to update my B and my tfZ with every iteration, but I don't know how to update my tfZ using the tensorflow session. tfY = tf.placeholder(shape=(15, 15), dtype=tf.float32) with…
NicolaiF
  • 1,283
  • 1
  • 20
  • 44
1
vote
1 answer

Tensorflow: Gradient Calculation from Input to Output

I would like to calculate the gradients of the output of a neural network with respect to the input. I have the following tensors: Input: (num_timesteps, features) Output: (num_timesteps, 1) For the gradients from the inputs to the entire output…
Torben
  • 335
  • 3
  • 17
1
2