How to Add adam optimizer metric to tensorboard?

Question

When I trained model for several epochs and want to retrain it again for more epochs. How would Adam optimizer work. will it initialize the time from t =0 or will it save the last time step?

a) The documentation in tensorflow shows the following calculations. Is there a away I can add these metrics to tensorboard.

t <- t + 1
lr_t <- learning_rate * sqrt(1 - beta2^t) / (1 - beta1^t)

m_t <- beta1 * m_{t-1} + (1 - beta1) * g
v_t <- beta2 * v_{t-1} + (1 - beta2) * g * g
variable <- variable - lr_t * m_t / (sqrt(v_t) + epsilon)

there are no answers for a few questions since a long time question1 and question2.

I am actually getting a problem with error rate when re-training the model from the last checkpoint and I was not sure what exactly is happening with Adam optimizer in this case ?

score 0 · Answer 1 · answered Mar 13 '17 at 10:40

0

Your answer is a bit similar to this question I think: Saving the state of the AdaGrad algorithm in Tensorflow If you save and reload the state of the optimizer it will continue, if you don't load the state of your optimizer after training it will simply start again!

answered Mar 13 '17 at 10:40

rmeertens

4,383
3
17
42

Thanks . I will check this and let you know. is there a way we can add time, beta1 and beta2 to tensorboard? – Prakash Vanapalli Mar 13 '17 at 11:12
And when I print train_step as shown in the link u have provided using sess.run. I get None for every iteration. Is there any mistake I am making. – Prakash Vanapalli Mar 13 '17 at 11:16

How to Add adam optimizer metric to tensorboard?

1 Answers1