1

This is probably an easy question, but I just can't find it. But I'm also pretty new to all this, so maybe I'm just blind.

What is the default learning rate when using TensorFlowDNNRegressor with SGD or Adagrad? The default when using Adam or Adadelta seems to be 0.001, but I cannot find a default for Adagrad which is the default optimizer for TensorFlowDNNRegressor, nor for classic SGD.

Thanks!

grinsbaeckchen
  • 531
  • 1
  • 6
  • 15

2 Answers2

2

AdaGrad doesn't need a learning rate as it adapts component-wise (thus the name). A pretty concise comment: https://xcorr.net/2014/01/23/adagrad-eliminating-learning-rates-in-stochastic-gradient-descent/

Phillip Bock
  • 1,879
  • 14
  • 23
  • It still uses a base-learning rate. Though yes, you're right, this base-learning rate is certainly not as important to the success of the algorithm as for SGD. Thats why I understand that it is often left at its default value. But that is one of the reason why I asked, just to get a sense of what the default values of the (base-) learning rates are. – grinsbaeckchen Jun 12 '16 at 12:55
1

https://github.com/tensorflow/skflow/blob/master/g3doc/api_docs/python/estimators.md#class-skflowtensorflowdnnregressor-tensorflowdnnregressor

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/learn/python/learn/estimators/base.py

Default learning rate for TensorFlowDNNRegressor is 0.1 as mentioned in the above doc and code.

I checked the code, but there is no default value for learning rate of Adagrad optimiser https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/adagrad.py

Aravind Pilla
  • 416
  • 4
  • 14
  • Perfect! That means, that TensorFlowDNNRegressor always uses 0.1 as a default, no matter which optimizer I use, right? If I specify `regressor = skflow.TensorFlowDNNRegressor( hidden_units=final_hidden_units, optimizer='SGD')` it will use 0.1. If I instead define `optimizer='Adam'` it also uses 0.1 even though the default for Adam is 0.001. Did I understand that correctly? – grinsbaeckchen Jun 07 '16 at 11:57
  • yes, TensorFlowDNNRegressor uses by default 0.1 as learning_rate and the same will be passed to the optimizer. – Aravind Pilla Jun 07 '16 at 12:19