validation during training of Estimator

Question

With the TensorFlow r1.3 monitors are deprecated:

"2016-12-05", "Monitors are deprecated. Please use tf.train.SessionRunHook.") and Estimator.train(input_fn,hooks,..) works only with hooks.

How to implement the functionality of validation monitor with hooks?

TensorFlow r1.3, but the problem starts since TensorFlow 1.2 — Aga, Aug 08 '17 at 10:30

simlmx · Answer 1 · 2017-11-14T19:59:34.317

3

EDIT: As pointed out in the comments, this feels like the right thing to do, but will reinitialize the weights every time it's evaluated, which makes it pretty much useless...

I ended up being able to monitor my validation error (which is what I understand you are trying to do) using the train_and_evaluate function. The EvalSpec object you have to use has parameters start_delay_secs and throttle_secs to define the frequency at which the error (or whatever you have defined in your estimator's EVAL mode) will be computed.

My code looks somewhat like

classifier = tf.estimator.Estimator(
    model_fn=model_fn,
    model_dir=model_dir,
    params=params)

train_spec = tf.estimator.TrainSpec(
    input_fn = input_fn,
)

eval_spec = tf.estimator.EvalSpec(
    input_fn = valid_input_fn,
    throttle_secs=120,
    start_delay_secs=120,
)

tf.estimator.train_and_evaluate(
    classifier,
    train_spec,
    eval_spec
)

edited Nov 14 '17 at 19:59

answered Nov 10 '17 at 18:13

simlmx

999
12
17

Hi simlmx, we tried the same but it seems that the net gets reloaded with initial weights for every evaluation when the hook fires in. Did you experience the same? Ideally we'd like to evaluate the network with its current weights without reloading anything, just the input_fn should change. – Philipp F Nov 14 '17 at 13:10
You are absolutely right, I experienced the same thing. I'm not sure but I think it didn't restart from initial weights, but rather from the last time it was saved. I tried to work around that by saving the model more often so that the problem wouldn't be as bad, but I ended up getting tired of `Estimator`s altogether and returned to more "classical" `tensorflow`. – simlmx Nov 14 '17 at 19:56

score 2 · Accepted Answer · edited Aug 13 '18 at 15:02

2

I have been using SummarySaverHook instead of Monitors. They are not "as powerful" just yet, and the training material has not been updated with a description on how exactly replicate the Monitor functionality.

Here is how i use it:

summary_hook = tf.train.SummarySaverHook(
    save_steps=SAVE_EVERY_N_STEPS,
    output_dir='./tmp/rnnStats',
    scaffold=tf.train.Scaffold(),
    summary_op=tf.summary.merge_all())

print("Classifier.train")
classifier.train(input_fn=train_input_fn, steps=1000, hooks=[summary_hook])

edited Aug 13 '18 at 15:02

MiniQuark

46,633
36
147
183

answered Aug 08 '17 at 12:59

Axen

36
1

7

How would you do a validation on a separate dataset with your solution? – Maxime De Bruyn Aug 29 '17 at 20:40
you may want to checkout this: https://stackoverflow.com/questions/44793703/replace-validation-monitors-with-tf-train-sessionrunhook-when-using-estimators – Jiang Xiang Oct 16 '17 at 02:38

validation during training of Estimator

2 Answers2

Linked