2

I was trying to train using h2o.automl(). But the training exited due to timeout. I know max_runtime_secs can be set to higher numbers. But it would be great if we can train for 1 hour, then save it somewhere. Again train it next day from where it was left in day 1.

How to do that ?

I have tried by setting project_name - but nothing is saved on exit. So if we turn off pc and restart, it is of no use.

I have used the following code for this :

library( h2o )

h2o.init( nthreads = -1, max_mem_size = '10240m' )

train = h2o.importFile( 'train.csv' )

automl_model = h2o.automl( y = 'outcome', training_frame = train, nfolds = 3, max_runtime_secs = 1800,

                           project_name = 'automl_aus_tennis' )

Link to train.csv : http://www.mediafire.com/file/qj7yiju15ncgnax/train.csv

Nicolás Ozimica
  • 9,481
  • 5
  • 38
  • 51
Soumya Boral
  • 1,191
  • 14
  • 28

2 Answers2

4

You can run h2o.automl() repeatedly with the same project_name and different seeds to build additional models and add them to the same leaderboard. I do this all the time.

There's a pull request, which should go in soon, which allows you to specify algos not to run. This allows you to tune which hyperparameter searches get executed on each AutoML run.

You'll need to keep the h2o-3 instance running to achieve your aim since you can't currently persist the state of an AutoML run to disk and load it into a new h2o-3 instance, or add models loaded from disk into a leaderboard. Those would be useful feature requests. :-)

Erin LeDell
  • 8,704
  • 1
  • 19
  • 35
Raymond Peck
  • 41
  • 1
  • 2
    Keeping the h2o-3 instance running is difficult. I will post this as a feature request in github. It would be highly helpful for complex models. Also why different seed is required. If different one is used, automl will start from another starting model ? Also how do we know what models are added in subsequent runs - using the leaderboard slot right ? – Soumya Boral Jan 07 '18 at 19:53
  • If you use the same seed for each run then the sequence of hyperparameter vectors will be the same for each hyperparameter search in the subsequent runs, so you'll get duplicate models between runs. The timestamp for the `AutoML` run will be part of the `model_id` of each model, so it'll be easy to tell which models came from which runs. – Raymond Peck Jan 08 '18 at 05:52
1

There's no way to continue running an H2O AutoML job if you shut down the H2O cluster (or machine) and restart the H2O cluster at a later date. If you leave the H2O cluster running, you can add more models to your leaderboard by running h2o.automl() again with the same value for project_name.

If you need to shut down the H2O cluster between runs, then the best you can do is to set a different seed in the h2o.automl() function when you run it a second, third, fourth, etc. time because your random grid searches within the AutoML run will be different. That way you're likely to get new models instead of models you've already trained in previous AutoML runs.

Erin LeDell
  • 8,704
  • 1
  • 19
  • 35