0

I want to deploy a stacked model to Azure Machine Learning Service. The architecture of the solution consists of three models and one meta-model. Data is a time-series data.

I'd like the model to automatically re-train based on some schedule. I'd also like to re-tune hyperparameters during each re-training.

AML Service offers HyperDriveStep class that can be used in the pipeline for automatic hyperparameter optimization.

Is it possible - and if so, how to do it - to use HyperDriveStep with time-series CV?

I checked the documentation, but haven't found a satisfying answer.

Aleksander Molak
  • 123
  • 7

1 Answers1

1

AzureML HyperDrive is a black box optimizer, meaning that it will just run your code with different parameter combinations based on the configuration you chose. At the same time, it supports Random and Bayesian sampling and has different policies for early stopping (see here for relevant docs and here for an example -- HyperDrive is towards the end of the notebook).

The only thing that your model/script/training needs to adhere to is to be launched from a script that takes --param style parameters. As long as that holds you could optimize the parameters for each of your models individually and then tune the meta-model, or you could tune them all in one run. It will mainly depend on the size of the parameter space and the amount of compute you want to use (or pay for).

Daniel Schneider
  • 1,797
  • 7
  • 20
  • Thanks @Daniel. I went thru the docs again and my understanding is that I can perform time series cross-validation inside my `Estimator` script. Then return whatever I want to optimize (e.g. mean loss over all the folds) and optimize using `HyperDriveStep` based on that information. Is that right? – Aleksander Molak Aug 14 '19 at 09:03
  • 1
    Yes, that is correct. It is all about what you log back via run.log('metric', value), with 'metric' being the priprimary_metric_name you provided in the HyperDriveConfig used. – Daniel Schneider Aug 14 '19 at 11:59