How to pass parameters to a training script in Azure Machine Learning service?

Question

I am trying to submit an experiment in Azure Machine Learning service locally on an Azure VM using a ScriptRunConfig object in my workspace ws, as in

from azureml.core import ScriptRunConfig    
from azureml.core.runconfig import RunConfiguration
from azureml.core import Experiment

experiment = Experiment(ws, name='test')
run_local = RunConfiguration()

script_params = {
    '--data-folder': './data',
    '--training-data': 'train.csv'
}

src = ScriptRunConfig(source_directory = './source_dir', 
                      script = 'train.py', 
                      run_config = run_local, 
                      arguments = script_params)

run = experiment.submit(src)

However, this fails with

ExperimentExecutionException: { "error_details": { "correlation": { "operation": "bb12f5b8bd78084b9b34f088a1d77224", "request": "iGfp+sjC34Q=" }, "error": { "code": "UserError", "message": "Failed to deserialize run definition"

Worse, if I set my data folder to use a datastore (which likely I will need to)

script_params = {
    '--data-folder': ds.path('mydatastoredir').as_mount(),
    '--training-data': 'train.csv'
}

the error is

UserErrorException: Dictionary with non-native python type values are not supported in runconfigs.
{'--data-folder': $AZUREML_DATAREFERENCE_d93269a580ec4ecf97be428cd2fe79, '--training-data': 'train.csv'}

I don't quite understand how I should pass my script_params parameters to my train.py (the documentation of ScriptRunConfig doesn't include a lot of details on this unfortunately).

Does anybody know how to properly create src in these two cases?

A workaround would be to add defaults to my ArgumentParser in my `train.py`, but that's not really the solution to this... — Davide Fiocco, Apr 09 '19 at 10:00
Can you use an `Estimator` instead? Asking because this approach works with `Estimators`, but not with `ScriptRunConfig` (and I've no idea why they're not accepting the same type of arguments for both). — Vlad Iliescu, Apr 09 '19 at 16:41
Hey! Thanks for this. So I first tried indeed with `Estimator`, but I have a bit of a esoteric dependency that I have to handle via `pip install git+https://github.com/...` as the package is not something I can add in `conda_packages` AFAIK. Therefore, following the guide I assumed "using RunConfiguration object and ScriptRunConfig object[...] gives you a lot of flexibility and maximum control" and installed that dependency locally on my VM. As this doesn't seem a viable route I might go for `Estimator` and a custom docker image with my dependency installed, but I still have to try that... — Davide Fiocco, Apr 09 '19 at 17:57

Davide Fiocco · Answer 1 · 2019-04-16T08:06:39.480

In the end I abandoned ScriptRunConfig and used Estimator as follows to pass script_params (after having provisioned a compute target):

estimator = Estimator(source_directory='./mysourcedir',
                      script_params=script_params,
                      compute_target='cluster',
                      entry_script='train.py',
                      conda_packages = ["pandas"],
                      pip_packages = ["git+https://github.com/..."], 
                      use_docker=True,
                      custom_docker_image='<mydockeraccount>/<mydockerimage>')

This also allowed me to install my pip_packages dependency by putting on https://hub.docker.com/ a custom_docker_image Docker image created from a Dockerfile like:

FROM continuumio/miniconda
RUN apt-get update
RUN apt-get install git gcc g++ -y

(it worked!)

For anyone reading this after 2021... Estimators are now deprecated in more recent versions of AzureML, since `azuremlsdk == 1.19.0` https://learn.microsoft.com/en-us/azure/machine-learning/how-to-migrate-from-estimators-to-scriptrunconfig — Davide Fiocco, Jan 09 '21 at 00:04

score 4 · Answer 2 · answered May 08 '19 at 06:41

The correct way of passing arguments to the ScriptRunConfig and RunConfig is as a list of strings according to https://learn.microsoft.com/nb-no/python/api/azureml-core/azureml.core.runconfiguration?view=azure-ml-py.

Modified and working code would be as follows.

from azureml.core import ScriptRunConfig    
from azureml.core.runconfig import RunConfiguration
from azureml.core import Experiment

experiment = Experiment(ws, name='test')
run_local = RunConfiguration()

script_params = [
    '--data-folder',
    './data',
    '--training-data',
    'train.csv'
]

src = ScriptRunConfig(source_directory = './source_dir', 
                      script = 'train.py', 
                      run_config = run_local, 
                      arguments = script_params)

run = experiment.submit(src)

Here is a full example from microsoft that shows the entire process. https://learn.microsoft.com/en-us/learn/modules/train-local-model-with-azure-mls/3-script-parameters — brian_ds, Mar 31 '22 at 16:07

How to pass parameters to a training script in Azure Machine Learning service?

2 Answers2