purpose of 'num_samples' in Tune of Ray package for hyperprameter optimization

Question

I am trying to perform a hyper parameter optimization task for a LSTM (pure Tensorflow) with Tune. I followed their example on the hyperopt algorithm. In the example they have used the below line inside the 'config' section.

"num_samples": 10 if args.smoke_test else 1000,

The documenatation does not explain what this is. I am uable to determine if this is a useful piece of code or how am I supposed to alter this for my scenario. So it will be great if I can know the meaning of this line of code.

The example hyperopt code can be found through this link

this is called a ternary expression - it sets the `"num_samples"` variable to 10 or 1000 depending on some other given variable. From the names I would _guess_ it samples some given data at 10 or 1000 points to create whatever it does ... you would need to look through the documentation to see for yourself. `"num_samples"` is provided as kwargs param. Maybesomewhere around here: https://ray.readthedocs.io/en/latest/tune-usage.html?highlight=num_samples — Patrick Artner, Jan 01 '19 at 17:00

score 4 · Answer 1 · answered Jan 01 '19 at 17:08

4

You can find the parameter in the documentation of run_experiments.

By default, each random variable and grid search point is sampled once. To take multiple random samples, add num_samples: N to the experiment config. If grid_search is provided as an argument, the grid will be repeated num_samples of times.

Essentially the parameter is part of the configuration and can be used to sample your data multiple times instead of only once.

Your demo code however uses run_experiment:

config = {
    "my_exp": {
        "run": "exp",
        "num_samples": 10 if args.smoke_test else 1000,
        "config": {
            "iterations": 100,
        },
        "stop": {
            "timesteps_total": 100
        },
    }
}
algo = HyperOptSearch(space, max_concurrent=4, reward_attr="neg_mean_loss")
scheduler = AsyncHyperBandScheduler(reward_attr="neg_mean_loss")
run_experiments(config, search_alg=algo, scheduler=scheduler)  # here the config is passed

answered Jan 01 '19 at 17:08

Patrick Artner

50,409
9
43
69

Thanks, any idea what 'args.smoke_test' is supposed to do? – Suleka_28 Jan 01 '19 at 19:39
args.smoke_test is just a boolean we use to make running tests easy. You can simply consider the line to basically be `"num_samples": 10`. – richliaw Jan 11 '19 at 04:35
@Sul `10 if args.smoke_test else 1000,` is a ternary expression that says "if `args.smoke_test` is true use `10` else `1000` - think of it like a "small" and a "big" testingscenario based on some input. Hint: if you put a @ before a username it auto_completes it for you if this person was inside this answer/qeustion and _more importantly_ the person gets a message that someone sain something - somehow your comment eluded me :) – Patrick Artner Jan 11 '19 at 06:36

score 1 · Answer 2 · answered Jan 01 '19 at 17:07

As per the documentation:

num_samples (int) – Number of times to sample from the hyperparameter space. Defaults to 1. If grid_search is provided as an argument, the grid will be repeated num_samples of times.

Substitue of repeat:

repeat (int) – Deprecated and will be removed in future versions of Ray. Use num_samples instead

Usage:

"num_samples": 10

num_samples=10

class ray.tune.Experiment(name,run,stop=None,config=None,trial_resources=None,
repeat=1,num_samples=1,local_dir=None,upload_dir=None,checkpoint_freq=0,
checkpoint_at_end=False,max_failures=3,restore=None)

purpose of 'num_samples' in Tune of Ray package for hyperprameter optimization

2 Answers2