1

I want to use W&B sweep parameters to dynamically construct a command line argument. How can I do this?

Modifying the W&B's example, suppose I have the following sweep:

program:
  train.py
method: grid
parameters:
  batch_size:
    values: [8, 10, 12]
  lr:
    values: [0.0001, 0.001]
command:
  - ${env}
  - python3
  - ${program}
  - ${args}

Suppose I want to pass in an additional flag like --output_dir /home/users/me/outputs/bs={args.bs}_lr={args.lr}. How can I do this?

Rylan Schaeffer
  • 1,945
  • 2
  • 28
  • 50

1 Answers1

0

From: https://docs.wandb.ai/guides/sweeps/faq#how-do-i-use-custom-cli-commands-with-sweeps

You can use W&B Sweeps with custom CLI commands if you normally configure some aspects of training by passing command line arguments.

For example, the proceeding code snippet demonstrates a bash terminal where the user is training a Python script named train.py. The user passes in values that are then parsed within the Python script:

/usr/bin/env python train.py -b \
    your-training-config \
    --batchsize 8 \
    --lr 0.00001

To use custom commands, edit the command key in your YAML file. For example, continuing the example above, that might look like so:

program:
  train.py
method: grid
parameters:
  batch_size:
    value: 8
  lr:
    value: 0.0001
command:
  - ${env}
  - python
  - ${program}
  - "-b"
  - your-training-config
  - ${args}

The ${args} key expands to all the parameters in the sweep configuration file, expanded so they can be parsed by argparse: --param1 value1 --param2 value2

Scott Condron
  • 1,902
  • 16
  • 20
  • Hi Scott, yes, I linked that very page in my first post! My question is more specific, and isn't answered on that page. The question is: how can one use `args` to construct a command line argument dynamically. In your answer, you don't actually answer this question – Rylan Schaeffer Feb 14 '23 at 04:41
  • 1
    Ok, I think I understand your question now. I would edit your script so that `output_dir` is defined within the script, and you use whatever is passed in from `bs` and `lr`. You can't reference `args.bs` within the YAML file. If you want to then save that path to W&B, you can add it to your wandb.config within your code. – Scott Condron Feb 20 '23 at 15:47
  • Your recommended solution is what I'm currently using but I'm trying to find something better. I'm using this solution because my code base relies on command line arguments being parsed by the Hugging Face Argument Parsers; consequently, I'm currently overriding one particular parser's values, but this is a hacky workaround – Rylan Schaeffer Feb 20 '23 at 21:12
  • Ok, that's useful context, thank you. Having just looked at the HfArgumentParser documentation, it seems like you can just omit the `output_dir` from your dataclass, or hardcode it to only be the _base_ directory `/home/users/me/outputs/`. That way you can append the dynamic parts of the directory (bs & lr) in Python. I'm sorry if this isn't elegant enough for you but I think it's the only option rather than overriding the parser value. – Scott Condron Feb 22 '23 at 17:43