how can I specify a list on the command line

Question

I have the following configuration with hydra:

data_files: null
theta: 0.2
use_weights: true

I can override this as:

python -m apps.test_hydra data.data_files=x

However, when I try to specify a list as:

python -m apps.test_hydra data.data_files=[x,y]

Then I get the following error:

zsh: no matches found: data.data_files=[x,y]

I am not sure if this is some issue with the zsh shell or hydra issue

zsh is interpreting `[` and `]` before exec-ing python. Try specifying your list of files as `date.data_files=x,y,z,` — James McPherson, Jan 16 '22 at 23:06
I tried that and that gives the error: `hydra.errors.OverrideParseException: mismatched input '' expecting {':', BRACKET_OPEN, BRACE_OPEN, FLOAT, INT, BOOL, NULL, UNQUOTED_CHAR, ID, ESC, WS, QUOTED_VALUE, INTERPOLATION}` — Luca, Jan 16 '22 at 23:12
@JamesMcPherson ok, it seems doing something like: `data.data_files=\[x,u\]` helps. — Luca, Jan 16 '22 at 23:14
BTW, this isn't just a zshism: you'd get similar behavior from bash if you turned on the `failglob` option (or your argument would silently disappear with `nullglob`); using the `[x,y]` unquoted when you mean the expression to become part of a literal argument isn't good practice in _any_ shell. — Charles Duffy, Jan 16 '22 at 23:15

jkr · Accepted Answer · 2022-01-16T23:11:49.087

8

Wrap the string in quotes to avoid [ being interpreted as a glob expression.

python -m apps.test_hydra 'data.data_files=[x,y]'

On a hydra github issue, they use the following syntax. You might have to do something similar.

$ python -m apps.test_hydra 'data@data_files=[x, y]'

edited Jan 16 '22 at 23:11

answered Jan 16 '22 at 23:09

jkr

Nice. Like the quotes option. – Luca Jan 16 '22 at 23:16
This is documented in Hydra's command line grammar page. https://hydra.cc/docs/advanced/override_grammar/basic/#dictionaries-and-lists – Omry Yadan Jan 17 '22 at 01:17

score 2 · Answer 2 · answered Jan 16 '22 at 23:11

2

Use " \ " before special characters.

python -m apps.test_hydra data.data_files=\[x,y\]

answered Jan 16 '22 at 23:11

Emile

3

You _could_ do that, surely, but quoting those characters is arguably easier to read. Backslash escapes are contextually dependent -- f/e, they're literal in single-quoted contexts, and can need to be doubled when inside of backticks; quotes don't have as many "gotchas" associated. – Charles Duffy Jan 16 '22 at 23:14

2 Answers2