Argparse positional multiple choices default subset: Invalid Choice

Question

I'm using argparse to obtain a subset of elements using choices with nargs='*'. If no element is introduced, I want to return by default a subset of the elements, but I get an Invalid Choice error.

Here is my sample code:

from argparse import ArgumentParser

parser = ArgumentParser()
parser.add_argument(
    "choices",
    nargs="*",
    type=int,
    choices=[1,2,3,4,5],
    default=[1,3,5]
)

if __name__ == "__main__":
    print(parser.parse_args().choices)

The output I get is:

$ ./thing.py 3
[3]
$ ./thing.py 1 2
[1, 2]
$ ./thing.py
usage: thing.py [-h] [{1,2,3,4,5} ...]
thing.py: error: argument choices: invalid choice: [1, 3, 5] (choose from 1, 2, 3, 4, 5)

I can set default to an int, but not a list. Is there any workaround to this?

Edit: This only seems to happen with positional arguments. There aren't any problems if using nargs="+" and required=False in a non-positional argument.

I ended giving up on using positional arguments, it makes the CLI less readable. I'll leave up the question in case anyone has the same problem and wants to fix it. — David Davó, Oct 28 '21 at 09:42
This is a very interesting question and almost feels like sort of a bug. I was trying to start investigating (through source code) but haven't got the time fully right now. From what I gather, if you actually pass arguments, they are actioned in the following order one by one: first convert according to `type` then check if they are in `choices` then append to a list (because of `nargs`). When the default value is used, it seems that it is saved directly to the `Namespace` and then checked in the `choices` which will fail... — Tomerikoo, Oct 28 '21 at 10:17
Because `*` is satisfied by an empty list, its `default` gets special handling. Check the `_get_values` submethod. That includes the choices test, something a non-string default normally bypasses. @Tomerikoo — hpaulj, Oct 28 '21 at 10:42
@hpaulj Thanks for your response. Not sure I fully understand but I did see in the [docs](https://docs.python.org/3/library/argparse.html#default) *"If the default value is a string, the parser parses the value as if it were a command-line argument. In particular, the parser applies any type conversion argument, if provided, before setting the attribute on the Namespace return value. Otherwise, the parser uses the value as is"* but when I tried that (`default="1"`) I still got `invalid choice: '1' (choose from 1, 2, 3, 4, 5)` — Tomerikoo, Oct 28 '21 at 10:48
@Tomerikoo, I elaborated on this handling of defaults in an answer. — hpaulj, Oct 28 '21 at 16:34
Another way around this is to not specify a default. Then check the `args.choices` after. It if is `[]` (or `None`), assign the desired default. — hpaulj, Oct 28 '21 at 17:35

hpaulj · Answer 1 · 2021-10-28T17:34:04.753

To elaborate on my comments.

Normal processing of defaults, as @Tomerikoo quotes, is to process string defaults through both the type and choices check. The defaults are placed in the args namespace at the start of parsing. At the end, if the defaults have not be overwritten by using inputs, they will conditionally converted. Non-string defaults are left as is, without type or choices checking.

But a positional with nargs='*' is different. It is "always seen", since an empty list (i.e. none) of strings satisfies its nargs. With usual handling this would overwrite the default, resulting in Namespace(choices=[]).

To get around that the _get_values method replaces the [] with the default. This new value is passed through the choices test, but not through type. This results in the behavior that the OP encountered.

The relevant parts of _get_values are quoted below. The first handles nargs='?', the second '*', and the third normal cases. _get_value does the type test. _check_value does the choices test.

def _get_values(self, action, arg_strings):
    ....

    # optional argument produces a default when not present
    if not arg_strings and action.nargs == OPTIONAL:
        if action.option_strings:
            value = action.const
        else:
            value = action.default
        if isinstance(value, str):
            value = self._get_value(action, value)
            self._check_value(action, value)

    # when nargs='*' on a positional, if there were no command-line
    # args, use the default if it is anything other than None
    elif (not arg_strings and action.nargs == ZERO_OR_MORE and
          not action.option_strings):
        if action.default is not None:
            value = action.default
        else:
            value = arg_strings
        self._check_value(action, value)

    # single argument or optional argument produces a single value
    elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]:
        arg_string, = arg_strings
        value = self._get_value(action, arg_string)
        self._check_value(action, value)

I should add another block, which is used for '*' and non-empty user input:

    # all other types of nargs produce a list
    else:
        value = [self._get_value(action, v) for v in arg_strings]
        for v in value:
            self._check_value(action, v)

Note that here, individual strings are type and checked, where as in the first case, it's the whole default that is checked.

There have been bug/issues related to this, but no one has some up with a good reason to change it.

Argparse positional multiple choices default subset: Invalid Choice

1 Answers1