0

I am trying to implement an argument parser with the argparse module in Python. My goal is to allow a variable number of positional input arguments to the main parser and then optionally call a sub parser to perform some task. I'm interested in understanding why my implementation doesn't work as I expect it to and how I could implement something that is as close as possible to what I originally intended.

Here is a minimal example to illustrate what I am trying. I used Python 3.11 to create this example, although I originally encountered the issue using Python 3.9.

In [1]: import argparse

In [2]: parser = argparse.ArgumentParser()
   ...: parser.add_argument('positional', type=str, nargs='+')
   ...: parser.add_argument('-f', '--foo')
   ...: 
   ...: subparsers = parser.add_subparsers(dest='subcommand', required=False)
   ...: subparser1 = subparsers.add_parser('subcommand1')
   ...: subparser1.add_argument('-b', '--bar', action='store_true', )
   ...: subparser2 = subparsers.add_parser('subcommand2')
   ...: subparser2.add_argument('-b', '--baz', action='store_true')
Out[2]: _StoreTrueAction(option_strings=['-b', '--baz'], dest='baz', nargs=0, const=True, default=False, type=None, choices=None, required=False, help=None, metavar=None)

In [3]: parser.parse_args(['-f', 'a', 'b', 'c', 'd', 'subcommand1'])
Out[3]: Namespace(positional=['b', 'c', 'd'], foo='a', subcommand='subcommand1', bar=False)

In [4]: parser.parse_args(['-f', 'a', 'b', 'c', 'd', 'subcommand2'])
Out[4]: Namespace(positional=['b', 'c', 'd'], foo='a', subcommand='subcommand2', baz=False)

In [5]: parser.parse_args(['-f', 'a', 'b', 'c', 'd'])
usage: ipython [-h] [-f FOO] positional [positional ...] {subcommand1,subcommand2} ...
ipython: error: argument subcommand: invalid choice: 'd' (choose from 'subcommand1', 'subcommand2')

I expected (hoped) that, since required=False in the add_subparsers command, that I would have to option of not specifying any sub parser at all. Notably, the In [5] works, if I set nags='3' in the positional arguments.

Is this the intended behaviour? If so, what would be the intended way to achieve what I am trying to do?

Benjamin Jung
  • 1
  • Your subparsers is effectively a positional with a '?' nargs. Strings are allocated to positionals by 'position', not value. Use the '-f' to separate them. – hpaulj Apr 04 '23 at 15:29
  • Technically `subparsers` is a `positional` with special '+...' `nargs`. Originally it was required, but some patch unintentionally made it default 'not-required', and several further patches gave it the `required` parameter. But like all `positional`, even when they have `choices`, the strings are assigned by position, only after that are they checked for value. In all your examples 'a'` is assigned to '-f', and the rest split between 'positional' and 'subcommand'. The fact that the last matches the 'subcommand' choices (or not) doesn't make a difference. – hpaulj Apr 04 '23 at 16:24
  • As a general practice I discourage the use of variable nargs positionals with subparsers. It's hard to ensure that the subparsers argument gets the right string. If you must, use a flagged argument to separate the positionals from the subparsers. The same would apply to a flagged argument with a variable nargs. – hpaulj Apr 04 '23 at 16:26
  • The github issue (posted by you?) is https://github.com/python/cpython/issues/103520. I tried to explain why "fixing" this is not a trivial task. – hpaulj Apr 20 '23 at 21:00
  • Thanks for the explanation. If I understand you correctly, the exact implementation I was trying to achieve is not feasible. (I did not post the GitHub issue) – Benjamin Jung Apr 21 '23 at 07:22
  • I am not sure what you mean by "use '-f' to separate them". For example, `parser.parse_args(['b', 'c', 'd', '-f', 'a', 'subcommand2'])` still produces an error. – Benjamin Jung Apr 21 '23 at 07:33
  • You are right; it's still trying to use the last of the strings, 'd', as the subparser name. It still tries to match ['b','c','d'] to both `positionals`, the '*' and the '+...' (subparser). You will need to change 'positional' to a flagged argument, '-p' or '--positional'. – hpaulj Apr 21 '23 at 16:21

1 Answers1

0

Changing positional to -p works (sort of):

In [18]: In [1]: import argparse
    ...: 
    ...: In [2]: parser = argparse.ArgumentParser()
    ...:    ...: parser.add_argument('-p','--positional', type=str, nargs='+')
    ...:    ...: parser.add_argument('-f', '--foo')
    ...:    ...:
    ...:    ...: subparsers = parser.add_subparsers(dest='subcommand', required=
    ...: False)
    ...:    ...: subparser1 = subparsers.add_parser('subcommand1')
    ...:    ...: subparser1.add_argument('-b', '--bar', action='store_true', )
    ...:    ...: subparser2 = subparsers.add_parser('subcommand2')
    ...:    ...: subparser2.add_argument('-b', '--baz', action='store_true')
Out[18]: _StoreTrueAction(option_strings=['-b', '--baz'], dest='baz', nargs=0, const=True, default=False, type=None, choices=None, required=False, help=None, metavar=None)

In [19]: parser.parse_args(['-p','b', 'c', 'd', '-f', 'a', 'subcommand2'])
Out[19]: Namespace(positional=['b', 'c', 'd'], foo='a', subcommand='subcommand2', baz=False)

In [20]: parser.parse_args(['-p','b', 'c', 'd', '-f', 'a'])
Out[20]: Namespace(positional=['b', 'c', 'd'], foo='a', subcommand=None)

In [21]: parser.parse_args(['-p','b', 'c', 'd'])
Out[21]: Namespace(positional=['b', 'c', 'd'], foo=None, subcommand=None)

Here's where using the '-f' as separator works. It marks the end of the strings that get assigned to '-p'.

In [22]: parser.parse_args(['-p','b', 'c', 'd','subcommand2'])
Out[22]: Namespace(positional=['b', 'c', 'd', 'subcommand2'], foo=None, subcommand=None)

In [23]: parser.parse_args(['-p','b', 'c', 'd','-f','a','subcommand2'])
Out[23]: Namespace(positional=['b', 'c', 'd'], foo='a', subcommand='subcommand2', baz=False)

This may be more relevant to the github issue, but I realized that making the 'subparser' more like the true '?' optional positional won't work:

In [27]: p1=argparse.ArgumentParser()
         p1.add_argument('foo',nargs='*');
         p1.add_argument('bar',nargs='?');    
In [29]: p1.parse_args('a b d'.split())
Out[29]: Namespace(foo=['a', 'b', 'd'], bar=None)

In the combination of '*?', the '*' is greedy, grabbing all strings, and leaving none for the '?' (which is ok with none). So as long as subparsers is a positional, there's no way making it work in a non-required sense with a '*' (or even '+') positional. Another way to put it, it's hard to use a '*' positional anywhere except at the end.

hpaulj
  • 221,503
  • 14
  • 230
  • 353