classification
Title: argparse: optional subparsers
Type: enhancement Stage: test needed
Components: Library (Lib) Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: DasIch, G2P, Julian, bethard, bewest, bkabrda, chris.jerdonek, csernazs, dsully, elsdoerfer, eric.araujo, labrat, nvie, paul.j3, r.david.murray, seblu, zzzeek
Priority: low Keywords: patch

Created on 2010-07-13 21:29 by nvie, last changed 2014-04-10 17:26 by paul.j3.

Files
File name Uploaded Description Edit
ed0fce615582.diff eric.araujo, 2012-01-12 16:02 review
check.py labrat, 2013-01-18 22:33 Test for optional subparsers vs. "too few arguments"
required.patch paul.j3, 2013-04-13 06:24 review
Repositories containing patches
https://bitbucket.org/bewest/argparse/
https://bitbucket.org/bewest/argparse/
https://bitbucket.org/bewest/cpython/
https://bitbucket.org/bewest/cpython/
http://bitbucket.org/bewest/cpython/#bewest/argparse
Messages (29)
msg110231 - (view) Author: Vincent Driessen (nvie) Date: 2010-07-13 21:29
**NOTE**: This is a re-post of http://code.google.com/p/argparse/issues/detail?id=47

What steps will reproduce the problem?
parser = argparse.ArgumentParser()
sub = parser.add_subparsers()
sub.add_parser("info")
parser.add_argument("paths", "+")
parser.parse_args(["foo", "bar"])

What is the expected output? What do you see instead?
Expected behavior is that, failing to match one of the subparser inputs
("info"), the parser checks if the argument matches any of the top-level
arguments, in this case the 'paths' multi-arg. In other words, it should be
possible to make the subparser be optional, such that when the subparser
argument fails to retrieve any valid subparser, the remaining args are
parsed as if no subparser exists. At present, it does not seem possible to
make a subparser be optional at all.
Perhaps this could be exposed to the user as:
parser.add_subparsers(nargs=argparse.OPTIONAL)
or something to that effect. Or, allow a default subparser to be specified.
I.e.,

sub = parser.add_subparsers()
info = sub.add_parser("info")
main = sub.add_parser("main")
sub.default = main

I'm sure the point will come up that the current behavior is correct,
because given a subparser like "info", a user could easily make a mistake
like "myapp ino foo bar" and rather than get a safe error be given
something unexpected. For this reason, I think the default behavior is
usually going to be correct. BUT, it would still be nice if it could be
optional, so that developers could be free to make that call. Sometimes the
potential user errors aren't really an issue, and having to explicitly set
a subparse arg every time can be a nuissance.
msg110232 - (view) Author: Vincent Driessen (nvie) Date: 2010-07-13 21:30
Changed the title, so it shows that the feature request is for argparse.
msg110244 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-07-13 23:42
I've added Steven as nosy so he knows this was reposted here.  I've also set the priority to low.  Personally I'm at least -0 on this, since if I use a command that has subcommands I expect to get an error if I supply an invalid subcommand.  As you say, however, the command designer *could* be supplied with the opportunity to shoot themselves in the foot :)

The issue isn't going to go anywhere unless someone proposes a patch, though.
msg110309 - (view) Author: Vincent Driessen (nvie) Date: 2010-07-14 18:18
Actually, this is a rather common concept. Broadly used tools like for example Git use this kind of subcommand handling.

This command shows all remotes:
   git remote            (i.e. is like git remote list)

Showing/removing remotes is done using subsubcommands:
   git remote show [...]
   git remote rm [...]

That, in combination with the explicit design goal that "[argparse] isn't dogmatic about what your command line interface should look like" should be enough reason to be wanting this, in my humble opinion.
msg113267 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-08-08 14:30
See also 9540, which has an alternate proposal (that I don't like as much) for how to handle parser arguments supplied after subparsers are declared.

Reviewing this, I'm now +1 on fixing this *somehow*, since clearly there is an ambiguity here that needs to be resolved.
msg113512 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2010-08-10 09:34
Seems like there's minimally the bug that argparse should currently throw an error if you add an argument after subparsers (since that argument will never be parsed under the current semantics).

I do believe that supporting an optional command like the "git remote" example is useful, but as RDM suggests, this probably won't go anywhere unless someone proposes a patch.
msg113558 - (view) Author: Michael.Elsdörfer (elsdoerfer) Date: 2010-08-10 21:06
To expand on my case from issue9540, I have a bunch of commands, each of which should enable a specific subset of options only available the individual command, but all of the commands share the same behavior in taking nargs='*' positional arguments:

./script.py --global-option command --command-option arg1 arg2 arg3

For example:

./backups.py -c /etc/tarsnap.conf make --no-expire job1 job2

If no positional arguments are given, all jobs defined in the config file are run. Or, in the above example, only "job1" and "job2" are run.

The positional arguments are the same for *all* commands. Now I can define them separately for each subparser, which is what I'm currently doing, but I kind of like having the global usage instructions (script.py -h) indicating the fact that positional arguments can be passed after the command. 

In fact, right now I'm able to sort of achieve this by defining the positional nargs arguments both globally (to have them show in usage) and in each subparser (to have them parsed). This wouldn't be possible anymore if argparse where to throw an error after adding arguments after a subparser, although probably a more correct behavior.

Anyway, while the two issues are clearly related, I don't think that the two are necessarily mutually exclusive. argparse could allow both optional subparsers (if no subparser matches), as well as pass control back to the parent parser once an already matched subparser is no longer able to handle further command line input. Or optionally, support defining subparsers as "options only", so that positional arguments would always be handled by the parent parser.

Now, I can see how this could potentially become messy if we start talking about these positional arguments handled by the parent then being followed by more flags, which would then presumably also be handled by the parent etc. On the other hand, my use case doesn't seem that strange to me.
msg113575 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-08-11 00:42
Stable releases don’t go into stable branches, so I’m editing versions. I also remove 3.3 since it doesn’t exist now, it means “this won’t go in 3.2”.
msg113577 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-08-11 01:22
Wow, it is late. I wanted to write: New features don’t go into stable branches.
msg121250 - (view) Author: (G2P) Date: 2010-11-15 23:23
Trying to spec this, here is a proposed API:

    parser = argparse.ArgumentParser()
    sub = parser.add_subparsers(default='show')
    sub_show = sub.add_parser('show')
    sub_add = sub.add_parser('add')

If default isn't passed, the subcommand isn't optional.
If default is passed, and no explicit subcommand is given,
the default subcommand is picked.
Arguments are given to the top parser; passing arguments
to the subcommand requires naming it explicitly.

As far as motivation, I'd like to change a program that
uses --choice options (that can have a default) to use
more expressive subcommands. Some programs rely on implicit
subcommands a lot; the ip command on linux is a good
example.
msg121271 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2010-11-16 08:04
I think the proposed API looks fine and should be backwards compatible since add_subparsers will currently throw an exception with a default= argument.

In case someone feels like writing a patch, you'll want to look at _SubParsersAction.__init__, which will need to grow the default= argument, and pass a different nargs= argument on. I think you'll need to define a new nargs type which means you probably also need to look at ArgumentParser._get_nargs_pattern as well.
msg144484 - (view) Author: Benjamin West (bewest) Date: 2011-09-24 06:55
I spent some time looking at this, as I was interested in
using this pattern to simulate what git and hg do.  I
considered a few modifications and then found this bug.  I
think the default keyword passed to
_SubParsersAction.__init__ makes sense.

I started on a patch, that looks promising, but I'm having
trouble getting the regexp right.

Here's a changeset higlighting where I think the
problematic regexp is:
https://bitbucket.org/bewest/argparse/changeset/938e1e91ddd0

https://gist.github.com/1202975#file_test_opt_subcommand.py
Is the meager little test I put together.
msg144512 - (view) Author: Benjamin West (bewest) Date: 2011-09-24 22:52
https://github.com/bewest/argparse/tree/bewest
https://bitbucket.org/bewest/argparse/changesets

I think this does the right thing.
msg149533 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2011-12-15 11:31
If you can make your patch relative to the cpython source tree, and add a couple tests, it will be easier to review.

Thanks for working on this!
msg150757 - (view) Author: Benjamin West (bewest) Date: 2012-01-06 18:08
Ok, Steven, that sounds reasonable.

I checked out git-svn python and started comparing diffs... I'm a little confused.  What version of argparse should be patched to provide this feature?

My HG version from https://code.google.com/p/argparse/ seems to contain a version of argparse 1.2 while, my git-svn checkout of python seems to contain an argparse 1.1.  Should I attempt to bring cpython's version up to date as well, or attempt to strip out the version bump changes?
msg150784 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-07 03:47
You should work in the 3.3 standard library, i.e. on Lib/argparse.py in the default branch of the CPython Mercurial repository.  See the devguide for more info.  Thanks!
msg150816 - (view) Author: Benjamin West (bewest) Date: 2012-01-07 19:52
Thanks Eric.  I was thrown by this document: http://wiki.python.org/moin/Git which describes fetching the sources from SVN using git.  I'm comfortable doing either, but it doesn't resolve my confusion.

The version of argparse in the python checkout is 1.1: http://hg.python.org/cpython/file/default/Lib/argparse.py
64 __version__ = '1.1' but differs from the SVN version.

whereas the argparse version available via google code is 1.2.  The diffs indicate several changes not related to the change I'm attempting to make, which prevent my patch from applying cleanly.  Looks like the HG version includes the 1.2 code... but I'm not sure why it would differ from SVN's trunk.
msg150821 - (view) Author: Benjamin West (bewest) Date: 2012-01-07 21:28
Ok, here's a rough attempt at stubbing this out against a python checkout.  Will try to look at adding tests.

(BTW, subsequent GETs should not modify the bug tracker... this seems like a bug since GET should be idempotent, but SFTN from the double posting.)
msg150953 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-09 16:47
Thanks for persevering in the face of VCS complications :)  I have added a warning to the obsolete Git wiki page; I can’t do anything for the argparse Google code page.  Anyway, trust us that argparse in the 3.3 stdlib is the place where development happens.  (The Python Subversion repository is now dead.)

As for the tracker, well, its use of HTTP and URIs is somewhat idiosyncratic.  It’s far from perfect and definitely not as elegant or REST-compliant that one could wish for.  Anyway, it’s just a tool that serves us rather well; I’ve never seen a double submission issues like here before.
msg153771 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2012-02-20 09:31
The implementation looks along the right track. Now it just needs some tests.
msg170491 - (view) Author: Benjamin West (bewest) Date: 2012-09-14 17:07
https://gist.github.com/1202975#file_test_opt_subcommand.py

I sketched out a sloppy test earlier.  I think this test is probably not quite comprehensive enough, and I'm not sure it fits into the python style either.  I suppose there are other tests I can more or less copy.
msg180229 - (view) Author: W. Trevor King (labrat) Date: 2013-01-18 22:33
Since [1] it seems like subparsers *are* optional by default.  At least I get “error: too few arguments” for version 70740 of Lib/argparse.py, but no error for version 70741.  It looks like this may be an unintentional side effect, since I see no mention of subparsers in #10424.

[1]: http://hg.python.org/cpython/rev/cab204a79e09
msg181855 - (view) Author: mike bayer (zzzeek) Date: 2013-02-10 21:09
um, this seems like a regression/bug?   I now have users complaining that my apps are broken because of this change as of Python 3.3.    My application is supposed to return the "help" screen when no command is given.  Now I get a None error because argparse is not trapping this condition:

from argparse import ArgumentParser
parser = ArgumentParser(prog='test')
subparsers = parser.add_subparsers()
subparser = subparsers.add_parser("foo", help="run foo")
parser.parse_args()

$ python3.2 test.py
usage: test [-h] {foo} ...
test: error: too few arguments

$ python3.3 test.py
$

This seems very much like a major feature has been yanked away from argparse, now I have to check for this condition explicitly.

am I on the right issue here or do I need to open something new ?
msg186052 - (view) Author: Sebastien Luttringer (seblu) Date: 2013-04-04 22:16
I got the same issue that mike bayer with argparse doesn't throw error when subparser are missing.

Is it a bug which should be fixed in Python or in all python script? This sounds like an API break.
msg186387 - (view) Author: paul j3 (paul.j3) * Date: 2013-04-09 07:08
I think this problem arises from a change made in http://bugs.python.org/issue10424

Changeset to default (i.e. development) is
http://hg.python.org/cpython/rev/cab204a79e09

Near the end of _parse_known_args it removes a:

    if positionals:
       self.error(_('too few arguments'))

with a scan for required options that have not been seen. 

Ordinary positionals are required.  But a SubParsersAction is not required.  So we no longer get a warning.

http://bugs.python.org/issue12776 changed this block of code as well.  Notice the 2.7 and 3.2 branches have this 'too few arguments' error, but the default does not.

The default value for Action.required is False.  In _get_positional_kwargs(), a positional's required is set based on nargs (e.g. '+' is required, '*' not).  But add_subparsers() does not use this, so its 'required' ends up False.

This fudge seems to do the trick:

   parser = ArgumentParser(prog='test')
   subparsers = parser.add_subparsers()
   subparsers.required = True
   subparsers.dest = 'command'
   subparser = subparsers.add_parser("foo", help="run foo")
   parser.parse_args()

producing an error message:

   usage: test [-h] {foo} ...
   test: error: the following arguments are required: command

subparsers.dest is set so the error message can give this positional a name.

I'll try to write a patch to do something similar in argparse itself.
msg186532 - (view) Author: paul j3 (paul.j3) * Date: 2013-04-10 21:31
Further observations:

parser.add_subparsers() accepts a 'dest' keyword arg, but not a 'required' one.  Default of 'dest' is SUPPRESS, so the name does not appear in the Namespace. Changing it to something like 'command' will produce an entry, e.g. Namespace(command=foo, ...).  Is this a problem?  

Assuming we have a clean way of assigning a name to 'subparsers', what should it be?  'command', '{cmd}', '{foo,bar,baz}' (like in the usage line)?  This name also could be used when telling the user the subparser choice is invalid (parser._check_value).

This issue exposes a problem with '_get_action_name()'.  This function gets a name from the action's option_strings, metavar or dest.  If it can't get a string, it returns None.

ArgumentError pays attention to whether this action name is a string or None, and adjusts its message accordingly.  But the new replacement for the 'too few arguments' error message does a ', '.join([action names]), which chokes if one of those names is None.  There is a mutually_exclusive_groups test that also uses this 'join'.  This bug should be fixed regardless of what is done with subparsers error messages.

So the issues are:
- making 'subparsers' a required argument
- choosing or generating an appropriate name for 'subparsers'
- passing this name to the error message (via _get_action_name?)
- correcting the handling of action names when they are unknown (None).
msg186695 - (view) Author: paul j3 (paul.j3) * Date: 2013-04-13 06:24
This patch addresses both issues raised here:
- throw an error when the subparser argument is missing
- allow the subparser argument to be optional

argparse.py:

_SubParsersAction -
   add 'required=True' keyword.

   name(self) method - creates a name of the form {cmd1,cmd2} for error messages.

_get_action_name() - try action.name() if it can't get a name from option_strings, dest or metavar.  Still can return None.

2 error cases do a join on a list of action_names.  If a name is None, this will choke.  Add a ['%s'%x for x in list] guard.

test_argparse.py: 
add cases to the subparser block to test the 'required' keyword, and to test the error message changes.

argparse.rst:
add a summary of the add_subparsers() arguments.
msg193956 - (view) Author: paul j3 (paul.j3) * Date: 2013-07-30 21:36
"msg113512 - (view)	Author: Steven Bethard (bethard) 
Seems like there's minimally the bug that argparse should currently throw an error if you add an argument after subparsers (since that argument will never be parsed under the current semantics)."

This isn't quite right.  If the main usage signature is:

usage: PROG [-h] foo {one,two} ... baz

the parser._match_arguments_partial() method will allocate the 1st string to 'foo', the last to 'baz', and pass the rest to the subparser(s).  It doesn't know how many the subparsers can use, but it knows that 'baz' requires one.  From the standpoint of matching argument strings and arguments, a subparser is essentially a '+' positional.

On the other hand if 'baz' (the positional after the subparser) was '*' or '?' it would not get any strings.

If it is possible that subparser(s) doesn't need all the strings passed to it, the user could use 'parse_known_args', and deal with the unparsed strings themselves (possibly with another parser).
msg215894 - (view) Author: paul j3 (paul.j3) * Date: 2014-04-10 17:26
http://stackoverflow.com/questions/22990977/why-does-this-argparse-code-behave-differently-between-python-2-and-3

is answered by this change in how `required` arguments are tested, and how subparsers fell through the cracks.
History
Date User Action Args
2014-04-10 17:26:35paul.j3setmessages: + msg215894
2013-07-30 21:36:27paul.j3setmessages: + msg193956
2013-04-13 06:24:33paul.j3setfiles: + required.patch

messages: + msg186695
2013-04-10 21:31:10paul.j3setmessages: + msg186532
2013-04-09 07:08:43paul.j3setnosy: + paul.j3
messages: + msg186387
2013-04-04 22:16:45seblusetnosy: + seblu
messages: + msg186052
2013-02-10 21:09:48zzzeeksetnosy: + zzzeek
messages: + msg181855
2013-02-07 11:25:21bkabrdasetnosy: + bkabrda
2013-01-26 01:00:45chris.jerdoneksetnosy: + chris.jerdonek
2013-01-18 22:33:03labratsetfiles: + check.py
nosy: + labrat
messages: + msg180229

2012-09-14 17:07:43bewestsetmessages: + msg170491
2012-09-13 19:25:30r.david.murraysetstage: needs patch -> test needed
2012-09-13 19:00:08Juliansetnosy: + Julian
2012-02-20 09:31:22bethardsetmessages: + msg153771
2012-02-18 09:47:43DasIchsetnosy: + DasIch
2012-01-12 16:02:48eric.araujosetfiles: + ed0fce615582.diff
keywords: + patch
2012-01-12 15:58:22eric.araujosethgrepos: + hgrepo102
2012-01-09 16:47:17eric.araujosetmessages: + msg150953
2012-01-09 16:41:00eric.araujosetmessages: - msg150820
2012-01-09 16:40:57eric.araujosetmessages: - msg150815
2012-01-09 16:40:54eric.araujosetmessages: - msg150758
2012-01-07 21:28:06bewestsethgrepos: + hgrepo101
messages: + msg150821
2012-01-07 21:28:05bewestsethgrepos: + hgrepo100
messages: + msg150820
2012-01-07 19:52:21bewestsetmessages: + msg150816
2012-01-07 19:52:20bewestsetmessages: + msg150815
2012-01-07 03:47:47eric.araujosetmessages: + msg150784
2012-01-06 18:08:38bewestsetmessages: + msg150758
2012-01-06 18:08:37bewestsetmessages: + msg150757
2011-12-15 11:31:12bethardsetmessages: + msg149533
2011-09-24 22:52:07bewestsethgrepos: + hgrepo69
messages: + msg144512
2011-09-24 06:55:33bewestsetnosy: + bewest

messages: + msg144484
hgrepos: + hgrepo68
2011-08-29 15:29:24eric.araujosetstage: test needed -> needs patch
versions: + Python 3.3, - Python 3.2
2011-08-22 14:00:32csernazssetnosy: + csernazs
2011-07-01 16:14:31dsullysetnosy: + dsully
2010-11-16 08:04:37bethardsetmessages: + msg121271
2010-11-15 23:23:49G2Psetnosy: + G2P
messages: + msg121250
2010-08-11 01:22:37eric.araujosetmessages: + msg113577
2010-08-11 00:42:54eric.araujosetnosy: + eric.araujo

messages: + msg113575
versions: - Python 2.7, Python 3.3
2010-08-10 21:06:20elsdoerfersetmessages: + msg113558
2010-08-10 09:34:24bethardsetmessages: + msg113512
2010-08-09 22:01:21elsdoerfersetnosy: + elsdoerfer
2010-08-08 14:30:12r.david.murraysetmessages: + msg113267
2010-08-08 14:26:54r.david.murraylinkissue9540 superseder
2010-07-14 18:18:39nviesetmessages: + msg110309
2010-07-13 23:42:39r.david.murraysetstage: test needed
2010-07-13 23:42:26r.david.murraysetpriority: normal -> low
2010-07-13 23:42:09r.david.murraysetnosy: + r.david.murray, bethard
messages: + msg110244
2010-07-13 21:30:38nviesetmessages: + msg110232
title: optional subparsers -> argparse: optional subparsers
2010-07-13 21:29:47nviecreate