Shoudl argparse be included in the Python Standard Library.

I know we already have getopt and optparse but optparse doesn't
support many features easily. (like options without hyphen, nargs=*)

Here a little about argparse:


argparse: Python command line parser
The argparse module makes writing command line tools in Python easy.
Just briefly describe your command line interface and argparse will take
care of the rest, including: 
parsing the arguments and flags from sys.argv 
converting arg strings into objects for your program 
formatting and printing any help messages 
and much more ... 
For those familiar with the optparse module from the Python standard
library, argparse improves on this module in a number of ways, including: 
handling positional arguments 
supporting sub-commands 
allowing alternative option prefixes like + and / 
handling zero-or-more and one-or-more style arguments 
producing more informative usage messages 
providing a much simpler interface for custom types and actions

One important prerequisite for including it is that the author of the
code contributes it for inclusion (irrespective of any license that may
allow us to fork the code), and that he, or somebody else, volunteers to
maintain it.

IIUC, you are not the author, so I have to reject this request.

If you find that some features are lacking in optparse, please provide
patches to enhance it.

From the author, Steven Berthard:
====================
Sorry about the delay - conferences and moving means that I haven't
had as much time for argparse as I'd like. Basically, the way things
get into the Python standard library is if a large enough community
requests it. I'm happy to contribute (and support) it if there's
enough demand, though since Python 3.1b1 is already out, the earliest
releases that might include it would be Python 2.7 and 3.2.
====================


From the conversation I have had with the author, he does seem to be
interested in it. I'll ask the author to respond to this bug report.

I hope you read this thread which discusses the same issue.

* http://mail.python.org/pipermail/python-list/2007-January/592646.html

MvL,effbot and jjlee had shared their views to the original author of
argparse module. It did not go anywhere from there.

I see that this issue might be rejected unless followed the steps as
mentioned in that mail thread.

Thanks for the link. As a user, I see many of the features that argparse
brings, to be helpful.

Since they are missing in optparse, and since it doesn't look like
argparse will be included, should I open new bugs for those features
against optparse ?

> Since they are missing in optparse, and since it doesn't look like
> argparse will be included, should I open new bugs for those features
> against optparse ?

I think chances are very low that any such bug report would be reacted
on within the next decade.

I'm happy to contribute argparse to the standard library and volunteer
to maintain it.

For what it's worth, I don't agree that there are already too many
argument parsing libraries in the standard library. I do agree that
there are already too many *option* parsing libraries. But both getopt
and optparse don't know how to parse positional arguments at all. For
example, if your program usage looks like:

    prog eggs [spam] [baz [baz [...]]]

then getopt and optparse are worthless to you - they'll just return
sys.argv[1:] since there are no options. The argparse module, on the
other hand, can correctly parse out "eggs", "spam" and the "baz" list
into appropriate attributes (in addition to doing all the other stuff
that optparse does).

I also don't think there's much of a chance of optparse ever growing
most of the argparse features. When I started argpasre, my goal was
exactly that - to keep the module fully backwards compatible with
optparse and just to add the missing features. The optparse code just
wasn't written in a way that allowed me to do that. In particular, the
optparse extension API is horribly designed, and exposes so many
internals of optparse that it's nearly impossible to add any new
features to optparse without breaking this.

Steven Bethard wrote:
> In particular, the
> optparse extension API is horribly designed, and exposes so many
> internals of optparse that it's nearly impossible to add any new
> features to optparse without breaking this.

It would be useful if several people could confirm that argparse
is *not* horribly designed.

I'm not sure about the design part, but as a user, I find both to have very
similar interfaces.

argparse is better because it handles nargs=*. This has long been
asked in optparse. Positional arguments is something I wanted recently,
and argparse makes it very easy to use that too.

On Tue, Jun 9, 2009 at 11:45 PM, Martin v. Löwis<report@bugs.python.org> wrote:
> It would be useful if several people could confirm that argparse
> is *not* horribly designed.

A totally reasonable request. Anyone willing to evaluate this can take
a look at:

http://argparse.googlecode.com/svn/trunk/argparse.py

It also may help to look at the extensive test suite:

http://argparse.googlecode.com/svn/trunk/test/test_argparse.py

If anyone has specific questions about how something works or how to
extend argparse with new functionality, I'm happy to answer those
questions.

I'll take a look at it. I've been meaning to use argparse, anyway.

It's unfortunate (at least to me, but I know I have a skewed view of
this) that the help string in add_argument uses %-formatting when
formatting the result. I'd much rather it use str.format().

I see a couple of options:
- leave it as-is and live with %-formatting forever
- add a different, mutually exclusive parameter,
  like help_ex or help_str
- try to guess whether to use %-formatting or str.format,
  based on inspecting the help string

I'm not really wild about any of these.

I'm still looking through the code, slowly.

Yeah, the % formatting is an artifact of argparse being around before
str.format was available. If argparse became part of the standard
library, it might be reasonable to change to str.format formatting
instead as part of the move.

It would mean that folks would have to modify their code to switch from
the current argparse module to the stdlib one, but maybe that's an
acceptable tradeoff. I'm certainly happy to change things to str.format
if that's agreed on.

Thanks again for looking at this.

I would also like to voice support for including argparse in the
standard library.  It seems silly to deny a module from being added just
because we already have two inferior ones.  Argparse adds so many new
(and badly needed!) features that it would be very difficult to make a
backwards-compatible wrapper around ArgumentParser.  It would be much
easier, and less error-prone, to leave optparse as-is and just add this
module in addition.

If the problem is having a third module clutter up the module list,
here's a simple solution: rename the existing optparse to _optparse,
rename argparse to optparse, and in (the new) optparse, from _optparse
import *.  Done.  Now you have a backwards-compatible OptionParser
(which ought to be deprecated), and the new ArgumentParser.  Would this
solution work?

we should never pretend an old module doesn't exist.  leave optparse as
optparse.  argparse can come in under its own name.

we can mark getopt and optparse as deprecated at some point and remove
them in 10 years :)

+1 on inclusion btw.  It looks like it'll obsolete a bunch of crappy
argument parsing code that exists in some projects i work on but I
haven't had time to clean them up and switch.

I'm +1 on inclusion one way or another as well.  I haven't made time to
do anything other than a cursory code review, but as an end user I find
the module fits my needs much better than optparse.

This was rejected prior to Steven Bethard becoming involved, so I'm
reopening.

+1 from me - argparse is a great module to use.

Only +0 purely because I haven't used argparse myself yet.

Otherwise I would probably be +1, since I have several scripts that have
fairly kludgy hacked together optparse based approaches to handling
positional arguments, subparsers and building new parsers that accept
the superset of options defined in existing parsers. The feature
comparison between argparse and optparse makes it sound like argparse
does a much better job of supporting these use cases.

The reasons I don't use argparse for them are that: a) I didn't know it
existed until recently; and b) the scripts are in an environment where
getting approval to use new third party modules is something of a pain.

Why does this have to go into the standard library? People that want to
use it can still install it from PyPI. -sys.maxint from me.

@Armin: Doesn't that argument apply to *any* library proposed for
inclusion in the standard library? By which logic we should never add
anything to the standard library ever again. Surely you don't mean that,
so could you be more specific on what you think is particularly
inappropriate about argparse?

> @Armin: Doesn't that argument apply to *any* library proposed for
> inclusion in the standard library? By which logic we should never add
> anything to the standard library ever again.

That's what I say. Do not add anything to the stdlib that is not needed
to keep applications platform independent. argparse adds zero value to
x-platform development.

Things I consider good for a stdlib:

  * portable filesystem notification hooks
  * stuff like os.path
  * distutils
  * socket library

Stuff that should not go into the stdlib:

  * xml parsers
  * command line parsers (one is enough, and that should be
    *stable* not replaced later with something like argparse)

Fix packaging and do not dump useless stuff into the standard library to
make it appear more modern. Decentralization is modern, not replacing
modules in the stdlib.

[snip]
> Fix packaging and do not dump useless stuff into the standard library to
> make it appear more modern. Decentralization is modern, not replacing
> modules in the stdlib.

I can respect that viewpoint. So what do you propose to do with
existing modules like optparse that aren't required to make platform
independent applications and are out of date and basically
unmaintained? One option would be to remove them, but that's probably
too drastic. Another option would be to add documentation to them
warning that they're out of date and pointing users to alternative
external libraries.

> I can respect that viewpoint. So what do you propose to do with
> existing modules like optparse that aren't required to make platform
> independent applications and are out of date and basically
> unmaintained? One option would be to remove them, but that's probably
> too drastic.

Documentation and a new kind of deprecation warning.  It's
documentation-deprecated in one version, one later a real deprecation
warning appears that sticks around for a couple of versions.  The
documentation would explain how to hide the deprecation warning and
tells the user to better use more modern alternatives.

This of course requires packaging to work flawlessly first which I
consider to be high priority.

I disagree.  I think option and argument parsing belongs in the standard
(batteries included) library, regardless of how well external package
management works.  

argparse has the advantage over optparse that it has an active
maintainer.  What we do about the stuff in the stdlib that is no longer
maintained is a different discussion, IMO.

Command line parsing is a basic need, including amongst other standard
library modules. argparse has many advantages over optparse (not the
least of which is that it has an active maintainer). Several of these
features *can't* be added to optparse whilst maintaining backwards
compatibility, which is what prompted Steven to create argparse in the
first place.

Improvements that I am aware of include:

- handling of standard Windows way of specifying options
- sub-commands
- handling of positional arguments

I had to implement my own technique for handling a sub-command in recent
unittest changes that would have been much simpler if argparse were in
the standard library.

It must be convenient to operate in an environment where you can install
new software so easily Armin.

For those of us that operate in environments where every new piece of
software has to go through contracts review to ensure that we can both
license it for our own purposes and also subsequently transfer those
licenses to our customers, having things in the standard library is a
*huge* benefit.

This is the usual batteries included vs better packaging argument. For
developers in an environment where adding new packages is a low overhead
operation, adding something to the stdlib isn't a big gain for them
while better packaging tools are awesome. For others (including me), the
actual package installation is the least of our hassles and anything
that helps us avoid dealing with the lawyers is a big gain.

> It must be convenient to operate in an environment where you can 
> install new software so easily Armin.

Trust me, it is.

> For others (including me), the actual package installation is the
> least of our hassles and anything that helps us avoid dealing with
> the lawyers is a big gain.

So you're suggesting Python should suffer because some companies have a
weird legal department?

@Michael Foord: I totally agree that argument parsing is something that
*should* be in the standard library because everybody needs it.  However
at the same time it's something I could imagine comes from an external
source.  I would rather maintain optparse myself for python.org than
seeing another argument parsing library in the stdlib so that everybody
has to switch over because the Python devs did not find someone to
maintain it.  The stdlib is very often unmaintained for large parts; we
can't just replace a module because the developer got bored...

I think optparse just got a new maintainer.

He has a few important feature requests to deal with as well.

On Mon, Sep 14, 2009 at 11:15 AM, Armin Ronacher <report@bugs.python.org> wrote:
>
> Armin Ronacher <armin.ronacher@active-4.com> added the comment:
>
>> @Armin: Doesn't that argument apply to *any* library proposed for
>> inclusion in the standard library? By which logic we should never add
>> anything to the standard library ever again.
>
> That's what I say. Do not add anything to the stdlib that is not needed
> to keep applications platform independent. argparse adds zero value to
> x-platform development.

By that logic we should remove getopt and optparse,

>
> Things I consider good for a stdlib:
>
>  * portable filesystem notification hooks
>  * stuff like os.path
>  * distutils
>  * socket library
>
> Stuff that should not go into the stdlib:
>
>  * xml parsers
>  * command line parsers (one is enough, and that should be
>    *stable* not replaced later with something like argparse)

One is only enough if its a useful one.  argparse can be that one.
optparse and getopt are both sorely lacking.  If anything deprecate
getopt and optparse so that they're gone in 3.4.

By your argument we shouldn't even have one command line parser
because it does nothing to cross platform support.

Please DO NOT drag an issue asking to add a useful library to the set
of batteries included into the packaging decentralization flamewar.
There are is a large python user base that can never code that does
not come as part of python itself for one or more of legal and
technical reasons.  This issue is not the place to debate how stupid
anyone thinks that concept is.

Armin; if you are serious in wanting to help out with the stdlib and core 
work, feel free to help us discuss this over on the stdlib-sig 
(http://mail.python.org/pipermail/stdlib-sig/2009-September/000398.html) 
or help commit patches and fixes for all of the modules that are growing 
long in the tooth and need help.

Argparse seems to be overloaded with rarely used features. Instead of providing 
API to add these features and allow users copy examples it tends to be an all-in-
one solution that is hard to use due to abundance of specific parameters.

Look at constructor, for example - 
http://argparse.googlecode.com/svn/tags/r101/doc/ArgumentParser.html#usage
{{{
description - Text to display before the argument help.
epilog - Text to display after the argument help.
version - A version number used to add a -v/–version option to the parser.
add_help - Add a -h/–help option to the parser. (default: True)
argument_default - Set the global default value for arguments. (default: None)
parents - A list of :class:ArgumentParser objects whose arguments should also be 
included.
prefix_chars - The set of characters that prefix optional arguments. (default: ‘-
‘)
fromfile_prefix_chars - The set of characters that prefix files from which 
additional arguments should be read. (default: None)
formatter_class - A class for customizing the help output.
conflict_handler - Usually unnecessary, defines strategy for resolving conflicting 
optionals.
prog - Usually unnecessary, the name of the program (default: sys.argv[0])
usage - Usually unnecessary, the string describing the program usage (default: 
generated)
}}}

the only useful arguments by default is 'description' and 'add_help' (which is 
better to see as 'no_help' to turn off default behaviour). 'version' is not 
useful, because it adds '-v' shorthand that is often used for verbosity. 
'prefix_chars' is not useful, because the only sense one may need it is to provide 
windows conventions like '/longarg' for '--longarg' and not '//longarg', but it 
doesn't allow to do so.

Everything else is constructor bloat, and even with such abundance of options it 
still unable to generate usage help prefixed by program name and version - the 
sole feature I miss from optparse.

-1 for now

@techtonik: If you have a specific feature request for argparse, I
recommend that you file an issue on the argparse tracker[1]. I assure
you that despite the fact that you only have need for a couple of the
constructor parameters, the rest exist because people have asked for
them. Fortunately, since they're keyword parameters, you only have to
specify the ones that you care about.

[1]http://code.google.com/p/argparse/issues/list

PEP 389 has been accepted.