optparse is a great option parser, but one thing that would make it even
greater would be if it provided a standard option (cf. --help) which
lists all the available options in a parseable form. Something prefixed
with --help, e.g. --help-options would be ideal since it doesn't clutter
the option namespace.

This would provide a simple command-line hook for e.g. bash completion
customisation with complete/compgen, which could then easily and
maintainably obtain the list of available switches via the
--help-options flag rather than hard-coding the option names or
attempting to grep the output of --help

It would also be good if the OptionParser provided a simple Python API
way to obtain the names of all option switches, rather than having to
loop over OptionGroups, calling the unadvertised 'option_list' and
'get_option_name' methods!

Are you saying, that for example for this:

gruszczy@gruszczy-laptop:~/Programs/logbuilder$ ./logbuilder --help
Usage: logbuilder [options] repo

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -r REGEXP, --regexp=REGEXP
                        filter revisions using regular expression
  -c CONTAINS, --contains=CONTAINS
                        filter revisions that doesn't contain given string
  -s START_REV, --start=START_REV
                        first revision to be used in log
  -e END_REV, --end=END_REV
                        last revision to be used in log
  -f FILE, --file=FILE  file where result will be stored
  -t TEMPLATE, --template=TEMPLATE
                        template used to display changes
  -p PURGE, --purge=PURGE
                        remove parts of a commit messages, that match given
                        regexp

you would like to get:

gruszczy@gruszczy-laptop:~/Programs/logbuilder$ ./logbuilder --help-options

  --version
  -h, --help
  -r, --regexp
  -c, --contains
  -s, --start
  -e, --end
  -f, --file
  -t, --template
  -p, --purge

?

That sort of idea, yes: just a wild thought, but it would be really nice if this was available so that in combination with a standard bash/zsh function, getting basic automatic command completion for scripts built with optparse (and any other implementer of such a scheme) was as simple as adding

complete -F _optparse -o default mycmdname

to the completion script library.

The simple scheme you laid out seems fine to me, but in the best bikeshedding tradition it would be useful to distinguish between options which take an argument and those which don't, pre-empt the need for a format version, and make the parsing even easier by removing cosmetic whitespace, commas etc.:

gruszczy@gruszczy-laptop:~/Programs/logbuilder$ ./logbuilder --help-options
#OPTPARSE_FORMAT 0
--version
-h --help
-r= --regexp=
-c= --contains=
-s= --start=
-e= --end=
-f= --file=
-t= --template=
-p= --purge=

Maybe this is just a pipe-dream, but the need to hand-write basic completion scripts seems so unnecessary, just for lack of any (even de-facto) standardisation. As optparse already enforces/encourages many good habits and conventions, some system like this would further help the integration with shell completion.

Or maybe the existing --help output is good enough for a rather more fiddly standard bash completion parsing function. I've tried writing one of these, but it would hard for it be generally robust since the descriptive strings can contain any structure that they feel like, and could hence mess up the pattern-matching. I'm very happy if someone can out-sed me and make that work!

I'll take a look at optparse code and try to provide a patch. But first holidays must finish and I must come back to ma usual residence, where I have programming environment.

Please target argparse rather than optparse, or better yet in addition to optparse.  And I'm +1 for making it easier to write completion scripts.

Someone pointed this out to me earlier:

http://pypi.python.org/pypi/genzshcomp

I believe it's trying to solve the same problem, and claims to work with both optparse and argparse, so it might be worth looking into what it's doing and seeing if there's a useful patch that could be proposed for argparse.

Thanks for the pointers to both of these... I wasn't aware of either. I see argparse has been recently approved for Python stdlib inclusion, too: http://www.python.org/dev/peps/pep-0389/ Congratulations!

As far as I can tell, genzshcomp is parsing the output of the help command to reverse-engineer what the allowed flags should be. Assuming that only one space occurs between the arg and its metavar, this should work 99% or the time... I'm not sure if there is any attempt to be clever when the formatting is ambiguous. But given that the opt parser already contains the structured information, life can be made easier by writing out a more readily parseable format.

Here's an example bash parser function and its usage, for a further-simplified form of the above format where each arg (long or short) gets a line of its own and the arguments are indicated by a separate word as in the current output:

Example input:
$ foo --help-options
#OPTPARSE_FORMAT 0
--version
-h
--help
-r REGEXP
--regexp REGEXP
-s N
--start N
-e M
--end M
-f FILE
--file FILE

and the parser/completion functions:

function _optparse_getargs() {
	local opts cur prev
	COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"

    PREVIFS=$IFS
    IFS=$'\n'
    for line in `$1 --help-options | egrep '^-'`; do
        opt=`echo $line | sed 's/^\(-[^ ]\+\).*$/\1/'`
        argeq=`echo $line | sed 's/^--[^ ]\+ \([A-Za-z0-9]*\)$/=/'`
        if [[ $argeq != "=" ]]; then argeq=""; fi
        opts="$opts $opt$argeq";
    done
    IFS=$PREVIFS
    unset PREVIFS
    opts=`echo $opts | sed -e 's/^ *//' -e 's/ *$//'`

    if [[ ${cur} == -* ]] ; then
        COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )        
        if test -n "$COMPREPLY"; then
            return 0
        fi
    fi

    return 0
}


function _foo() {
    _optparse_getargs foo
    return 0
}

complete -F _foo -o default foo

Patch for optparse with tests. If it's ok, I'll sit down to argparse.

Sorry, what does "I'll sit down to" mean? Does that mean you're offering to try to do the argparse patch too? Or that you'd rather someone else do it? (Either one's fine - I just couldn't tell which you meant.)

I guess I am using my English too little, that's why I am using polish expressions too often. What I meant was of course, that I will do argparse patch too.

I haven't provided docs for --help-options yet, becuase it is not clear to me, it is an accepted feature. When it is, I will update the patch.

Ok, here comes patch for argparse too.

Thanks for the patch! One concern I have is that adding --help-options by default has the potential to break existing code, e.g. if someone using optparse or argparse was already defining their own --help-options flag. The backward compatible solution is to have --help-options disabled by default, and ask people to enable it with add_interface=True.

Comments on the argparse patch: I think it's probably overkill to create InterfaceFormatter - just do the appropriate formatting in the _InterfaceAction. I also wouldn't add format_interface or print_interface until someone requests them. Last nit: don't add the takes_value method, just inline your "self.nargs != 0" check in the one place you need it.

> The backward compatible solution is to have --help-options disabled by default, and ask people to enable it with add_interface=True.

Or to add the option just before arg parsing, if it has not already been defined?

Thanks for the patches, Filip!

Andy

I prefer an approach that allows this option to be defined by default, since if it is not defined by default it defeats part of the purpose of having the option.  The program author may not be concerned with completions (or even know about them), but if the option is defined by default then even the programs of those authors can be auto-completed by the generic script.

Removing 2.7 since it is now in feature freeze.

On Wed, Apr 21, 2010 at 12:36 AM, Andy Buckley <report@bugs.python.org> wrote:
> Or to add the option just before arg parsing, if it has not already been defined?

Something like this was suggested before and it doesn't really work
out well. It means the first time you call .parse_args(), your options
get modified. So if you do anything with the parser before
.parse_args() -- for example, calling .print_help() -- then you don't
get the right options.

On Wed, Apr 21, 2010 at 5:12 AM, R. David Murray <report@bugs.python.org> wrote:
> I prefer an approach that allows this option to be defined by default

I agree that it would be best if all command line utilities supported
this by default[1]. I'm just not sure how to do it in a backwards
compatible way. The fact that the most recent patch against argparse
has to modify so many test cases suggests that it's going to have
unexpected consequences for a bunch of users.

[1] Though I'd feel more confident in that belief if someone could
point me to what the output of other programs that do this looks like
so that I could see we were following a standard somewhere.

Steve
-- 
Where did you get that preposterous hypothesis?
Did Steve tell you that?
        --- The Hiphopopotamus

Hello

Here’s another approach, which has to be used explicitly but provides much more flexible completion: http://pypi.python.org/pypi/optcomplete

I ask the author some time ago if he’d adapt it to argparse; he answered he wouldn’t have time but it shouldn’t be too hard. It’s 221 lines of code (according to sloccount), perhaps worth merging into argparse.

Regards

I don't see why --help-options would need to be listed in help.  We could pick a more obscure name, too.  The point of this option is to support tools, not users.

I'll be happy to both fix things pointed by Steven and try some other approach, if that's required, but I would rather do it after a consesus is reached, so I don't have to do the same stuff several times (changing argparse tests requires some work - it's really awesome test suite).

What about optparse? Maybe there we can add this option without much thinking, by checking if help-options is used at the end of parsing? If not, we face the same problem as with argparse.

I don't really understand, why can't we just check if help-options is provided by the user and add our own, if it is not? Even if options are parsed several times (when doing some debugging I have seen my messages printed multiple times in help formatter), we should be able to inspect options in raw form and determine, whether --help-options is used.

2010/4/21 Filip Gruszczyński <report@bugs.python.org>:
> I don't really understand, why can't we just check if
> help-options is provided by the user and add our own,
> if it is not?

I'm sure it would be possible to do it this way. The question is
whether it makes sense to from the perspectives of code
maintainability and explaining awkward corner cases to users.

On Wed, Apr 21, 2010 at 12:29 PM, R. David Murray wrote:
> I don't see why --help-options would need to be listed in help.

Right. For argparse, suppressing the printing of --help-options in the
help message is as simple as setting help=SUPPRESS.

>  We could pick a more obscure name, too.

I think this is probably the best way forward. What is the format
that's being printed out? Is this a standard somewhere? Can we name
the flag something like "--print-parser-options-in-XXX-format"?

Steve

An obscure name reusing terms like “compword” that can be found easily in Python docs and Bash completion docs would be best.

On Wed, Apr 21, 2010 at 12:45 PM, Éric Araujo <report@bugs.python.org> wrote:
> An obscure name reusing terms like “compword” that can be found easily in Python docs and Bash completion docs would be best.

Seems sensible. Does anyone know if zsh uses the same or a different
mechanism? If possible, we should be producing output that either of
these could use.

Steve
-- 
Where did you get that preposterous hypothesis?
Did Steve tell you that?
        --- The Hiphopopotamus

So, is there any decision here, so that I could get down to providing better patch?

2010/5/9 Filip Gruszczyński <report@bugs.python.org>:
> So, is there any decision here, so that I could get down to providing better patch?

I guess I'd like to hear from someone about how these things work in
zsh. If we're going to add a hidden flag to *all* parsers, we should
really make sure it's compatible/useful for as many of the shells that
support this kind of autocomplete as possible...

zsh's completion system is completely programmable.  I looks like it would be pretty easy to add generic 'python script' support widgets(*) using this hidden option, and probably other neat tricks as well.  Something that would make it even more useful for zsh completion would be to include information on the type of the argument when known.  A zsh completer could then be programmed to do smart completion on the option value as well.

(*) You can write a 'widget' and assign it to a key, and then when you use that key the completion widget (shell function) is called and could run the command with the hidden option to get the option info and generate the completion list.  That's just the *easiest* way to integrate support for this into zsh completion.

Hello Filip.  Could you give us a status update on this patch?

zsh completion is much more powerful. E.g. for git<SP>log<SP><TAB> I see:
completing head
<list-of-heads>
completing commit object name
completing cached file
abspath.c                   git-lost-found.sh           README                    
aclocal.m4                  git-merge-octopus.sh        reflog-walk.c             
...

git<SP><TAB>
completing alias
diffab                    -- alias for 'diff --src-prefix=a/ --dst-prefix=b/'
lol                       -- alias for 'log --graph --decorate --pretty=oneline --abbrev-commit'
lola                      -- alias for 'log --graph --decorate --pretty=oneline --abbrev-commit --all'
mergeu                    -- alias for 'merge --ff-only @{u}'
completing main porcelain command
add                       -- add file contents to index
am                        -- apply patches from a mailbox
...

The header parts ('completing commit object name', 'completing head',
'completing cached file') are in bold red. So different completions
types are supported, and some help is provided, and completions
are split in groups.

Completion for options knows which short/long options go together:
git log -<TAB> prints:
...
--oneline                                          -- shorthand for --pretty=oneline --abbrev-commit    
--ours                    -2                       -- diff against "our branch" version                 
--parents                                          -- display parents of commit                         
--patch                   -u           -p          -- generate diff in patch format                     
...

fish ("a friendly interactive shell" which I don't use but
which has some very cool features) prints something like
a\%b                                                                                         (Branch)
abspath.c                                                                      (C source code, 4.2kB)
abspath.o                                                                         (Object code, 13kB)
aclocal.m4                                                                          (M4 macro, 1.4kB)
adres                                                                                     (File, 23B)
advice.c                                                                       (C source code, 2.4kB)
advice.h                                                                             (C header, 555B)

I think that for --help-options to be usefull, it should list more information
than is needed just for bash completion. At least:
- options, with long and short options specified together
- short help for options (or maybe all of the help)
- option groups if such are used
- metavar names

This last part could be used by the completion script to customize completions
for a specific program. E.g. the completion script could know that FILE means a file,
and HOST means a host name.

So it seems like what bash needs and what zsh needs are pretty different. My feeling so far is that if there isn't "one true format" that argparse can produce and be useful to a wide variety of shells, then it's probably not functionality that belongs in Python core, and instead belongs on PyPI.

So I guess my recommended next step would be to have someone offer help to the maintainer of http://pypi.python.org/pypi/optcomplete to update it to support argparse as well. If and when optcomplete supports argparse, bash, zsh and whatever other common shells people are using, and when optcomplete has significant usage in the field, then we can consider integrating it into the Python stdlib.

> zsh completion is much more powerful.
I beg to differ :)  bash completion can also list more that files, for example only .bz2 files when I complete the bunzip2 command, or Mercurial branch and tag names when I complete hg update, etc.  It all depends on the completion script.

I think that there may be enough common ground between the two shells that argparse could print out enough information for both systems.  I haven’t read the code of genzshcomp, though.

ZSH can just present it in a prettier way, and also includes slightly
more info (the short explanations, ordering).

 > could print out enough information for both systems.
Exactly.

ZSH can use bash completion, but then it doesn't display the extra info. 
It would be nice to keep those optional features in mind to avoid 
limiting the exported information to that useful for bash.

This issue is 11 years old has a patch (with 2 versions): it's far from being "newcomer friendly", I remove the "Easy" label.