classification
Title: argparse help formatter crashes
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Endre, berker.peksag, bethard, paul.j3, xiang.zhang
Priority: normal Keywords:

Created on 2016-05-04 15:14 by Endre, last changed 2018-09-23 03:26 by paul.j3.

Messages (8)
msg264823 - (view) Author: Endre (Endre) Date: 2016-05-04 15:14
I am using argparse-1.4.0.

For this code exception is thrown when the script is started with the -h option:
http://pastebin.com/dFF1paFA

This exception is thrown:
Traceback (most recent call last):
  File "C:\Users\ebak\Picea\tools\buildsystem\python\kw_set_checkers.py", line 129, in <module>
    args = parser.parse_args()
  File "C:\Python27\lib\argparse.py", line 1688, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "C:\Python27\lib\argparse.py", line 1720, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "C:\Python27\lib\argparse.py", line 1926, in _parse_known_args
    start_index = consume_optional(start_index)
  File "C:\Python27\lib\argparse.py", line 1866, in consume_optional
    take_action(action, args, option_string)
  File "C:\Python27\lib\argparse.py", line 1794, in take_action
    action(self, namespace, argument_values, option_string)
  File "C:\Python27\lib\argparse.py", line 994, in __call__
    parser.print_help()
  File "C:\Python27\lib\argparse.py", line 2327, in print_help
    self._print_message(self.format_help(), file)
  File "C:\Python27\lib\argparse.py", line 2301, in format_help
    return formatter.format_help()
  File "C:\Python27\lib\argparse.py", line 279, in format_help
    help = self._root_section.format_help()
  File "C:\Python27\lib\argparse.py", line 209, in format_help
    func(*args)
  File "C:\Python27\lib\argparse.py", line 317, in _format_usage
    action_usage = format(optionals + positionals, groups)
  File "C:\Python27\lib\argparse.py", line 388, in _format_actions_usage
    start = actions.index(group._group_actions[0])
IndexError: list index out of range
msg264834 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-05-04 16:14
I don't think add_mutually_exclusive_group and add_argument_group are designed to work together. Did this worked with argparse 1.3.0 or stdlib version of argparse before? I'm getting the same traceback in 2.7 and 3.4+.
msg264835 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016-05-04 16:15
This error occurs due to there is no action belong to the mutexGroup, the code requires at least one. You can simply add

    mutexGroup.add_argument('--foo', action='store_true')

and you'll see the error disappears. I have no idea that this behaviour is intended or not.
msg264837 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016-05-04 16:23
I agree with Berker. Even without any exceptions, the help message given is not what add_argument_group and add_mutually_exclusive_group intend.

You can see

usage: argparse_test.py [-h] [--foo] -u URL -p PROJECT [--dump] --mergeInput
                        MERGEINPUT [--removeDisabled] -c CHECKERS

Sets the checkers of a klocwork project

optional arguments:
  -h, --help            show this help message and exit
  --foo
  -c CHECKERS, --checkers CHECKERS
                        File which lists the checkers to be enabled.

There is not group name and still optional arguments. And the command line does not show exclusive. I believe they are not designed to work like this.
msg264839 - (view) Author: Endre (Endre) Date: 2016-05-04 16:28
Hi,

Thanks for taking a look at it.
I have only tried it with argparse-1.4.0.

"I don't think add_mutually_exclusive_group and add_argument_group are designed to work together."

Yes, it really works strange in this case. I guess the framework should raise an exception which states that this use case is not supported, e.g. at the "serverGroup = mutexGroup.add_argument_group('serverGroup')
" line.
msg265164 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2016-05-08 20:22
Argument Groups are not designed for nesting, and despite their names and subclassing, Mutually exclusive groups and Argument Groups are not meant to be used together (with one exception).

I agree that the error is obscure, but it occurs in a particularly fragile function in the formatter, '_format_actions_usage'.  That function needs a major rewrite (that's in another bug/issue).

Argument Groups serve only as a way of grouping help lines.  Mutually exclusive groups test arguments during parsing, and add some markings to the usage line.  So they have very different purposes.

I have seen questions on Stackoverflow where people try to use Argument Groups as a way of adding some sort of subgroup to the Mutually Exclusive one, one for example that implements a 'allow any of this group' logic.  There is a bug/issue asking for 'inclusive' nesting groups, but production patch of that sort is long ways off.  http://bugs.python.org/issue11588

This usage line

    (--url URL --project Prj [--dump]) | (--mergeInput input.txt [--removeDisabled])

suggests that this what you are trying do - allow any of -u,-p,-d, but disallow one of these with -m or -r.  That logic is beyond the current group testing mechanism, and beyond the usage formatting code.  You'll have to do your own tests, and write a custom usage line.

Mutually exclusive groups can be nested in other mutual groups, but the effect isn't what you might hope.  It just forms a larger mutually exclusive group; there's no subgrouping.

It is possible to nest a mutually exclusive group in an Argument group; the effect is to give the mutually exclusive group a title.
msg326127 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2018-09-23 03:14
If I add an argument to the mutexGroup as suggested by xiang

In [7]: mutexGroup.add_argument('--foo', action='store_true')
Out[7]: _StoreTrueAction(option_strings=['--foo'], dest='foo', nargs=0, const=True, default=False, type=None, choices=None, help=None, metavar=None)
In [8]: parser.print_usage()
usage: ipython3 [-h] -u URL -p PROJECT [--dump] --mergeInput MERGEINPUT
                [--removeDisabled] -c CHECKERS --foo
In [9]: parser.print_help()
usage: ipython3 [-h] -u URL -p PROJECT [--dump] --mergeInput MERGEINPUT
                [--removeDisabled] -c CHECKERS --foo

Sets the checkers of a klocwork project

optional arguments:
  -h, --help            show this help message and exit
  -c CHECKERS, --checkers CHECKERS
                        File which lists the checkers to be enabled.

the argument_group arguments show up in the usage, but without mutually_exclusive markings.  But they don't show up in the help lines.  I suspect all those group arguments will be tested as one mutually exclusive group, not two subgroups.

If I put the argument groups in the main parser, and omit the mutually_exclusive group I get this help:

In [11]: parser.print_help()
usage: ipython3 [-h] -u URL -p PROJECT [--dump] --mergeInput MERGEINPUT
                [--removeDisabled] -c CHECKERS

Sets the checkers of a klocwork project

optional arguments:
  -h, --help            show this help message and exit
  -c CHECKERS, --checkers CHECKERS
                        File which lists the checkers to be enabled.

serverGroup:
  -u URL, --url URL     klocwork server URL.
  -p PROJECT, --project PROJECT
                        klocwork project name.
  --dump                Dump the current checkers config.

mergeGroup:
  --mergeInput MERGEINPUT
                        Input file to merge for the '--checkers' file. Format:
                        checkerName Enabled|Disabled
  --removeDisabled      Disabled checkers will be removed from the '--
                        checkers' file.

The argument groups appear as intended in the help lines.

add_argument_group is inherited from _ActionsContainer.  The parser ArgumentParser also inherits this method.  A possible fix is to override this method in the _MutuallyExclusiveGroup class to raise some sort of not-implemented error.

The documentation, as written, does not suggest or hint that an argument_group can be added to anything but a parser.  But a  stronger disclaimer could be added.
msg326129 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2018-09-23 03:26
https://bugs.python.org/issue29553 deals with a similar problem, the usage display when one mutually exclusive group is embedded in another mutually exclusive group.  In that case, usage is displayed, but with some missing brackets.  

As here there are two issues - the usage formatter is brittle, and groups are not designed for nesting.
History
Date User Action Args
2018-09-23 03:26:59paul.j3setmessages: + msg326129
2018-09-23 03:14:51paul.j3setmessages: + msg326127
2018-07-11 07:43:28serhiy.storchakasettype: crash -> behavior
2016-05-08 20:22:45paul.j3setnosy: + paul.j3
messages: + msg265164
2016-05-04 16:30:54xiang.zhangsetnosy: + bethard
2016-05-04 16:28:41Endresetmessages: + msg264839
2016-05-04 16:23:49xiang.zhangsetmessages: + msg264837
2016-05-04 16:15:10xiang.zhangsetmessages: + msg264835
2016-05-04 16:14:41berker.peksagsetnosy: + berker.peksag
messages: + msg264834
2016-05-04 16:01:22xiang.zhangsetnosy: + xiang.zhang
2016-05-04 15:14:47Endrecreate