classification
Title: argparse help formatter raises IndexError
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Endre, berker.peksag, bethard, iritkatriel, kapsh, miss-islington, paul.j3, xiang.zhang
Priority: normal Keywords: patch

Created on 2016-05-04 15:14 by Endre, last changed 2021-12-15 12:25 by iritkatriel. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 30099 merged iritkatriel, 2021-12-14 13:16
PR 30114 merged miss-islington, 2021-12-15 10:08
PR 30115 merged miss-islington, 2021-12-15 10:08
Messages (13)
msg264823 - (view) Author: Endre (Endre) Date: 2016-05-04 15:14
I am using argparse-1.4.0.

For this code exception is thrown when the script is started with the -h option:
http://pastebin.com/dFF1paFA

This exception is thrown:
Traceback (most recent call last):
  File "C:\Users\ebak\Picea\tools\buildsystem\python\kw_set_checkers.py", line 129, in <module>
    args = parser.parse_args()
  File "C:\Python27\lib\argparse.py", line 1688, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "C:\Python27\lib\argparse.py", line 1720, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "C:\Python27\lib\argparse.py", line 1926, in _parse_known_args
    start_index = consume_optional(start_index)
  File "C:\Python27\lib\argparse.py", line 1866, in consume_optional
    take_action(action, args, option_string)
  File "C:\Python27\lib\argparse.py", line 1794, in take_action
    action(self, namespace, argument_values, option_string)
  File "C:\Python27\lib\argparse.py", line 994, in __call__
    parser.print_help()
  File "C:\Python27\lib\argparse.py", line 2327, in print_help
    self._print_message(self.format_help(), file)
  File "C:\Python27\lib\argparse.py", line 2301, in format_help
    return formatter.format_help()
  File "C:\Python27\lib\argparse.py", line 279, in format_help
    help = self._root_section.format_help()
  File "C:\Python27\lib\argparse.py", line 209, in format_help
    func(*args)
  File "C:\Python27\lib\argparse.py", line 317, in _format_usage
    action_usage = format(optionals + positionals, groups)
  File "C:\Python27\lib\argparse.py", line 388, in _format_actions_usage
    start = actions.index(group._group_actions[0])
IndexError: list index out of range
msg264834 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-05-04 16:14
I don't think add_mutually_exclusive_group and add_argument_group are designed to work together. Did this worked with argparse 1.3.0 or stdlib version of argparse before? I'm getting the same traceback in 2.7 and 3.4+.
msg264835 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016-05-04 16:15
This error occurs due to there is no action belong to the mutexGroup, the code requires at least one. You can simply add

    mutexGroup.add_argument('--foo', action='store_true')

and you'll see the error disappears. I have no idea that this behaviour is intended or not.
msg264837 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016-05-04 16:23
I agree with Berker. Even without any exceptions, the help message given is not what add_argument_group and add_mutually_exclusive_group intend.

You can see

usage: argparse_test.py [-h] [--foo] -u URL -p PROJECT [--dump] --mergeInput
                        MERGEINPUT [--removeDisabled] -c CHECKERS

Sets the checkers of a klocwork project

optional arguments:
  -h, --help            show this help message and exit
  --foo
  -c CHECKERS, --checkers CHECKERS
                        File which lists the checkers to be enabled.

There is not group name and still optional arguments. And the command line does not show exclusive. I believe they are not designed to work like this.
msg264839 - (view) Author: Endre (Endre) Date: 2016-05-04 16:28
Hi,

Thanks for taking a look at it.
I have only tried it with argparse-1.4.0.

"I don't think add_mutually_exclusive_group and add_argument_group are designed to work together."

Yes, it really works strange in this case. I guess the framework should raise an exception which states that this use case is not supported, e.g. at the "serverGroup = mutexGroup.add_argument_group('serverGroup')
" line.
msg265164 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2016-05-08 20:22
Argument Groups are not designed for nesting, and despite their names and subclassing, Mutually exclusive groups and Argument Groups are not meant to be used together (with one exception).

I agree that the error is obscure, but it occurs in a particularly fragile function in the formatter, '_format_actions_usage'.  That function needs a major rewrite (that's in another bug/issue).

Argument Groups serve only as a way of grouping help lines.  Mutually exclusive groups test arguments during parsing, and add some markings to the usage line.  So they have very different purposes.

I have seen questions on Stackoverflow where people try to use Argument Groups as a way of adding some sort of subgroup to the Mutually Exclusive one, one for example that implements a 'allow any of this group' logic.  There is a bug/issue asking for 'inclusive' nesting groups, but production patch of that sort is long ways off.  http://bugs.python.org/issue11588

This usage line

    (--url URL --project Prj [--dump]) | (--mergeInput input.txt [--removeDisabled])

suggests that this what you are trying do - allow any of -u,-p,-d, but disallow one of these with -m or -r.  That logic is beyond the current group testing mechanism, and beyond the usage formatting code.  You'll have to do your own tests, and write a custom usage line.

Mutually exclusive groups can be nested in other mutual groups, but the effect isn't what you might hope.  It just forms a larger mutually exclusive group; there's no subgrouping.

It is possible to nest a mutually exclusive group in an Argument group; the effect is to give the mutually exclusive group a title.
msg326127 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2018-09-23 03:14
If I add an argument to the mutexGroup as suggested by xiang

In [7]: mutexGroup.add_argument('--foo', action='store_true')
Out[7]: _StoreTrueAction(option_strings=['--foo'], dest='foo', nargs=0, const=True, default=False, type=None, choices=None, help=None, metavar=None)
In [8]: parser.print_usage()
usage: ipython3 [-h] -u URL -p PROJECT [--dump] --mergeInput MERGEINPUT
                [--removeDisabled] -c CHECKERS --foo
In [9]: parser.print_help()
usage: ipython3 [-h] -u URL -p PROJECT [--dump] --mergeInput MERGEINPUT
                [--removeDisabled] -c CHECKERS --foo

Sets the checkers of a klocwork project

optional arguments:
  -h, --help            show this help message and exit
  -c CHECKERS, --checkers CHECKERS
                        File which lists the checkers to be enabled.

the argument_group arguments show up in the usage, but without mutually_exclusive markings.  But they don't show up in the help lines.  I suspect all those group arguments will be tested as one mutually exclusive group, not two subgroups.

If I put the argument groups in the main parser, and omit the mutually_exclusive group I get this help:

In [11]: parser.print_help()
usage: ipython3 [-h] -u URL -p PROJECT [--dump] --mergeInput MERGEINPUT
                [--removeDisabled] -c CHECKERS

Sets the checkers of a klocwork project

optional arguments:
  -h, --help            show this help message and exit
  -c CHECKERS, --checkers CHECKERS
                        File which lists the checkers to be enabled.

serverGroup:
  -u URL, --url URL     klocwork server URL.
  -p PROJECT, --project PROJECT
                        klocwork project name.
  --dump                Dump the current checkers config.

mergeGroup:
  --mergeInput MERGEINPUT
                        Input file to merge for the '--checkers' file. Format:
                        checkerName Enabled|Disabled
  --removeDisabled      Disabled checkers will be removed from the '--
                        checkers' file.

The argument groups appear as intended in the help lines.

add_argument_group is inherited from _ActionsContainer.  The parser ArgumentParser also inherits this method.  A possible fix is to override this method in the _MutuallyExclusiveGroup class to raise some sort of not-implemented error.

The documentation, as written, does not suggest or hint that an argument_group can be added to anything but a parser.  But a  stronger disclaimer could be added.
msg326129 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2018-09-23 03:26
https://bugs.python.org/issue29553 deals with a similar problem, the usage display when one mutually exclusive group is embedded in another mutually exclusive group.  In that case, usage is displayed, but with some missing brackets.  

As here there are two issues - the usage formatter is brittle, and groups are not designed for nesting.
msg347469 - (view) Author: Alexander Kapshuna (kapsh) Date: 2019-07-07 10:14
I have a feeling that nesting groups has nothing to do with it. Got same traceback when I forgot to fill my mutually exclusive groups with arguments. 

import argparse

parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
parser.parse_args(['-h'])

# Result:

Traceback (most recent call last):
  File "/home/kapsh/.PyCharmCE2019.1/config/scratches/scratch_12.py", line 5, in <module>
    parser.parse_args(['-h'])
  File "/usr/lib/python3.7/argparse.py", line 1749, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python3.7/argparse.py", line 1781, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python3.7/argparse.py", line 1987, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/usr/lib/python3.7/argparse.py", line 1927, in consume_optional
    take_action(action, args, option_string)
  File "/usr/lib/python3.7/argparse.py", line 1855, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib/python3.7/argparse.py", line 1037, in __call__
    parser.print_help()
  File "/usr/lib/python3.7/argparse.py", line 2474, in print_help
    self._print_message(self.format_help(), file)
  File "/usr/lib/python3.7/argparse.py", line 2458, in format_help
    return formatter.format_help()
  File "/usr/lib/python3.7/argparse.py", line 284, in format_help
    help = self._root_section.format_help()
  File "/usr/lib/python3.7/argparse.py", line 215, in format_help
    item_help = join([func(*args) for func, args in self.items])
  File "/usr/lib/python3.7/argparse.py", line 215, in <listcomp>
    item_help = join([func(*args) for func, args in self.items])
  File "/usr/lib/python3.7/argparse.py", line 322, in _format_usage
    action_usage = format(optionals + positionals, groups)
  File "/usr/lib/python3.7/argparse.py", line 397, in _format_actions_usage
    start = actions.index(group._group_actions[0])
IndexError: list index out of range
msg347477 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2019-07-07 17:02
Xiang Zhang pointed out that the immediate error in this case was caused by the empty mutually exclusive group:

https://bugs.python.org/issue26952#msg264835

The nesting fails because adding actions to the argument_group does not enroll them in the mutually exclusive group.  So the group._group_actions list remains empty, just as it is in your case.

As I pointed out earlier, the usage formatter is brittle.  The addition of mutually exclusive group markings is particularly clunky.
msg408592 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-15 10:08
New changeset 86de99588db3beff964137f4fe27dd1077a09b35 by Irit Katriel in branch 'main':
bpo-26952: [argparse] clearer error when formatting an empty mutually… (GH-30099)
https://github.com/python/cpython/commit/86de99588db3beff964137f4fe27dd1077a09b35
msg408597 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-15 12:20
New changeset 8e4c96295bd78ae5f70b908e5dbac0da7c4c21bd by Miss Islington (bot) in branch '3.10':
bpo-26952: [argparse] clearer error when formatting an empty mutually… (GH-30099) (GH-30114)
https://github.com/python/cpython/commit/8e4c96295bd78ae5f70b908e5dbac0da7c4c21bd
msg408598 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-15 12:24
New changeset f0b274d2e21e6c7c1c0f56464070108f9bd00d20 by Miss Islington (bot) in branch '3.9':
bpo-26952: [argparse] clearer error when formatting an empty mutually… (GH-30099) (GH-30115)
https://github.com/python/cpython/commit/f0b274d2e21e6c7c1c0f56464070108f9bd00d20
History
Date User Action Args
2021-12-15 12:25:22iritkatrielsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-12-15 12:24:53iritkatrielsetmessages: + msg408598
2021-12-15 12:20:25iritkatrielsetmessages: + msg408597
2021-12-15 10:08:49miss-islingtonsetpull_requests: + pull_request28337
2021-12-15 10:08:44miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request28336
2021-12-15 10:08:37iritkatrielsetmessages: + msg408592
2021-12-14 13:16:26iritkatrielsetkeywords: + patch
nosy: + iritkatriel

pull_requests: + pull_request28321
stage: patch review
2021-05-28 13:15:38iritkatrielsettitle: argparse help formatter crashes -> argparse help formatter raises IndexError
versions: + Python 3.9, Python 3.10, Python 3.11, - Python 2.7
2019-07-07 17:02:38paul.j3setmessages: + msg347477
2019-07-07 10:14:49kapshsetnosy: + kapsh
messages: + msg347469
2018-09-23 03:26:59paul.j3setmessages: + msg326129
2018-09-23 03:14:51paul.j3setmessages: + msg326127
2018-07-11 07:43:28serhiy.storchakasettype: crash -> behavior
2016-05-08 20:22:45paul.j3setnosy: + paul.j3
messages: + msg265164
2016-05-04 16:30:54xiang.zhangsetnosy: + bethard
2016-05-04 16:28:41Endresetmessages: + msg264839
2016-05-04 16:23:49xiang.zhangsetmessages: + msg264837
2016-05-04 16:15:10xiang.zhangsetmessages: + msg264835
2016-05-04 16:14:41berker.peksagsetnosy: + berker.peksag
messages: + msg264834
2016-05-04 16:01:22xiang.zhangsetnosy: + xiang.zhang
2016-05-04 15:14:47Endrecreate