classification
Title: argparse drops '|'s when the arguments are too long
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: caveman, eric.smith, louielu, paul.j3
Priority: normal Keywords:

Created on 2017-10-11 21:49 by caveman, last changed 2017-10-12 20:52 by paul.j3.

Files
File name Uploaded Description Edit
31768.py eric.smith, 2017-10-11 22:32
Messages (8)
msg304184 - (view) Author: caveman (caveman) Date: 2017-10-11 21:49
if you execute the code below, mutually exclusive agrs are separated by '|' as expected. but if you uncomment the 1 line down there, then the args list will be too long, and as argparse tries to wrap the args around, it drops all '|'s which semantically destroys the intended syntax of the arguments.

import argparse

parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-v", "--verbose", action="store_true")
group.add_argument("-q", "--quiet", action="store_true")
group.add_argument("-x", metavar='X', type=str, help="the base", nargs='?')
group.add_argument("-y", metavar='Y', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
#group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')

args = parser.parse_args()
msg304186 - (view) Author: caveman (caveman) Date: 2017-10-11 21:59
forgot to add: when you execute the code, pass the argument '-h' in order to have the script generate the help menu text.
msg304187 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2017-10-11 22:32
Removing versions 3.4 and 3.8.

Attaching a reproducing script.  Run it with a parameter of the number of arguments to add. The behavior changes between 7 and 8, although I'm not sure either is wrong, just different. This is from Windows:

% python3 31768.py 7
usage: 31768.py [-h] [-v | -q | -x [X] | -y [Y] | Z | Z | Z | Z | Z | Z | Z]

positional arguments:
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent

optional arguments:
  -h, --help     show this help message and exit
  -v, --verbose
  -q, --quiet
  -x [X]         the base
  -y [Y]         the exponent

% python3 31768.py 8
usage: 31768.py [-h] [-v] [-q] [-x [X]] [-y [Y]]
                [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z]

positional arguments:
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent

optional arguments:
  -h, --help     show this help message and exit
  -v, --verbose
  -q, --quiet
  -x [X]         the base
  -y [Y]         the exponent
msg304188 - (view) Author: caveman (caveman) Date: 2017-10-11 22:37
When | is dropped, it means that the arguments/options are no longer mutually exclusive, which renders the resultant help menu incorrect.
msg304189 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2017-10-11 22:40
Good point.
msg304202 - (view) Author: Louie Lu (louielu) * Date: 2017-10-12 04:11
the format of usage is do at argparse.py:296.

So how do we format this kind of situation,
to be like this, if the group also too long?

  usage: test.py [-h]
                 [-v | -q | -x [X] | -y [Y] | Z | Z | Z | Z | Z |
                  Z | Z | Z]


Also, another bug I think is, given two mutul group,
it will not show out the correct grouping:

usage: test.py [-h] [-v] [-e] [Z] [G] [Z] [G]

positional arguments:
  Z              the exponent
  G              the exponent
  Z              the exponent
  G              the exponent

optional arguments:
  -h, --help     show this help message and exit
  -v, --verbose
  -e, --eeeeee


--- 2 mutul group ---

import sys
import argparse
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-v", "--verbose", action="store_true")
g2 = parser.add_mutually_exclusive_group()
g2.add_argument("-e", "--eplog", action="store_true")
g2.add_argument("-g", "--quiet", action="store_true")

for i in range(int(sys.argv[1])):
    group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
    g2.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')

parser.parse_args(['-h'])
msg304284 - (view) Author: paul j3 (paul.j3) * Date: 2017-10-12 19:48
As documented in many other issues, the usage formatter is brittle.  It formats the individual usages, joins them into a string. Then if too long to fit on one line it tries t split into actions, etc.  This split produces an assertion error if there are 'wierd' characters in the names (e.g. #[]).

With mutually exclusive groups it gets even worse.  The brackets and | are spliced into the original string, and then excess [] and spaces are removed.  Once recent issue complained about its handling of nested groups (which are borderline wrong).

So I"m not surprised that a long group that spans a couple of lines gets messed up.   It requires a major rewrite, and even then I there will be formats involving groups that fall through the cracks.
msg304287 - (view) Author: paul j3 (paul.j3) * Date: 2017-10-12 20:52
I've changed your example a bit to clarify some points.

    parser.add_argument('n', type=int)
    group = parser.add_mutually_exclusive_group()
    group.add_argument("-v", "--verbose", action="store_true")
    group.add_argument("-q", "--quiet", action="store_true")
    group.add_argument("-x", metavar='X', type=str, help="the base", nargs='?')
    group.add_argument("-y", metavar='Y', type=str, help="the exponent", nargs='?')

    for i in range(int(sys.argv[1])):
         group.add_argument("z%s"%i, nargs='?')

With `n=0`, usage is:

    usage: PROG [-h] [-v | -q | -x [X] | -y [Y]] n

Notice that the positional is placed last.  That is, regardless of the definition order, 'usage' shows optionals first, positionals last.

With 2 'z':

    usage: PROG [-h] [-v] [-q] [-x [X]] [-y [Y]] n [z0] [z1]

All the mutually exclusive markings have been dropped.  That's because the component actions are not consecutive, having been split up by the reordering of positionals.  Marking the group is done only if it can do so in a simple minded way.  The group testing is not dependent on being able to format it.  That's done by a different part of the code.

For 7 'z' it splits the line.  Positionals are printed on a new line.

    usage: PROG [-h] [-v] [-q] [-x [X]] [-y [Y]]
                n [z0] [z1] [z2] [z3] [z4] [z5] [z6]


I noticed this problem with formating group positional in multiline usage some time ago, https://bugs.python.org/issue10984#msg192954


Another thing to watch out for - multiple positionals in a group don't make sense.  Only the first one can ever be filled.  I don't see a formal check, but it doesn't make logical sense.

If I change the code so it produces multiple optionals:

    usage: PROG [-h] [-v | -q | -x X | -y Y | --z0 [Z0] | --z1 [Z1] | --z2 [Z2] |
                --z3 [Z3] | --z4 [Z4]]
                n

the group markings are preserved across lines.  Note again that the positional (n) is on its own line.

---

Here's another weird behavior, with just one positional in the group:

    1314:~/mypy$ python3 argdev/issue31768.py 1 -v 3
    usage: PROG [-h] [-v] [-q] [-x X] [-y Y] n [z0]
    PROG: error: unrecognized arguments: 3

It complains about not recognizing the '3', rather than complaining about 'z0' not allow with '-v'.  That's because the '?' has already been satisfied with [] when parsing the 'n' (see https://bugs.python.org/issue15112).

---

In sum, I think this is pathological case that doesn't need to be fixed.  

Usage is ok if there's a long list of optionals.  Usage (and parsing) with just one positional in the group is brittle.  We shouldn't expect it work with many positionals in the group.

---

@Louie, you 2nd example interweaves the definitions of two groups.  Except for the optionals/positions reordering that I mentioned above, usage does not reorder arguments.  Mutually exclusive groups are marked only if they are contiguous.  

https://bugs.python.org/issue10984 has a patch that can format groups even if the actions don't occur in the right order.  It isn't a trivial fix.
History
Date User Action Args
2017-10-12 20:52:05paul.j3setmessages: + msg304287
2017-10-12 19:48:11paul.j3setnosy: + paul.j3
messages: + msg304284
2017-10-12 04:11:32louielusetnosy: + louielu
messages: + msg304202
2017-10-11 22:40:58eric.smithsetmessages: + msg304189
2017-10-11 22:37:34cavemansetmessages: + msg304188
2017-10-11 22:32:34eric.smithsetfiles: + 31768.py
versions: - Python 3.4, Python 3.8
nosy: + eric.smith

messages: + msg304187
2017-10-11 21:59:39cavemansetmessages: + msg304186
2017-10-11 21:49:25cavemancreate