This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: argparse assertion failure with multiline metavars
Type: behavior Stage: test needed
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: MaT1g3R, bethard, iritkatriel, martin.panter, paul.j3, terry.reedy
Priority: normal Keywords:

Created on 2018-02-18 05:06 by MaT1g3R, last changed 2022-04-11 14:58 by admin.

Messages (7)
msg312299 - (view) Author: (MaT1g3R) * Date: 2018-02-18 05:06
If I run this script with -h
-----8<------------------
from argparse import ArgumentParser
mapping = ['123456', '12345', '12345', '123']
p = ArgumentParser('11111111111111')
p.add_argument('-v', '--verbose', help='verbose mode', action='store_true')
p.add_argument('targets', help='installation targets',  nargs='+', metavar='\n'.join(mapping))
p.parse_args()
---------8<--------------------
I get an error:
---------8<--------------------
Traceback (most recent call last):
  File "tmp.py", line 7, in <module>
    p.parse_args()
  File "/usr/lib/python3.6/argparse.py", line 1730, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python3.6/argparse.py", line 1762, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python3.6/argparse.py", line 1968, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/usr/lib/python3.6/argparse.py", line 1908, in consume_optional
    take_action(action, args, option_string)
  File "/usr/lib/python3.6/argparse.py", line 1836, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib/python3.6/argparse.py", line 1020, in __call__
    parser.print_help()
  File "/usr/lib/python3.6/argparse.py", line 2362, in print_help
    self._print_message(self.format_help(), file)
  File "/usr/lib/python3.6/argparse.py", line 2346, in format_help
    return formatter.format_help()
  File "/usr/lib/python3.6/argparse.py", line 282, in format_help
    help = self._root_section.format_help()
  File "/usr/lib/python3.6/argparse.py", line 213, in format_help
    item_help = join([func(*args) for func, args in self.items])
  File "/usr/lib/python3.6/argparse.py", line 213, in <listcomp>
    item_help = join([func(*args) for func, args in self.items])
  File "/usr/lib/python3.6/argparse.py", line 334, in _format_usage
    assert ' '.join(pos_parts) == pos_usage
AssertionError
-----8<------------------
msg312674 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-02-23 23:52
I don't understand the -h (display help part) but pasting code into repository 3.8 IDLE editor and running, I get same traceback.

Never having used argparse, I don't know if a multiline metavar is expected to work.  Assuming so, can you suggest a fix?
msg312680 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2018-02-24 00:42
This looks like the same assertion failure as described in Issue 16360. Paul pointed to a patch in Issue 11874, so that may also be relevant.

However I agree that embedding newlines in a metavar doesn’t make much sense. What’s the use case?
msg312683 - (view) Author: (MaT1g3R) * Date: 2018-02-24 01:15
I tried to include line breaks for listing options for a positional argument.

The default metavar for that is something like {opt1, opt2, op3},
however I wanted it to display the options on separate lines.
msg312693 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-02-24 02:42
If newlines are not permitted in metavars, the user should see
ValueError: newline not permitted in metavar
msg314063 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2018-03-18 22:03
I haven't seen anyone try to use \n in a metavar before, but other special characters like [] and () produce this assertion error.  At this point the code is trying split the usage into 2 or more lines, because it's too long for one.  It creates a usage for optionals and positionals separately.

In a clumsy way, it formats the whole usage string, and tries to split it into pieces so it can decide to split long lines.  The 'assert' is used to make sure it has split the line into meaningful blocks.  It splits with

    r'\(.*?\)+|\[.*?\]+|\S+'

basically white space (including nl) and [] and () which are used to mark optional arguments and groups.  So including any of these characters in text via metavar will screwup this split.

We could try to refine this splitting expression, but that feels like a never ending task as users become more inventive.

I suggested a major rewrite of this section, one that keeps the pieces a list, and joins them after deciding how many can fit on a line.

No one has, to my knowledge, come up with a comprehensive list of characters that will cause problems here.

argparse does provide a backup - a user provided usage string.  That's not as nice as a automatically generated one, but if you have to have something special, that's the way to go.  In the long run there's only so much that general purpose parser can do to accommodate special needs.
msg408641 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-15 19:58
It works for me on 3.11:

% ./python.exe b.py -h
usage: 11111111111111 [-h] [-v] 123456
12345
12345
123 [123456
12345
12345
123 ...]

positional arguments:
  123456
12345
12345
123
                        installation targets

options:
  -h, --help            show this help message and exit
  -v, --verbose         verbose mode
History
Date User Action Args
2022-04-11 14:58:57adminsetstatus: pending -> open
github: 77048
2021-12-15 19:58:48iritkatrielsetstatus: open -> pending

nosy: + iritkatriel
messages: + msg408641

resolution: fixed
2018-03-18 22:03:00paul.j3setnosy: + paul.j3
messages: + msg314063
2018-02-24 02:42:59terry.reedysetmessages: + msg312693
2018-02-24 01:15:06MaT1g3Rsetmessages: + msg312683
2018-02-24 00:42:48martin.pantersetnosy: + martin.panter
messages: + msg312680
2018-02-23 23:52:52terry.reedysetversions: + Python 3.7, Python 3.8
nosy: + terry.reedy, bethard

messages: + msg312674

stage: test needed
2018-02-18 05:06:43MaT1g3Rcreate