This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: [argparse] Bad error message formatting when using custom usage text
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: TurboTurtle, bobblanchett, paul.j3, rhettinger
Priority: normal Keywords:

Created on 2020-11-09 17:25 by TurboTurtle, last changed 2022-04-11 14:59 by admin.

Messages (7)
msg380598 - (view) Author: Jake Hunsaker (TurboTurtle) Date: 2020-11-09 17:25
In the sos project, we build a custom `usage` string for our argparser parser, and have noticed that doing so causes error messages from argparse to be badly formatted.

For example if a bad option value is given, the error message is mangled into the last line of our usage string:

```
# python3 bin/sos report --all-logs=on
usage: sos report [options]
sos <component> [options]

[..snip...]
	collect, collector            Collect an sos report from multiple nodes simultaneously report: error: argument --all-logs: ignored explicit argument 'on'
```


This is especially strange since we build the usage string with a trailing newline character:

```
        for com in self._components:
            aliases = self._components[com][1]
            aliases.insert(0, com)
            _com = ', '.join(aliases)
            desc = self._components[com][0].desc
            _com_string += (
                "\t{com:<30}{desc}\n".format(com=_com, desc=desc)
            )
        usage_string = ("%(prog)s <component> [options]\n\n"
                        "Available components:\n")
        usage_string = usage_string + _com_string
        epilog = ("See `sos <component> --help` for more information")
        self.parser = ArgumentParser(usage=usage_string, epilog=epilog)
```


So it appears the trailing newlines are being stripped (in our case, unintentionally?). As expected, removing the trailing newline when passing `usage_string` to our parse does not change this behavior.

However, if we don't set the usage string at all when instantiating our parser, the error message is properly formatted beginning on a new line. Slightly interesting is that without the usage_string being passed, the error message is prefixed with "sos: report:" as expected for %(prog)s expansion, but when the error message is mangled `%(prog)s` is left out as well.

A little more context is available here: https://github.com/sosreport/sos/issues/2285
msg380611 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2020-11-09 19:45
Provide a minimal reproducible example.  I can't reproduce that run on error message.

Also test with arguments like '--all-logs on', which issues an 'unrecognizeable argument' error (with a different error reporting path).

Stripping excess newlines is normal, both in the full help and error.  That's done at the end of help formatting.
msg380612 - (view) Author: Jake Hunsaker (TurboTurtle) Date: 2020-11-09 19:56
I'll try and get a simple reproducer made shortly, however as a quick note I've found that using '--all-logs on' results in a properly formatted error message.
msg380615 - (view) Author: Jake Hunsaker (TurboTurtle) Date: 2020-11-09 20:29
Ah, ok - so I neglected to mention we're using subparsers which appears to be relevant here. My apologies.

Here's a minimal reproducer that shows the behavior when using './arg_test.py foo --bar=on'

```
#! /bin/python3

import argparse

usage_string = 'test usage string ending in newlines\n\n'
sub_cmd_usage = ''

for i in range(0, 3):
    sub_cmd_usage += '\tfoo  --bar\n'

usage_string += sub_cmd_usage

parser = argparse.ArgumentParser(usage=usage_string)
subparser = parser.add_subparsers(dest='subcmd', metavar='subcmd')
subcmd_parser = subparser.add_parser('foo')
subcmd_parser.add_argument('--bar', action="store_true", default=False)

if __name__ == '__main__':
    args = parser.parse_args()

```
msg380616 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2020-11-09 21:26
It's the subparser that's producing this error, specifically its 'prog' attribute.

If I use a custom usage with a simple parser:

    1129:~/mypy$ python3 issue42297.py --foo=on
    usage: issue42297.py
    one
    two
    three
    issue42297.py: error: argument --foo: ignored explicit argument 
    'on'

Notice that the error line includes the 'prog'.  

With subparsers, the main usage is included in the subcommand prog:

    print(subcmd_parser.prog)

produces:

    test usage string ending in newlines

	foo  --bar
	foo  --bar
	foo  --bar foo

That's the usage plus the subcommand name, 'foo'.

Generating the explicit error in the subcommand:

    1244:~/mypy$ python3 issue42297.py foo --bar=on
    test usage string ending in newlines

	foo  --bar
	foo  --bar
	foo  --bar foo: error: argument --bar: ignored explicit 
    argument 'on'

'issue42297.py: ' has been replaced by the usage+'foo', and no newline.

We don't see this in the 'unrecognized' case because that error issued by the main parser.

    issue42297.py: error: unrecognized arguments: on

If I explicitly set the prog of the subcommand:

    subcmd_parser = subparser.add_parser('foo', prog='myscript foo')

The error becomes:

    1256:~/mypy$ python3 issue42297.py foo --bar=on
    usage: myscript foo [-h] [--bar]
    myscript foo: error: argument --bar: ignored explicit argument 'on'

I can also add 'usage=usage_string' to the add_parser.  For the most part add_parser takes the same parameters as ArgumentParser.

Alternatively we can specify prog in 

    subparser = parser.add_subparsers(dest='subcmd', metavar='subcmd', prog='myscript')

resulting in:

    myscript foo: error: argument --bar: ignored explicit argument 'on'

I recently explored how 'prog' is set with subparsers in

https://bugs.python.org/issue41980

I don't think anything needs to be corrected in argparse.  There are enough options for setting prog and usage in subcommands to get around this issue.  

In the worse case, you might want to create an alternative 

    _SubParsersAction

Action subclass that defines the prog/usage differently.
msg380619 - (view) Author: Jake Hunsaker (TurboTurtle) Date: 2020-11-09 22:03
Ok, yeah there seem to be several paths to avoid this behavior then. We should be fine exploring those options.


Thanks for the pointer!
msg380701 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2020-11-10 19:44
We could look into using a different more compact 'prog' for these error messages.

Currently the subparser 'prog' uses the main prog plus positionals plus the subparser name.  The goal is to construct a subparser usage that reflects required input.  

This is fine for the usage line, but could be too verbose for some errors.  In an error, the 'prog' just helps identify which argument has problems, and doesn't need the extra usage information.   Most of the time that isn't an issue, since we don't use positional much in the main parser (and when used can't have variable nargs).

But I don't have immediate ideas as to what can be conveniently (and safely) changed.
History
Date User Action Args
2022-04-11 14:59:37adminsetgithub: 86463
2021-08-07 02:02:50bobblanchettsetnosy: + bobblanchett
2020-11-10 19:44:18paul.j3setmessages: + msg380701
2020-11-09 22:03:22TurboTurtlesetmessages: + msg380619
2020-11-09 21:26:16paul.j3setmessages: + msg380616
2020-11-09 20:29:35TurboTurtlesetmessages: + msg380615
2020-11-09 19:56:41TurboTurtlesetmessages: + msg380612
2020-11-09 19:45:37paul.j3setmessages: + msg380611
2020-11-09 17:29:05xtreaksetnosy: + rhettinger, paul.j3
2020-11-09 17:25:59TurboTurtlecreate