This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: ArgumentParser behavior does not match generated help
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.4
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: paul.j3, tellendil
Priority: normal Keywords:

Created on 2015-05-11 14:15 by tellendil, last changed 2022-04-11 14:58 by admin.

File name Uploaded Description Edit tellendil, 2015-05-11 14:15 Example of the problem with argparse's subparsers
Messages (7)
msg242895 - (view) Author: Benjamin Schubert (tellendil) Date: 2015-05-11 14:15
When creating a ArgumentParser on which we attach a subparser with different options and then add a nargs="+" argument to the initial parser, the command format string generated does not match the behavior.

for example it would generate : [-h] {ls,du} ... vm [vm ...]

but only accept one vm.

I would suspect a bug when parsing the arguments (as the help meets the desired behavior).

Attached is a little script to reproduce the error.

Thank you !
msg242902 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2015-05-11 17:06
I wouldn't describe this as bug, just a nuance on how parsers and subparsers play together.

To the main parser, the subparser argument looks like another positional.   It allocates strings to it and any following positionals based on their respective 'nargs'.

The nargs for a subparser is 'A...' (argparse.PARSER), which is similar to '+' (it takes one or more strings)



    Namespace(arg1=['create', 'test'], arg2=['test2'])

Notice how 2 of the strings are allocated to arg1, and only 1 to arg2. arg2 is happy with just 1, so arg1 gets the rest.

In your example it's the subparser that is issuing the 'unrecognized arguments' message, because it doesn't have a positional argument that would take it.

Having more than one positional that takes are variable number of arguments is tricky.  I find it helpful to think in terms of how `re` would handle a pattern like `(A+)(A*)(A)`.
msg242905 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2015-05-11 17:26
And the behavior does match the help

    {ls,du} ... vm [vm ...]

It's just that one of the strings is allocated to the first `...`, whereas you expected it to be put in the second.
msg242947 - (view) Author: Benjamin Schubert (tellendil) Date: 2015-05-12 07:29
Thanks a lot for this explanation ! It is more clear now for why it is not working.

However, I would suggest that the ArgumentParser should still try to match anything the subparser could not, or would this be too complicated ?

Moreover, if this scheme is not feasible, I would suggest modifying the documentation, or having a warning, or something to let people know. If this is possible.
msg242948 - (view) Author: Benjamin Schubert (tellendil) Date: 2015-05-12 08:19
I solved my problem by subclassing the ArgumentParser and redefining parse_args as follow :

class MultipleArgumentParser(ArgumentParser):
    def parse_args(self, args=None, namespace=None):
        args, argv = self.parse_known_args(args, namespace)

        if not argv:
            return args

        # save old actions, before rerunning the parser without the _SubParsersActions
        self._old_actions = self._actions.copy()
        self._actions = [action for action in self._old_actions if not isinstance(action, _SubParsersAction)]

        # parse the remaining command line
        args2, argv2 = self.parse_known_args(argv, None)

        self._actions = self._old_actions.copy()

        if argv2:
            msg = _('unrecognized arguments: %s')
            self.error(msg % ' '.join(argv2))

        for key, value in vars(args2).items():
            if isinstance(value, collections.Iterable):
                setattr(args, key, [value for value in itertools.chain(getattr(args, key), value)])

        return args

I know this is not generic enough and not cleanly done. However, would this be an interesting addition to the argumentparser ? If so, I can try to make a generic implementation, which would allow having multiple arguments after a subparser which did not match them
msg243025 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2015-05-12 22:12
Look at
argparse optionals with nargs='?', '*' or '+' can't be followed by positionals

That has a proposed patch that wraps the main argument consumption loop in another loop.

The current loop alternatively consumes optionals and positionals until the argv list is done.  The `consume_loop` method in that patch tries various allocations of argv strings between optionals and positionals.  It performs 'dry' runs until it finds something that consumes most of the strings, and then does the actual parsing with changes to the namespace.

The idea might be adapted to work with subparsers, paying attention, as you do, to the 'extras' from parse_known_args.  But it might be hard to reliably perform a 'dry' run when subparsers are involved.

I suspect that any change along this line will be too complex to ever be the default behavior.  The chances of messing with backward compatibility are just too great.  It might pass as an alternative parsing call.
msg246482 - (view) Author: Benjamin Schubert (tellendil) Date: 2015-07-09 10:48
Ok, sorry for the delay.

I see your point and understand the difficulty of having done right.

Should I close the issue, or propose something else ?

Date User Action Args
2022-04-11 14:58:16adminsetgithub: 68354
2015-07-09 10:48:00tellendilsetmessages: + msg246482
2015-05-12 22:12:04paul.j3setmessages: + msg243025
2015-05-12 08:19:21tellendilsetmessages: + msg242948
2015-05-12 07:29:34tellendilsetmessages: + msg242947
2015-05-11 17:26:42paul.j3setmessages: + msg242905
2015-05-11 17:06:32paul.j3setnosy: + paul.j3
messages: + msg242902
2015-05-11 14:15:33tellendilcreate