Title: argparse optionals with nargs='?', '*' or '+' can't be followed by positionals
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Kotan, TD22057, berker.peksag, bethard, catherine, elsdoerfer, jwilk, merwok, paul.j3, phawkins, wrobell
Priority: high Keywords: patch

Created on 2010-07-23 10:46 by bethard, last changed 2017-01-20 01:30 by paul.j3.

File name Uploaded Description Edit
test_pos_after_var_args.patch catherine, 2010-08-02 18:42 review
issue9338_argparse.patch Kotan, 2010-11-20 19:05 review
issue9338_3.patch paul.j3, 2013-05-17 17:02 review paul.j3, 2013-05-17 17:10
issue9338_7.patch paul.j3, 2013-05-23 16:23 review
Messages (21)
msg111270 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2010-07-23 10:46
[From the old argparse tracker:]

You can't follow a nargs='+' optional argument with a positional argument:

>>> import argparse
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--badger', nargs='+')
>>> parser.add_argument('spam')
>>> parser.parse_args('--badger A B C D'.split())
usage: PROG [-h] [--badger BADGER [BADGER ...]] spam
PROG: error: too few arguments

Ideally, this should produce:

>>> parser.parse_args('--badger A B C D'.split())
Namespace(badger=['A', 'B', 'C'], spam='D')

The problem is that the nargs='+' causes the optional to consume all the arguments following it, even though we should know that we need to save one for the final positional argument.

A workaround is to specify '--', e.g.:

>>> parser.parse_args('--badger A B C -- D'.split())
Namespace(badger=['A', 'B', 'C'], spam='D')

The problem arises from the fact that argparse uses regular-expression style matching for positional arguments, but it does that separately from what it does for optional arguments.

One solution might be to build a regular expression of the possible things a parser could match. So given a parser like::

  parser = argparse.ArgumentParser()
  parser.add_argument('-x', nargs='+')
  parser.add_argument('z', nargs='*')

the regular expression might look something like (where positionals have been replaced by the character A)::

  (-w A)? (-x A+)? A (-w A)? (-x A+)? A* (-w A)? (-x A+)?

Note that the optionals can appear between any positionals, so I have to repeat their regular expressions multiple times. Because of this, I worry about how big the regular expression might grow to be for large parsers. But maybe this is the right way to solve the problem.
msg111280 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2010-07-23 11:26
This is definitely a different bug from the one you just marked it as a duplicate of.
msg111282 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-07-23 11:41
I read too fast, I’m sorry for that.
msg112511 - (view) Author: Catherine Devlin (catherine) Date: 2010-08-02 18:42
Here's a unit test for the simplest cases.
msg121013 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-11-12 02:32
Looks good to me.  Do you want to propose a code patch too, and/or more tests for non-simple cases?
msg121720 - (view) Author: Daniel Albeseder (Kotan) Date: 2010-11-20 19:05
My attached patch adds the "--" between the optionals and the arguments, if there are optionals which have variable length and at least some positional argument can be provided.

Patch is for python 3.2 svn revision 86553.
msg122016 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-11-21 23:20
Patch looks good, thanks!  Does it integrate Catherine’s tests?
msg141639 - (view) Author: wrobell (wrobell) Date: 2011-08-04 23:03
is there a chance to fix this issue?
msg141660 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2011-08-05 15:20
There is.  Someone wanting to help could reply to the question I asked :)
msg166237 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2012-07-23 16:57
So Kotan's patch doesn't actually solve the original problem. Instead, it inserts the workaround into the help message of the parser. I think this is probably not the right fix. We should probably do two things:

(1) Right now: create a documentation patch which at least explains the current limitations of argparse parsing, and describes the '--' workaround. Probably this patch should add a separate section about '--', give an example like the one in this issue, and then cross-reference this section from nargs='?', nargs='*', nargs='+' and the "Arguments containing -" section.

(2) Longer term: create a code patch that implements the changes to the regular expression-based parsing like I've suggested.
msg166239 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2012-07-23 17:04
And I guess Issue 9182 is the right place for (1).
msg189198 - (view) Author: paul j3 (paul.j3) * Date: 2013-05-14 04:28
I've played a bit the idea that barthard sketched.  I don't have all the details worked out, but I believe this is what will happen:


  parser = argparse.ArgumentParser()
  parser.add_argument('-x', nargs='+')
  parser.add_argument('z', nargs='*')

some possible parses are

    '-w 1 -x 2 3 4 5',     # w:1, x:[2,3,4], y:5, z:[] -
                           # fail +
    '-w 1 2 -x 3 4 5',     # w:1, y:2, x:[3 4 5], z:[] +

    '-w 1 -x 2 3',         # w:1, x:[2], y:3, z:[]     -
                           # fail +
    '-x 1 2 -w 3 4 5 6',   # w:3, x:[1,2], y:4, z:[5,6] +
                           # w:3, x:[1], y:2, z:[4,5,6] -

    '-x 1 2 3 4 -w 5 6 7', # w:5, x:[1,2,3,4], y:5, z:[7] +
                           # w:5, x:[1,2,3], y:4, z:[6,7] -

    '1 2 3 -x 4 5 -w 6',   # w:6, x:[4,5], y:1, z:[2,3] +
'+' lines are those currently produced
'-' lines are ones that would be produced by these ideas

'-w 1 -x 2 3 4 5' is the protypical problem case.  The current parser allocates all [2,3,4,5] to -x, leaving none for y, thus failing.  So desired solution is to give 5 to y, leaving -x with the rest.

'-x 1 2 -w 3 4 5 6' is a potentially ambiguous case.  The current parser lets -x grab [1,2]; y then gets 4, and z the remainder.  But the alternative is to give 2 to y, leaving -x with just [1].

In this case 
arg_strings_pattern = 'OAAOAAAA'

replacing the Os with the option flags: '-xAA-wAAAA'

I match this with a refined version of bethard's regex:

    pat = _re.compile('%s?(?P<y>A)%s?(?P<z>A*)%s?'%(pat1,pat1,pat1))

groups (without the Nones) and groupdict are

    ['-xA', 'A', '-wA', 'AAA']
    {'z': 'AAA', 'y': 'A'}

So this does effectively give y the 2nd argument, leaving -x with just the 1st.

The current parser effectively groups the arguments as

    ['-xAA, '-wA', 'A', 'AA']

In the real world, generating and apply a global pattern like this could get complicated.  For example there are long option names ('--water'), and combined argument strings ('-x1', '-x=1').
msg189200 - (view) Author: paul j3 (paul.j3) * Date: 2013-05-14 07:45
I need to make one correction to my last post:

    '-x 1 2 -w 3 4 5 6',   # w:3, x:[1,2], y:4, z:[5,6] +
                           # w:3, x:[1], y:2, z:[4,5,6] -

The second solution is only possible if 'z' is not consumed when 'y' is being processed.  In current version, if consume_positionals() is called with a 'AOAAAA' pattern, 'y' will match the first 'A', and 'z' will match ''.  That means '4 5 6' will be left over.

It's only when I use the patch in
(argparse doesn't allow optionals within positionals)
that the processing 'z' is delayed, so it can get [4,5,6].

So at least with the 4 arguments in this example, bethard's idea only seems to make a difference in the case of '-w 1 -x 2 3 4 5', where 'y' lays claim to the last string, and '-x' gets the rest.
msg189470 - (view) Author: paul j3 (paul.j3) * Date: 2013-05-17 17:02
This patch implements, I think, the ideas bethard proposed.  It is test patch, not intended for production.

Most of work is in ArgumentParser._get_alt_length() which
- generates a pattern along the lines bethard proposed
- generates a string like arg_strings_pattern, but with optionals strings ('-x') instead of 'O'.
- runs a match
- from groups like '-xAAA', creates dict entries like: 
    alt_opt_length['x'] = 3

Later, in consume_optionals(), this alternative count replaces arg_count if it is lower.  The next consume_positionals() then takes care of consuming the unconsumed arguments.

If _get_alt_length() has any problems, it logs an error, and returns an otherwise empty dict.  So it 'fails' quietly without affecting regular parsing.

Reasons for failing include (for now) the use of subparsers, optionals with explicit args, and special prefix_chars.  With exclusions like this, runs without errors or failures.

Since this is still a testing vehicle, it writes an issue9338.log file with debugging entries.

This version works, but is both not sufficiently general and too general.  As bethard notes, the testing pattern could get very large if there are many optionals.  Ideally the pattern will allow the optionals in any order and combination between positionals.  The ambiguities that I discussed in the previous 2 posts disappear if the patching pattern is sufficiently general.

But I also suspect it is too general.  It does not need to match every case, just those where an optional is consuming arguments that should go to a positional.  But if we come up with something more specific, this could still be a useful testing tool.
msg189472 - (view) Author: paul j3 (paul.j3) * Date: 2013-05-17 17:10
This is a test file for the patch I just submitted.

It is not a formal unitttest, but uses print output as much as assert.

Cases include the example bethard used, as well as ones from that initially caused problems.
msg189867 - (view) Author: paul j3 (paul.j3) * Date: 2013-05-23 16:20
Here's another approach to the problem, using an iterative localized search. For simple cases it produces the same thing, but in complex cases it is more robust.

It is based on two ideas:

- if the action in consume_optional() is being 'greedy', use

    slots = self._match_arguments_partial([action]+positionals, selected_patterns)

  to determine if this action can share arguments with any of the remaining positionals.  This is similar to how consume_positionals() allocates arguments to the set of positionals.

- try this 'sharing' with the last optional.  If that is not enough, try the penultimate optional as well, and continue working toward the start as needed.

Since the normal parsing is from left to right, figuring out when to start 'sharing' requires some sort of search strategy.  I have moved the start_index loop into a consume_loop() function, and added a switch so it can parse the arguments without invoking take_action() (so arguments are not evaluated, and namespace is not changed).

If there is a suspected 'greed' problem, consume_loop() is called (in test mode) one or more times to determine the right-most optionals to use, and once more (with take_action) to parse and evaluate the arguments.

As in the previous patch this writes a log file for debugging purposes. now has a number of tests for this issue.  

It is more robust than the previous patch, and does not need special handling for things like subparsers and explicit arguments.
msg189868 - (view) Author: paul j3 (paul.j3) * Date: 2013-05-23 16:23
Oops, I attached the wrong file.  Here's the correct one.
msg200858 - (view) Author: Paul Hawkins (phawkins) Date: 2013-10-22 01:51
I ran into this bug the first time I needed nargs + in a tool.  

I found of course that if the option with the nargs is followed by another option before the positional arguments it will work as expected.  But then the help would have to point this out, and it still could be used incorrectly (so then I get a mail about a bug in my tool.) ;-)>

My workaround was to use action=append instead of nargs, then user would just have to give the option for each nargs desired.  Since my use would be short this was OK.  But the usage message does not reflect the multiple use nature of this option

But what I expected to find in the doc was a way to specify the use of a separator char between the nargs option arguments.  For example specify that ',' is the separator arg (currently a space is the separator.)  So if option is -foo the cli could be: -foo bar1,bar2,bar3 pos1 pos2

(Of course I could just have the tool take a comma delimited single argument and parse it in the tool's logic, but again then a custom usage message would be needed.)

Has this solution been considered?
msg200862 - (view) Author: paul j3 (paul.j3) * Date: 2013-10-22 02:55
parse_args() would see ['-foo', 'bar1,bar2,bar3', 'pos1', 'pos2'].  The splitting on space is done by the shell.  So having your own code split 'bar1,bar2,bar3' is simplest.  But that would be messed up if the user entered 'bar1, bar2, bar3...'.  You could also ask the user to use quotes - "bar1, bar2, bar3".
msg216990 - (view) Author: paul j3 (paul.j3) * Date: 2014-04-22 06:33
breaks one test that I added to issue

    +class TestPositionalsAfterOptionalsPlus(ParserTestCase):
    +    """Tests specifying a positional that follows an arg with nargs=+
    +    prototypical problem"""
    +    argument_signatures = [
    +        Sig('-w'),
    +        Sig('-x', nargs='+'),
    +        Sig('y', type=int),
    +        Sig('z', nargs='*', type=int)]
    +    failures = ['1 -x 2 3 -w 4 5 6' # error: unrecognized arguments: 5 6
    +                # z consumed in 1st argument group '1'
    +    ]

This no longer fails.  Due to 15112, z=[5,6].  That is, it is no longer consumed by the 1st argument group.
msg285866 - (view) Author: paul j3 (paul.j3) * Date: 2017-01-20 01:30
Recent StackOverFlow question related to this issue - where the following positional is a subparsers.
Date User Action Args
2017-01-20 01:30:00paul.j3setmessages: + msg285866
2017-01-19 23:03:18paul.j3setpriority: normal -> high
2017-01-08 04:45:32berker.peksagsetnosy: + berker.peksag
2014-04-23 11:40:03jwilksetnosy: + jwilk
2014-04-22 06:33:18paul.j3setmessages: + msg216990
2013-10-22 02:55:49paul.j3setmessages: + msg200862
2013-10-22 01:51:34phawkinssetnosy: + phawkins
messages: + msg200858
2013-07-17 15:23:22TD22057setnosy: + TD22057
2013-05-23 16:23:52paul.j3setfiles: + issue9338_7.patch

messages: + msg189868
2013-05-23 16:22:48paul.j3setfiles: -
2013-05-23 16:20:42paul.j3setfiles: +

messages: + msg189867
2013-05-17 17:10:16paul.j3setfiles: +

messages: + msg189472
2013-05-17 17:02:40paul.j3setfiles: + issue9338_3.patch

messages: + msg189470
2013-05-14 07:45:16paul.j3setmessages: + msg189200
2013-05-14 04:29:00paul.j3setmessages: + msg189198
2013-05-13 02:29:52paul.j3setnosy: + paul.j3
2012-07-23 17:04:07bethardsetmessages: + msg166239
2012-07-23 16:57:36bethardsetmessages: + msg166237
title: argparse optionals with nargs='+' can't be followed by positionals -> argparse optionals with nargs='?', '*' or '+' can't be followed by positionals
2012-07-23 16:46:01bethardlinkissue15433 superseder
2011-08-05 15:20:14merwoksettype: enhancement -> behavior
messages: + msg141660
versions: + Python 3.3
2011-08-04 23:03:52wrobellsetnosy: + wrobell
messages: + msg141639
2010-11-21 23:20:04merwoksetnosy: bethard, merwok, catherine, elsdoerfer, Kotan
messages: + msg122016
components: + Library (Lib)
stage: needs patch -> patch review
2010-11-20 19:05:01Kotansetfiles: + issue9338_argparse.patch
nosy: + Kotan
messages: + msg121720

2010-11-12 02:32:40merwoksetmessages: + msg121013
versions: + Python 2.7, Python 3.2
2010-08-12 10:17:53elsdoerfersetnosy: + elsdoerfer
2010-08-02 18:42:03catherinesetfiles: + test_pos_after_var_args.patch

nosy: + catherine
messages: + msg112511

keywords: + patch
2010-07-23 11:41:47merwoksetnosy: + merwok
messages: + msg111282
2010-07-23 11:26:16bethardsetstatus: closed -> open
resolution: duplicate ->
messages: + msg111280

superseder: argparse does not accept options taking arguments beginning with dash (regression from optparse) ->
stage: resolved -> needs patch
2010-07-23 10:47:34merwoksetstatus: open -> closed
resolution: duplicate
superseder: argparse does not accept options taking arguments beginning with dash (regression from optparse)
stage: needs patch -> resolved
2010-07-23 10:46:27bethardcreate