classification
Title: argparse: positional args with nargs='*' defaults to []
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: bethard, chris.jerdonek, paul.j3, r.david.murray, sebix
Priority: normal Keywords: patch

Created on 2013-01-06 09:41 by chris.jerdonek, last changed 2018-10-16 17:01 by sebix.

Files
File name Uploaded Description Edit
issue-16878-1.patch chris.jerdonek, 2013-01-07 22:29 review
issue-16878-2.patch chris.jerdonek, 2013-01-10 02:25 review
Messages (10)
msg179174 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2013-01-06 09:41
In argparse, positional arguments with nargs='*' default to [] rather None, even if default=None is passed explicitly.  The documentation says otherwise:

"The default keyword argument of add_argument(), whose value defaults to None, specifies what value should be used if the command-line argument is not present. ... For positional arguments with nargs equal to ? or *, the default value is used when no command-line argument was present:"

(from http://docs.python.org/dev/library/argparse.html#default )

import argparse

def parse(args, **kwargs):
    parser = argparse.ArgumentParser()
    parser.add_argument('foo', **kwargs)
    ns = parser.parse_args(args)
    print(repr(ns.foo))

parse([], nargs='?')                # None
parse([], nargs='*')                # []    <--
parse([], nargs='*', default=None)  # []    <--
parse([], nargs='*', default=False) # False
parse([], nargs='*', default=0)     # 0

Three options include (there may be more):

(1) document the behavior
(2) make a default of None yield None
(3) do (2), but change the default to [] instead of None when nargs='*'
msg179184 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-06 14:43
I'd prefer to fix it, since having the action='append' and nargs='*' behaviors be different looks like a bug.  However, fixing it to match 'append' (default really is None) would be likely to break working code, so we certainly couldn't backport that, and I'd be really reluctant to fix it that way in 3.4, either.  (3) is thus preferable to (2), even though it leaves us with an inconsistency.  It would still be a judgment call on whether or not to backport it.  It seems like it would be reasonably safe, but such changes always worry me :)
msg179236 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2013-01-06 23:02
I agree it would be very likely to break working code.  Can you elaborate on your point about 'append' though?  I'm not sure I see it.

Aside from consistency, I'm wondering if there is ever a case where it would help to return None for positional arguments.  For example, unlike with optional arguments, it doesn't seem like it would ever make sense to distinguish between the option being present and the option being present with no values (which is why const is needed in the optional case).  In other words, there is no loss of information by returning [].
msg179237 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-07 00:51
>>> p.add_argument('-a', action="append", default=None)
>>> p.parse_args([])
Namespace(a=None)
>>> p.parse_args(['-a', '1', '-a', '2'])
Namespace(a=['1', '2'])

So there's a logical correspondence there (repeated option vs multiple values, each producing a list).

I'm not sure what you mean by the difference between options and positionals.  I note that you can use nargs with an optional, and in that case the default works, which is even more unfortunate than the difference with 'append'.  But if you can articulate a logical difference between optionals and positional here, maybe we can make it look reasonable :)
msg179239 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2013-01-07 01:47
I was referring to the fact that optionals have an additional case that positionals don't have: "Note that for optional arguments, there is an additional case -- the option string is present but not followed by a command-line argument."

(from http://docs.python.org/dev/library/argparse.html#nargs )

>>> p.add_argument('--foo', nargs='*', default=None)
>>> p.parse_args([])
Namespace(foo=None)
>>> p.parse_args(['--foo'])
Namespace(foo=[])

So it could be argued that positionals (at least by default) are behaving like the second case.  But that's as far as the parallel goes apparently.  *default* affects the first case and not the second case for optional arguments:

>>> p.add_argument('--foo', nargs='*', default=False)
>>> p.parse_args([])
Namespace(foo=False)
>>> p.parse_args(['--foo'])
Namespace(foo=[])
msg179289 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2013-01-07 22:29
Attached is a doc patch.  I also improved some other aspects of the *default* section while I was there.

We should probably make sure a test exists for the newly-documented behavior (i.e. for passing no arguments for a positional argument with nargs='*' and default=None).
msg179511 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2013-01-10 02:25
It turns out that there is already a test case:

http://hg.python.org/cpython/file/05183ce544be/Lib/test/test_argparse.py#l799

(at the line "('', NS(foo=[])),")

I've updated the patch with a note to reflect this.
msg187146 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2013-04-17 07:11
In this example:

    >>> p.add_argument('--foo', nargs='*', default=None)
    >>> p.parse_args([])
    Namespace(foo=None)
    >>> p.parse_args(['--foo'])
    Namespace(foo=[])

'p.parse_args([])' just assigns the default to 'foo' in the Namespace.

"p.parse_args(['--foo'])" invokes 'take_action(<dest='foo'>,[]).  That is, it 'assigns' an empty array to 'foo'.   The same thing would happen if 'foo' was a positional.  

'take_action' then passes these arguments to '_get_values'.  That is where the differences between '?' and '*' arise.

The key pieces of code in '_get_values' when arg_strings==[] are:

        # optional argument produces a default when not present
        if not arg_strings and action.nargs == OPTIONAL:
            ....value = action.default
            # and evaluate 'value' if is a string

        # when nargs='*' on a positional, if there were no command-line
        # args, use the default if it is anything other than None
        elif (not arg_strings and action.nargs == ZERO_OR_MORE ...):
            if action.default is not None:
                value = action.default
            else:
                value = arg_strings # i.e. []

In other words, if nargs='?', the attribute gets its default value.  But for '*', this is true only if the default is not None.

So in:

parse([], nargs='?')                # get the default value: None
parse([], nargs='*')                # default is None, get arg_strings []
parse([], nargs='*', default=None)  # same case
parse([], nargs='*', default=False) # default is not None, get default
parse([], nargs='*', default=0)     # same case

I tried changing the _get_values() so '*' got the default (like '?' does), and got 54 failures when running test_argparse.py.
msg198035 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2013-09-18 22:16
On a related point, the 'action.required' value is set differently for '?' and '*' positionals.  

    if kwargs.get('nargs') not in [OPTIONAL, ZERO_OR_MORE]:
        kwargs['required'] = True
    if kwargs.get('nargs') == ZERO_OR_MORE and 'default' not in kwargs:
        kwargs['required'] = True

OPTIONAL is always not required, ZERO_OR_MORE is not required if it has a default.  But for reasons discussed here, that 'required' value makes little difference.  

`parse_args` checks that all 'required' arguments have been seen, but a ZERO_OR_MORE positional is always seen (i.e. it matches an empty string).  

Usage formatting always uses '[%s [%s ...]]' with ZERO_OR_MORE, regardless of the 'required' attribute.

The only place where this 'required' value seems to matter is when adding such an argument to a mutually exclusive group.  But if an unused '*' positional is going to get a '[]' value anyways, why should it be excluded from such a use?

If I remove the 

    if kwargs.get('nargs') == ZERO_OR_MORE and 'default' not in kwargs:

test, test_argparse.py still runs fine.

http://bugs.python.org/issue18943 is a possibly related issue, involving  a 'is not action.default' test in a mutually exclusive group.
msg223160 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2014-07-15 23:53
The documentation patch here should note that a positional '*' string default is not parsed (i.e. does not pass through _get_value).

(see http://bugs.python.org/issue17250)
History
Date User Action Args
2018-10-16 17:01:51sebixsetnosy: + sebix
2014-07-15 23:53:48paul.j3setmessages: + msg223160
2013-09-18 22:16:57paul.j3setmessages: + msg198035
2013-04-17 07:11:51paul.j3setnosy: + paul.j3
messages: + msg187146
2013-01-10 02:25:59chris.jerdoneksetfiles: + issue-16878-2.patch

messages: + msg179511
2013-01-07 22:29:31chris.jerdoneksetfiles: + issue-16878-1.patch
keywords: + patch
messages: + msg179289
2013-01-07 01:47:47chris.jerdoneksetmessages: + msg179239
2013-01-07 00:51:54r.david.murraysetmessages: + msg179237
2013-01-06 23:02:01chris.jerdoneksetmessages: + msg179236
2013-01-06 14:43:19r.david.murraysetnosy: + r.david.murray
messages: + msg179184
2013-01-06 09:41:39chris.jerdonekcreate