classification
Title: argparse: default args in mutually exclusive groups
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: bethard Nosy List: bethard, louielu, paul.j3, pitrou, talkless, wolma
Priority: normal Keywords: patch

Created on 2013-09-06 09:00 by arigo, last changed 2017-12-08 16:20 by talkless.

Files
File name Uploaded Description Edit
test_argparse.diff arigo, 2013-09-06 09:00 Patch for test_argparse.py (for trunk, but also applies on 2.7 head) review
patch_1.diff paul.j3, 2013-09-09 22:38 review
Messages (24)
msg197058 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2013-09-06 09:00
In argparse, default arguments have a strange behavior that shows up in mutually exclusive groups: specifying explicitly on the command-line an argument, but giving it its default value, is sometimes equivalent to not specifying the argument at all, and sometimes not.

See the attached test diff: it contains two apparently equivalent pieces of code, but one passes and one fails.  The difference is that, in CPython, int("42") is 42 but int("4200") is not 4200 (in the sense of the operator "is").

The line that uses "is" in this way is this line in argparse.py (line 1783 in 2.7 head):

            if argument_values is not action.default:
msg197121 - (view) Author: paul j3 (paul.j3) * Date: 2013-09-06 23:59
The patch isn't a good unittest case because it produces an Error, not a Failure.  It does, though, raise a valid question about how a Mutually_exclusive_group tests for the use of its arguments.

As you note, argparse does use the `is` test: `argument_values is not action.default`.  argument_values is the result of passing an argument_string through its 'type' function.

This reworks your test case a bit:

    group = parser.add_mutually_exclusive_group()
    group.add_argument('--foo', default='test')
    group.add_argument('--bar', type=int, default=256)
    group.add_argument('--baz', type=int, default=257)

'--foo test --baz 257' will give the `argument --foo: not allowed with argument --baz` error message, but '--foo test --baz 256' does not.

So which is right?  Should it complain because 2 exclusive arguments are being used?  Or should it be excused from complaining because the values match their defaults?

The other issue is whether the values really match the defaults or not.  With an `is` test, the `id`s must match. The ids for small integers match all the time, while ones >256 differ.

Strings might have the same id or not, depending on how they are created.  If I create `x='test'`, and `y='--foo test'.split()[1]`.  `x==y` is True, but `x is y` is False.  So '--foo test' argument_value does not match the 'foo.default'.

So large integers (>256) behave like strings when used as defaults in this situation.  It's the small integers that have unique ids, and hence don't trigger mutually_exclusive_group errors when they should.

This mutually_exclusive_group 'is' test might not be ideal (free from all ambiguities), but I'm not sure it needs to be changed.  Maybe there needs to be a warning in the docs about mutually_exclusive_groups and defaults other than None.
msg197128 - (view) Author: paul j3 (paul.j3) * Date: 2013-09-07 02:13
A further complication on this.  With the arguments I defined in the previous post

    p.parse_args('--foo test --baz 257'.split())

gives the mutually exclusive error message.  `sys.argv` does the same.

    p.parse_args(['--foo', 'test', '--baz', '257'])

does not give an error, because here the 'test' argument string is the same as the default 'test'.  So the m_x_g test thinks `--foo' is the default, and does not count as an input.

Usually in testing an argparse setup I use the list and split arguments interchangeably, but this shows they are not equivalent.
msg197141 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2013-09-07 07:02
Getting consistently one behavior or the other would be much better imho; I think it's wrong-ish to have the behavior depend uncontrollably on implementation details.  But I agree that it's slightly messy to declare which of the two possible fixes is the "right" one.  I'm slightly in favor of the more permissive solution ("--bar 42" equivalent to no arguments at all if 42 is the default) only because the other solution might break someone's existing code.  If I had no such backward-compatibility issue in mind, I'd vote for the other solution (you can't specify "--bar" with any value, because you already specified "--foo").
msg197161 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-09-07 13:39
> The patch isn't a good unittest case because it produces an Error, not 
> a Failure.

Please let's not be pedantic about what a "good unittest" is.
msg197225 - (view) Author: paul j3 (paul.j3) * Date: 2013-09-08 05:59
Changing the test from

    if argument_values is not action.default:

to 

    if argument_values is not action.default and \
        (action.default is None or argument_values != action.default):

makes the behavior more consistent.  Strings and large ints behave like small ints, matching the default and not counting as "present"

Simply using `argument_values != action.default` was not sufficient, since it raised errors in existing test cases (such as ones involving Nones).
msg197275 - (view) Author: paul j3 (paul.j3) * Date: 2013-09-08 16:12
A possibly unintended consequence to this `seen_non_default_actions` testing is that default values do not qualify as 'present' when testing for a required mutually exclusive group.

    p=argparse.ArgumentParser()
    g=p.add_mutually_exclusive_group(required=True)
    g.add_argument('--foo',default='test')
    g.add_argument('--bar',type=int,default=42)
    p.parse_args('--bar 42'.split())

raises an `error: one of the arguments --foo --bar is required`

In the original code

    p.parse_args('--foo test'.split())

does not raise an error because 'test' does not qualify as default.  But with the change I proposed, it does raise the error.

This issue may require adding a `failures_when_required` category to the test_argparse.py MEMixin class.  Currently nothing in test_argparse.py tests for this issue.

Note that this contrasts with the handling of ordinarily required arguments.

    p.add_argument('--baz',type=int,default=42,required=True)

'--baz 42' does not raise an error.  It is 'present' regardless of whether its value matches the default or not.

This argues against tightening the `seen_non_default_actions` test.  Because the current testing only catches a few defaults (None and small ints) it is likely that no user has come across the required group issue.  There might actually be fewer compatibility issues if we simply drop the default test (or limit it to the case where the default=None).
msg197290 - (view) Author: paul j3 (paul.j3) * Date: 2013-09-08 17:25
I should add that defaults with required arguments (or groups?) doesn't make much sense.  Still there's nothing in the code that prevents it.
msg197295 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2013-09-08 17:44
Fwiw I agree with you :-)  I'm just relaying a bug report that originates on PyPy (https://bugs.pypy.org/issue1595).
msg197336 - (view) Author: paul j3 (paul.j3) * Date: 2013-09-08 23:33
This `argument_values` comes from `_get_values()`.  Most of the time is derived from the `argument_strings`.  But in a few cases it is set to `action.default`, specifically when the action is an optional postional with an empty `argument_strings`.

test_argparse.TestMutuallyExclusiveOptionalAndPositional is such a case.  `badger` is an optional positional in a mutually exclusive group.  As such it can be 'present' without really being there (tricky).  Positionals are always processed - otherwise it raises an error.

If this is the case, what we need is a more reliable way of knowing whether `_get_values()` is doing this, one that isn't fooled by this small int caching.

We could rewrite the `is not` test as:

    if not argument_strings and action.nargs in ['*','?'] and  argument_values is action.default:
        pass # _get_values() has set: argument_values=action.default
    else:
        seen_non_default_actions.add(action)
        ...

is a little better, but still feels like a kludge.  Having `_get_values` return a flag that says "I am actually returning action.default" would be clearer, but, I think, too big of a change.
msg197350 - (view) Author: paul j3 (paul.j3) * Date: 2013-09-09 06:47
At the very least the `is not action.default` needs to be changed.  Else where in argparse `is` is only used with `None` or constant like `SUPPRESS`.  So using it with a user defined parameter is definitely not a good idea.

Possible variations on how `is` behaves across implementations (pypy, ironpython) only complicates the issue.  I'm also familiar with a Javascript translation of argparse (that uses its `!==` in this context).
msg197414 - (view) Author: paul j3 (paul.j3) * Date: 2013-09-09 22:38
This patch uses a narrow criteria - if `_get_values()` sets the value to `action.default`, then argument counts as 'not present'.  I am setting a `using_default` flag in `_get_values`, and return it for use by `take_action`.

In effect, the only change from previous behavior is that small ints (<257) now behave like large ints, strings and other objects.

It removes the nonstandard 'is not action.default' test, and should behave consistently across all platforms (including pypy).
msg197427 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2013-09-10 08:06
The patch looks good to me.  It may break existing code, though, as reported on https://bugs.pypy.org/issue1595.  I would say that it should only go to trunk.  We can always fix PyPy (at Python 2.7) in a custom manner, in a "bug-to-bug" compatibility mode.
msg211229 - (view) Author: paul j3 (paul.j3) * Date: 2014-02-14 17:57
This patch corrects the handling of `seen_non_default_action` in another case - a positional with '?' and `type=int` (or other conversion).

if

    parser.add_argument('badger', type=int, nargs='?', default=2) # or '2'

and the original test 'seen_non_default_actions' is:

    if argument_values is not action.default

'argument_values' will be an 'int' regardless of the default.  But it will pass the 'is' test with the (small) int default but not the string default.

With the patch proposed here, both defaults behave the same - 'badger' will not appear in 'seen_non_default_actions' if it did not occur in the argument_strings (i.e. match an empty string).

I may add case like this to `test_argparse.py` for this patch.
msg212693 - (view) Author: paul j3 (paul.j3) * Date: 2014-03-04 07:00
I need to tweak the last patch so 'using_default' is also set when an "nargs='*'" positional is set to the '[]' default.

             if action.default is not None:
                 value = action.default
    +            using_default = True
             else:
                 value = arg_strings
    +            using_default = True  # tweak
msg292305 - (view) Author: paul j3 (paul.j3) * Date: 2017-04-26 06:03
This came up again, http://bugs.python.org/issue30163

An optional with int type and small integer default.
msg292313 - (view) Author: Louie Lu (louielu) * Date: 2017-04-26 07:36
paul, will you work on this patch? or I can help this issue, too.
msg293931 - (view) Author: paul j3 (paul.j3) * Date: 2017-05-18 17:21
I haven't downloaded the development distribution to this computer, so can't write formal patches at this time.
msg299355 - (view) Author: paul j3 (paul.j3) * Date: 2017-07-27 23:16
Another manifestation of the complications in handling '?' positionals is in

http://bugs.python.org/issue28734

argparse: successive parsing wipes out nargs=? values
msg307710 - (view) Author: Vincas Dargis (talkless) Date: 2017-12-06 09:07
Any progress with this? I believe it would fix my use case:

```
import argparse
import pprint

parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group(required=True)

group.add_argument('--device-get-capabilities',
                   action='store_true',
                   help='Execute GetCapabilities action from ONVIF devicemgmt.wsdl')

group.add_argument('--ptz-absolute-move',
                   nargs=3,
                   metavar=('x', 'y', 'z'),
                   help='Execute AbsoluteMove action from ONVIF ptz.wsdl')

group.add_argument('--ptz-get-status',
                   metavar='MEDIA_PROFILE',
                   default='MediaProfile000',
                   help='Execute GetSatus action from ONVIF ptz.wsdl for a media profile (default=%(default)s)')

pprint.pprint(parser.parse_args(['--ptz-get-status']))
```

Outputs (using 3.6.3):

```
 python3 ./test-ex-group-with-defult.py 
usage: test-ex-group-with-defult.py [-h]
                                    (--device-get-capabilities | --ptz-absolute-move x y z | --ptz-get-status MEDIA_PROFILE)
test-ex-group-with-defult.py: error: argument --ptz-get-status: expected one argument
```

Are there know workarounds for this?
msg307758 - (view) Author: paul j3 (paul.j3) * Date: 2017-12-06 17:43
Did you copy the output right?  Testing your parser:

Without any arguments, I get the exclusive group error - the group is required:

0930:~/mypy/argdev$ python3 issue18943.py 
usage: issue18943.py [-h]
                     (--device-get-capabilities | --ptz-absolute-move x y z | --ptz-get-status MEDIA_PROFILE)
issue18943.py: error: one of the arguments --device-get-capabilities --ptz-absolute-move --ptz-get-status is required

0931:~/mypy/argdev$ python3 --version
Python 3.5.2

With one flag but not its argument, I get the error that you display.  That has nothing to do with the grouping.

0932:~/mypy/argdev$ python3 issue18943.py --ptz-get-status
usage: issue18943.py [-h]
                     (--device-get-capabilities | --ptz-absolute-move x y z | --ptz-get-status MEDIA_PROFILE)
issue18943.py: error: argument --ptz-get-status: expected one argument
msg307760 - (view) Author: Vincas Dargis (talkless) Date: 2017-12-06 17:58
On 2017-12-06 19:43, paul j3 wrote:
> With one flag but not its argument, I get the error that you display.  That has nothing to do with the grouping.
> 
> 0932:~/mypy/argdev$ python3 issue18943.py --ptz-get-status
> usage: issue18943.py [-h]
>                       (--device-get-capabilities | --ptz-absolute-move x y z | --ptz-get-status MEDIA_PROFILE)
> issue18943.py: error: argument --ptz-get-status: expected one argument

In my example I pasted, I had hardcoded arguments:

```
pprint.pprint(parser.parse_args(['--ptz-get-status']))
``

I expected `python myscript.py --ptz-get-status` to work, because default value is set.

I do not compute that "With one flag but not its argument", sorry. It has default argument set, shoudn't that work?

Thanks!
msg307762 - (view) Author: paul j3 (paul.j3) * Date: 2017-12-06 18:28
That's not how flagged (optionals) arguments work.

The default value is used if the flag is not provided at all.  One of your arguments is a 'store_true'.  Its default value if False, which is changed to True if the '--device-get-capabilities' flag is provided.

"nargs='?'" provides a third option, assigning the 'const' value if the flag is used without an argument.

In any case your problem isn't with a required mutually exclusive group (defaults or not).  It has to do with understanding optionals and their defaults.
msg307852 - (view) Author: Vincas Dargis (talkless) Date: 2017-12-08 16:20
On 2017-12-06 20:28, paul j3 wrote:
> The default value is used *if the flag is not provided at all.*
> 
> "nargs='?'" provides a third option, assigning the 'const' value *if the flag is used without an argument*.

This did a "click" in my head. It works now with `nargs='?'` and `const='MediaProfile000'` as expected, thanks!

I am really sorry for the noise, due to misunderstanding while reading (skipping-throuhg?) Python documentation.
History
Date User Action Args
2017-12-08 16:20:17talklesssetmessages: + msg307852
2017-12-06 18:28:18paul.j3setmessages: + msg307762
2017-12-06 17:58:07talklesssetmessages: + msg307760
2017-12-06 17:43:10paul.j3setmessages: + msg307758
2017-12-06 09:07:03talklesssetnosy: + talkless
messages: + msg307710
2017-07-27 23:16:10paul.j3setmessages: + msg299355
2017-05-18 21:02:57arigosetnosy: - arigo
2017-05-18 17:21:22paul.j3setmessages: + msg293931
2017-04-27 10:06:35berker.peksaglinkissue30163 superseder
2017-04-26 07:54:51rhettingersetpriority: high -> normal
assignee: bethard
2017-04-26 07:36:08louielusetversions: + Python 3.7
nosy: + louielu

messages: + msg292313

type: behavior
2017-04-26 06:03:51paul.j3setpriority: normal -> high

messages: + msg292305
2016-11-22 16:56:24wolmasetnosy: + wolma
2014-03-04 07:00:30paul.j3setmessages: + msg212693
2014-02-14 17:57:45paul.j3setmessages: + msg211229
2013-09-10 08:06:01arigosetmessages: + msg197427
2013-09-09 22:38:23paul.j3setfiles: + patch_1.diff
keywords: + patch
messages: + msg197414
2013-09-09 06:47:08paul.j3setmessages: + msg197350
2013-09-08 23:33:41paul.j3setmessages: + msg197336
2013-09-08 17:44:06arigosetmessages: + msg197295
2013-09-08 17:25:09paul.j3setmessages: + msg197290
2013-09-08 16:12:22paul.j3setmessages: + msg197275
2013-09-08 05:59:47paul.j3setmessages: + msg197225
2013-09-07 13:39:59pitrousetnosy: + pitrou
messages: + msg197161
2013-09-07 07:02:53arigosetmessages: + msg197141
2013-09-07 06:15:56terry.reedysetnosy: + bethard
2013-09-07 02:13:38paul.j3setmessages: + msg197128
2013-09-06 23:59:44paul.j3setnosy: + paul.j3
messages: + msg197121
2013-09-06 09:12:01arigosetcomponents: + Library (Lib)
2013-09-06 09:01:37arigosetkeywords: - patch
2013-09-06 09:00:02arigocreate