In argparse, default arguments have a strange behavior that shows up in mutually exclusive groups: specifying explicitly on the command-line an argument, but giving it its default value, is sometimes equivalent to not specifying the argument at all, and sometimes not.

See the attached test diff: it contains two apparently equivalent pieces of code, but one passes and one fails.  The difference is that, in CPython, int("42") is 42 but int("4200") is not 4200 (in the sense of the operator "is").

The line that uses "is" in this way is this line in argparse.py (line 1783 in 2.7 head):

            if argument_values is not action.default:

The patch isn't a good unittest case because it produces an Error, not a Failure.  It does, though, raise a valid question about how a Mutually_exclusive_group tests for the use of its arguments.

As you note, argparse does use the `is` test: `argument_values is not action.default`.  argument_values is the result of passing an argument_string through its 'type' function.

This reworks your test case a bit:

    group = parser.add_mutually_exclusive_group()
    group.add_argument('--foo', default='test')
    group.add_argument('--bar', type=int, default=256)
    group.add_argument('--baz', type=int, default=257)

'--foo test --baz 257' will give the `argument --foo: not allowed with argument --baz` error message, but '--foo test --baz 256' does not.

So which is right?  Should it complain because 2 exclusive arguments are being used?  Or should it be excused from complaining because the values match their defaults?

The other issue is whether the values really match the defaults or not.  With an `is` test, the `id`s must match. The ids for small integers match all the time, while ones >256 differ.

Strings might have the same id or not, depending on how they are created.  If I create `x='test'`, and `y='--foo test'.split()[1]`.  `x==y` is True, but `x is y` is False.  So '--foo test' argument_value does not match the 'foo.default'.

So large integers (>256) behave like strings when used as defaults in this situation.  It's the small integers that have unique ids, and hence don't trigger mutually_exclusive_group errors when they should.

This mutually_exclusive_group 'is' test might not be ideal (free from all ambiguities), but I'm not sure it needs to be changed.  Maybe there needs to be a warning in the docs about mutually_exclusive_groups and defaults other than None.

A further complication on this.  With the arguments I defined in the previous post

    p.parse_args('--foo test --baz 257'.split())

gives the mutually exclusive error message.  `sys.argv` does the same.

    p.parse_args(['--foo', 'test', '--baz', '257'])

does not give an error, because here the 'test' argument string is the same as the default 'test'.  So the m_x_g test thinks `--foo' is the default, and does not count as an input.

Usually in testing an argparse setup I use the list and split arguments interchangeably, but this shows they are not equivalent.

Getting consistently one behavior or the other would be much better imho; I think it's wrong-ish to have the behavior depend uncontrollably on implementation details.  But I agree that it's slightly messy to declare which of the two possible fixes is the "right" one.  I'm slightly in favor of the more permissive solution ("--bar 42" equivalent to no arguments at all if 42 is the default) only because the other solution might break someone's existing code.  If I had no such backward-compatibility issue in mind, I'd vote for the other solution (you can't specify "--bar" with any value, because you already specified "--foo").

> The patch isn't a good unittest case because it produces an Error, not 
> a Failure.

Please let's not be pedantic about what a "good unittest" is.

Changing the test from

    if argument_values is not action.default:

to 

    if argument_values is not action.default and \
        (action.default is None or argument_values != action.default):

makes the behavior more consistent.  Strings and large ints behave like small ints, matching the default and not counting as "present"

Simply using `argument_values != action.default` was not sufficient, since it raised errors in existing test cases (such as ones involving Nones).

A possibly unintended consequence to this `seen_non_default_actions` testing is that default values do not qualify as 'present' when testing for a required mutually exclusive group.

    p=argparse.ArgumentParser()
    g=p.add_mutually_exclusive_group(required=True)
    g.add_argument('--foo',default='test')
    g.add_argument('--bar',type=int,default=42)
    p.parse_args('--bar 42'.split())

raises an `error: one of the arguments --foo --bar is required`

In the original code

    p.parse_args('--foo test'.split())

does not raise an error because 'test' does not qualify as default.  But with the change I proposed, it does raise the error.

This issue may require adding a `failures_when_required` category to the test_argparse.py MEMixin class.  Currently nothing in test_argparse.py tests for this issue.

Note that this contrasts with the handling of ordinarily required arguments.

    p.add_argument('--baz',type=int,default=42,required=True)

'--baz 42' does not raise an error.  It is 'present' regardless of whether its value matches the default or not.

This argues against tightening the `seen_non_default_actions` test.  Because the current testing only catches a few defaults (None and small ints) it is likely that no user has come across the required group issue.  There might actually be fewer compatibility issues if we simply drop the default test (or limit it to the case where the default=None).

I should add that defaults with required arguments (or groups?) doesn't make much sense.  Still there's nothing in the code that prevents it.

Fwiw I agree with you :-)  I'm just relaying a bug report that originates on PyPy (https://bugs.pypy.org/issue1595).

This `argument_values` comes from `_get_values()`.  Most of the time is derived from the `argument_strings`.  But in a few cases it is set to `action.default`, specifically when the action is an optional postional with an empty `argument_strings`.

test_argparse.TestMutuallyExclusiveOptionalAndPositional is such a case.  `badger` is an optional positional in a mutually exclusive group.  As such it can be 'present' without really being there (tricky).  Positionals are always processed - otherwise it raises an error.

If this is the case, what we need is a more reliable way of knowing whether `_get_values()` is doing this, one that isn't fooled by this small int caching.

We could rewrite the `is not` test as:

    if not argument_strings and action.nargs in ['*','?'] and  argument_values is action.default:
        pass # _get_values() has set: argument_values=action.default
    else:
        seen_non_default_actions.add(action)
        ...

is a little better, but still feels like a kludge.  Having `_get_values` return a flag that says "I am actually returning action.default" would be clearer, but, I think, too big of a change.

At the very least the `is not action.default` needs to be changed.  Else where in argparse `is` is only used with `None` or constant like `SUPPRESS`.  So using it with a user defined parameter is definitely not a good idea.

Possible variations on how `is` behaves across implementations (pypy, ironpython) only complicates the issue.  I'm also familiar with a Javascript translation of argparse (that uses its `!==` in this context).

This patch uses a narrow criteria - if `_get_values()` sets the value to `action.default`, then argument counts as 'not present'.  I am setting a `using_default` flag in `_get_values`, and return it for use by `take_action`.

In effect, the only change from previous behavior is that small ints (<257) now behave like large ints, strings and other objects.

It removes the nonstandard 'is not action.default' test, and should behave consistently across all platforms (including pypy).

The patch looks good to me.  It may break existing code, though, as reported on https://bugs.pypy.org/issue1595.  I would say that it should only go to trunk.  We can always fix PyPy (at Python 2.7) in a custom manner, in a "bug-to-bug" compatibility mode.

This patch corrects the handling of `seen_non_default_action` in another case - a positional with '?' and `type=int` (or other conversion).

if

    parser.add_argument('badger', type=int, nargs='?', default=2) # or '2'

and the original test 'seen_non_default_actions' is:

    if argument_values is not action.default

'argument_values' will be an 'int' regardless of the default.  But it will pass the 'is' test with the (small) int default but not the string default.

With the patch proposed here, both defaults behave the same - 'badger' will not appear in 'seen_non_default_actions' if it did not occur in the argument_strings (i.e. match an empty string).

I may add case like this to `test_argparse.py` for this patch.

I need to tweak the last patch so 'using_default' is also set when an "nargs='*'" positional is set to the '[]' default.

             if action.default is not None:
                 value = action.default
    +            using_default = True
             else:
                 value = arg_strings
    +            using_default = True  # tweak

This came up again, http://bugs.python.org/issue30163

An optional with int type and small integer default.

paul, will you work on this patch? or I can help this issue, too.

I haven't downloaded the development distribution to this computer, so can't write formal patches at this time.

Another manifestation of the complications in handling '?' positionals is in

http://bugs.python.org/issue28734

argparse: successive parsing wipes out nargs=? values

Any progress with this? I believe it would fix my use case:

```
import argparse
import pprint

parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group(required=True)

group.add_argument('--device-get-capabilities',
                   action='store_true',
                   help='Execute GetCapabilities action from ONVIF devicemgmt.wsdl')

group.add_argument('--ptz-absolute-move',
                   nargs=3,
                   metavar=('x', 'y', 'z'),
                   help='Execute AbsoluteMove action from ONVIF ptz.wsdl')

group.add_argument('--ptz-get-status',
                   metavar='MEDIA_PROFILE',
                   default='MediaProfile000',
                   help='Execute GetSatus action from ONVIF ptz.wsdl for a media profile (default=%(default)s)')

pprint.pprint(parser.parse_args(['--ptz-get-status']))
```

Outputs (using 3.6.3):

```
 python3 ./test-ex-group-with-defult.py 
usage: test-ex-group-with-defult.py [-h]
                                    (--device-get-capabilities | --ptz-absolute-move x y z | --ptz-get-status MEDIA_PROFILE)
test-ex-group-with-defult.py: error: argument --ptz-get-status: expected one argument
```

Are there know workarounds for this?

Did you copy the output right?  Testing your parser:

Without any arguments, I get the exclusive group error - the group is required:

0930:~/mypy/argdev$ python3 issue18943.py 
usage: issue18943.py [-h]
                     (--device-get-capabilities | --ptz-absolute-move x y z | --ptz-get-status MEDIA_PROFILE)
issue18943.py: error: one of the arguments --device-get-capabilities --ptz-absolute-move --ptz-get-status is required

0931:~/mypy/argdev$ python3 --version
Python 3.5.2

With one flag but not its argument, I get the error that you display.  That has nothing to do with the grouping.

0932:~/mypy/argdev$ python3 issue18943.py --ptz-get-status
usage: issue18943.py [-h]
                     (--device-get-capabilities | --ptz-absolute-move x y z | --ptz-get-status MEDIA_PROFILE)
issue18943.py: error: argument --ptz-get-status: expected one argument

On 2017-12-06 19:43, paul j3 wrote:
> With one flag but not its argument, I get the error that you display.  That has nothing to do with the grouping.
> 
> 0932:~/mypy/argdev$ python3 issue18943.py --ptz-get-status
> usage: issue18943.py [-h]
>                       (--device-get-capabilities | --ptz-absolute-move x y z | --ptz-get-status MEDIA_PROFILE)
> issue18943.py: error: argument --ptz-get-status: expected one argument

In my example I pasted, I had hardcoded arguments:

```
pprint.pprint(parser.parse_args(['--ptz-get-status']))
``

I expected `python myscript.py --ptz-get-status` to work, because default value is set.

I do not compute that "With one flag but not its argument", sorry. It has default argument set, shoudn't that work?

Thanks!

That's not how flagged (optionals) arguments work.

The default value is used if the flag is not provided at all.  One of your arguments is a 'store_true'.  Its default value if False, which is changed to True if the '--device-get-capabilities' flag is provided.

"nargs='?'" provides a third option, assigning the 'const' value if the flag is used without an argument.

In any case your problem isn't with a required mutually exclusive group (defaults or not).  It has to do with understanding optionals and their defaults.

On 2017-12-06 20:28, paul j3 wrote:
> The default value is used *if the flag is not provided at all.*
> 
> "nargs='?'" provides a third option, assigning the 'const' value *if the flag is used without an argument*.

This did a "click" in my head. It works now with `nargs='?'` and `const='MediaProfile000'` as expected, thanks!

I am really sorry for the noise, due to misunderstanding while reading (skipping-throuhg?) Python documentation.