classification
Title: argparse: append action with default list adds to list instead of overriding
Type: behavior Stage: patch review
Components: Documentation, Library (Lib) Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Gabriel Devenyi, Markus.Amalthea.Magnuson, SylvainDe, Yclept.Nemo, bethard, docs@python, gun146, michal.klich, paul.j3, r.david.murray, rhettinger, roysmith, shihai1991
Priority: normal Keywords: patch

Created on 2012-11-04 01:30 by Markus.Amalthea.Magnuson, last changed 2020-01-02 11:22 by shihai1991.

Files
File name Uploaded Description Edit
argparse_foo_test.py Markus.Amalthea.Magnuson, 2012-11-04 01:30 Test script demonstrating the described behavior.
append_to_default.patch michal.klich, 2016-10-02 15:16 patch documenting behaviour review
Pull Requests
URL Status Linked Edit
PR 17793 open shihai1991, 2020-01-02 05:14
Messages (14)
msg174735 - (view) Author: Markus Amalthea Magnuson (Markus.Amalthea.Magnuson) Date: 2012-11-04 01:30
If the default value for a flag is a list, and the action append is used, argparse doesn't seem to override the default, but instead adding to it. I did this test script:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument(
    '--foo',
    action='append',
    default=['bar1', 'bar2']
)
args = parser.parse_args()

print args.foo

Output is as follows:

$ ./argparse_foo_test.py
['bar1', 'bar2']

$ ./argparse_foo_test.py --foo bar3
['bar1', 'bar2', 'bar3']

I would expect the last output to be ['bar3'].

Is this on purpose (although very confusing) or is it a bug?
msg174757 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-11-04 05:07
This behavior is inherited from optparse.  I think it is more-or-less intentional, and in any case it is of long enough standing that I don't think we can change it.  We documented it for optparse in another issue, but I don't think we made the corresponding improvement to the argparse docs.
msg185994 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2013-04-04 02:21
The test file, test_argparse.py, has a test case for this:
'class TestOptionalsActionAppendWithDefault'

    argument_signatures = [Sig('--baz', action='append', default=['X'])]
    successes = [
        ('--baz a --baz b', NS(baz=['X', 'a', 'b'])),
    ]
msg220969 - (view) Author: SylvainDe (SylvainDe) * Date: 2014-06-19 11:23
As this is likely not to get solved, is there a recommanded way to work around this issue ?

Here's what I have done :

  import argparse
  def main():
      """Main function"""
      parser = argparse.ArgumentParser()
      parser.add_argument('--foo', action='append')
      for arg_str in ['--foo 1 --foo 2', '']:
          args = parser.parse_args(arg_str.split())
          if not args.foo:
              args.foo = ['default', 'value']
          print(args)

printing

  Namespace(foo=['1', '2'])
  Namespace(foo=['default', 'value'])

as expected but I wanted to know if there a more argparse-y way to do this. I have tried using `set_defaults` without any success.

Also, as pointed out the doc for optparse describes the behavior in a simple way : "The append action calls the append method on the current value of the option. This means that any default value specified must have an append method. It also means that if the default value is non-empty, the default elements will be present in the parsed value for the option, with any values from the command line appended after those default values".
msg221005 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2014-06-19 16:22
It should be easy to write a subclass of Action, or append Action, that does what you want.  It just needs a different `__call__` method.  You just need a way of identifying an default that needs to be overwritten as opposed to appended to.


    def __call__(self, parser, namespace, values, option_string=None):
        current_value = getattr(namspace, self.dest)
        if 'current_value is default':
            setattr(namespace, self.dest, values)
            return
        else:
            # the normal append action
            items = _copy.copy(_ensure_value(namespace, self.dest, []))
            items.append(values)
            setattr(namespace, self.dest, items)

People on StackOverFlow might have other ideas.
msg224964 - (view) Author: Yclept Nemo (Yclept.Nemo) Date: 2014-08-06 19:30
Well that won't work. Example:

import argparse

class TestAction(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        print("default: {}({})\tdest: {}({})".format(self.default, type(self.default), getattr(namespace, self.dest), type(getattr(namespace, self.dest))))
        if getattr(namespace, self.dest) is self.default:
            print("Replacing with: ", values)
            setattr(namespace, self.dest, values)
        # extra logical code not necessary for testcase

parser = argparse.ArgumentParser()

parser.add_argument\
        ( "-o", "--output"
        , type=int
        , action=TestAction
        , default=42
        )

args = parser.parse_args()

$ ./argparse_test -o 42 -o 100
default: 42(<class 'int'>)      dest: 42(<class 'int'>)
Replacing with:  42
default: 42(<class 'int'>)      dest: 42(<class 'int'>)
Replacing with:  100
100

Use this approach:
class ExtendAction(argparse.Action):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if not isinstance(self.default, collections.abc.Iterable):
            self.default = [self.default]
        self.reset_dest = False
    def __call__(self, parser, namespace, values, option_string=None):
        if not self.reset_dest:
            setattr(namespace, self.dest, [])
            self.reset_dest = True
        getattr(namespace, self.dest).extend(values)

Anyway, this should be properly documented...
msg224969 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2014-08-06 22:13
In my suggestion I used 

    if 'current_value is default':

without going into detail.  The use of an 'is' test for integer values is tricky, as discussed in http://bugs.python.org/issue18943

    int("42") is 42    # True
    int("257") is 257  # False

As with your 'self.reset_dest', in 18943 I suggested using a boolean flag instead of the 'is' test.
msg263352 - (view) Author: Gabriel Devenyi (Gabriel Devenyi) Date: 2016-04-13 19:10
From what I can tell a workaround for this still isn't documented.
msg277895 - (view) Author: MichaƂ Klich (michal.klich) Date: 2016-10-02 15:16
The documentation for argparse still does not mention this behaviour.
I decided to make a patch based no the optparse issue.
Hopefully it is good enough to be merged.
msg277915 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2016-10-02 23:05
It may help to know something about how defaults are handled - in general.

`add_argument` and `set_defaults` set the `default` attribute of the Action (the object created by `add_argument` to hold all of its information).  The default `default` is `None`.

At the start of `parse_args`, a fresh Namespace is created, and all defaults are loaded into it (I'm ignoring some details).

The argument strings are then parsed, and individual Actions update the Namespace with their values, via their `__call__` method.

At the end of parsing it reviews the Namespace.  Any remaining defaults that are strings are evaluated (passed through `type` function that converts a commandline string).  The handling of defaults threads a fine line between giving you maximum power, and keeping things simple and predictable.

The important thing for this issue is that the defaults are loaded into the Namespace at the start of parsing.

The `append` call fetches the value from the Namespace, replaces it with `[]` if it is None, appends the new value(s), and puts it back on the Namespace.  The first `--foo` append is handled in just the same way as the 2nd and third (fetch, append, and put back).  The first can't tell that the list it fetches from the namespace came from the `default` as opposed to a previous `append`.  

The `__call__` for `append` was intentionally kept simple, and predictable.  As I demonstrated earlier it is possible to write an `append` that checks the namespace value against some default, and does something different.  But that is more complicated.

The simplest alternative to this behavior is to leave the default as None.  If after parsing the value is still None, put the desired list (or any other object) there.  

The primary purpose of the parser is to parse the commandline - to figure out what the user wants to tell you.  There's nothing wrong with tweaking (and checking) the `args` Namespace after parsing.
msg277919 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2016-10-03 00:31
One thing that this default behavior does is allow us to append values to any object, just so long as it has the `append` method.  The default does not have to be a standard list.

For example, in another bug/issue someone asked for an `extend` action.  I could provide that with `append` and a custom list class

    class MyList(list):
        def append(self,arg):
            if isinstance(arg,list):
                self.extend(arg)
            else:
                super(MyList, self).append(arg)

This just modifies `append` so that it behaves like `extend` when given a list argument.

     parser = argparse.ArgumentParser()
     a = parser.add_argument('-f', action='append', nargs='*',default=[])
     args = parser.parse_args('-f 1 2 3 -f 4 5'.split())

produces a nested list:

     In [155]: args
     Out[155]: Namespace(f=[['1', '2', '3'], ['4', '5']])

but if I change the `default`: 

     a.default = MyList([])
     args = parser.parse_args('-f 1 2 3 -f 4 5'.split())

produces a flat list:

     In [159]: args
     Out[159]: Namespace(f=['1', '2', '3', '4', '5'])

I've tested this idea with an `array.array` and `set` subclass.
msg322664 - (view) Author: Evgeny (gun146) Date: 2018-07-30 10:13
You don't need action='append'. 
For desired behavior you can pass action='store' with nargs='*'.
I think it's a simplest workaround.
msg358871 - (view) Author: Roy Smith (roysmith) Date: 2019-12-25 21:03
I just got bit by this in Python 3.5.3.

I get why it does this.  I also get why it's impractical to change the behavior now.  But, it really isn't the obvious behavior, so it should be documented at https://docs.python.org/3.5/library/argparse.html?highlight=argparse#default.
msg359190 - (view) Author: hai shi (shihai1991) * Date: 2020-01-02 11:22
I update the doc of argparse and think this bpo could be closed when PR merged.
History
Date User Action Args
2020-01-02 11:22:33shihai1991setnosy: + rhettinger
messages: + msg359190
2020-01-02 11:06:00shihai1991setnosy: + shihai1991
2020-01-02 05:14:33shihai1991setstage: patch review
pull_requests: + pull_request17226
2019-12-25 21:03:02roysmithsetnosy: + roysmith
messages: + msg358871
2018-07-30 10:13:05gun146setnosy: + gun146
messages: + msg322664
2016-10-03 00:31:40paul.j3setmessages: + msg277919
2016-10-02 23:05:19paul.j3setmessages: + msg277915
2016-10-02 15:16:57michal.klichsetfiles: + append_to_default.patch

nosy: + michal.klich
messages: + msg277895

keywords: + patch
2016-04-13 19:10:51Gabriel Devenyisetnosy: + Gabriel Devenyi
messages: + msg263352
2014-08-06 22:13:17paul.j3setmessages: + msg224969
2014-08-06 19:30:21Yclept.Nemosetnosy: + Yclept.Nemo
messages: + msg224964
2014-06-19 16:22:50paul.j3setmessages: + msg221005
2014-06-19 11:23:07SylvainDesetnosy: + SylvainDe
messages: + msg220969
2013-04-04 02:21:12paul.j3setnosy: + paul.j3
messages: + msg185994
2012-11-04 05:07:01r.david.murraysetversions: + Python 3.2, Python 3.3, Python 3.4
nosy: + r.david.murray, docs@python, bethard

messages: + msg174757

assignee: docs@python
components: + Documentation
2012-11-04 01:31:33Markus.Amalthea.Magnusonsettitle: argparse: -> argparse: append action with default list adds to list instead of overriding
2012-11-04 01:30:23Markus.Amalthea.Magnusoncreate