This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: argparse set_defaults on subcommands should override top level set_defaults
Type: behavior Stage: needs patch
Components: Library (Lib) Versions: Python 3.4, Python 3.5, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Changaco, bethard, ezio.melotti, mikn, nailor, paul.j3, petri.lehtinen, python-dev, r.david.murray, remram, shihai1991, smparkes
Priority: normal Keywords: patch

Created on 2010-07-23 14:06 by bethard, last changed 2022-04-11 14:57 by admin.

File name Uploaded Description Edit
issue9351.patch nailor, 2012-10-30 13:01
Messages (21)
msg111324 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2010-07-23 14:06
If you use set_defaults on a subparser, but a default exists on the top level parser, the subparser defaults are ignored:

>>> parser = argparse.ArgumentParser()
>>> xparser = parser.add_subparsers().add_parser('X')
>>> parser.set_defaults(foo=1)
>>> xparser.set_defaults(foo=2)
>>> parser.parse_args(['X'])

This is counter to what people probably expect, that the subparser, when selected, would override the top level parser.

The behavior is because of the following code in parse_known_args:

        for dest in self._defaults:
            if not hasattr(namespace, dest):
                setattr(namespace, dest, self._defaults[dest])

This happens before the subparser sees the namespace object, and so the subparser sees that no defaults need to be filled in.
msg173602 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2012-10-23 12:13
I've attached a proposed fix where the subparser parses to an empty namespace and that namespace is then merged with the parent namespace
msg174147 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2012-10-29 19:39
+        for key in vars(subnamespace):
+            setattr(namespace, key, getattr(subnamespace, key))

There might be even more clever ways to achieve this, but what about at least saying "for key, value in vars(subnamespace).items()", and then using the value accordingly, to avoid the getattr() call?
msg174206 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2012-10-30 13:00
Yeah, I tried figuring out something more clever, as this, in the current form, has a bit too hackish feeling in it, but I couldn't find a proper tool for the job.

Anyway, attached a patch with the getattr removed.
msg174208 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2012-10-30 13:12
LGTM. Steven?
msg206993 - (view) Author: Mikael Knutsson (mikn) Date: 2013-12-27 14:04
Just wanted to drop in here to let you know that I hit this behaviour recently when writing a small tool using both configparser and argparse and the workaround proves rather annoying (custom namespace object or similar).

Would be awesome if this moved forward with the proposed patch!
msg219332 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2014-05-29 04:39
This is not getting much attention for several reasons:

- there's quite a backlog of argparse patches and issues

- 'set_defaults' is not commonly used.  Setting default in 'add_argument' seems more common.

- defining the same argument for both the parser and subparser can create other difficulties.

- in Bethard's example, 'set_default' sets an attribute that has no connection to any argument.  It's a cleaver trick that few users will think to use, and probably won't be of much value.

The idea proposed here of using a subnamespace for the subparser is interesting.  It reminds me of a StackOverflow question about implementing nested namespaces.
msg229619 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-10-18 00:07
New changeset e9cb45ccf42b by R David Murray in branch '3.4':
#9351: set_defaults on subparser is no longer ignored if set on parent.

New changeset b35a811d4420 by R David Murray in branch 'default':
Merge: #9351: set_defaults on subparser is no longer ignored if set on parent.

New changeset 1a3143752db2 by R David Murray in branch '2.7':
#9351: set_defaults on subparser is no longer ignored if set on parent.
msg229620 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-10-18 00:08
Thanks, Jyrki.
msg229968 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2014-10-24 23:24
A possible problem with this patch is that it overrides any Namespace given by the user.  In the test example:

    parser.parse_args(['X'], namespace=argparse.Namespace(foo=3))

the result is 'foo=2', the setdefault from the subparser.  Previously, and with a single level parser, the result would be 'foo=3'.  This is also true for an ordinary argument default in the subparser.

It could be argued that a user is unlikely to use both the 'namespace=...' parameter, and 'setdefault' for the same variable, especially one that isn't an argument 'dest'.  But the fact that the  custom namespace does not override regular subparser argument defaults bothers me.

Also, should an untested change like this be applied to 3.4 and 2.7?  This is not a backward compatible bug fix.

The test case also touches on a real bug in the recent releases - subparsers are now optional.  'parser.parse_args([])' runs without complaint about the missing 'X'.  Actually it is convenient for testing this case.  But it is still an incompatibility with earlier behavior
msg229983 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-10-25 05:02
Yeah, I hesitated a bit about the backports, but didn't visualize the scenario you are suggesting and thought it would be safe.  Perhaps it should be backed out of 2.7 and 3.4.

For 3.5, do you have any thoughts about how to make namespace= play nicely with this problem?
msg230056 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2014-10-27 02:25
I'm exploring modifying that patch by adding a 'subnamespace' attribute to the subparsers Action object.  This would give the user, or the main parser control over the namespace that is passed to a subparser.

For example: 

        subnamespace = getattr(self, 'subnamespace', False)
        if subnamespace is None:
        elif not  subnamespace:
            subnamespace = namespace
        # if None, subparser will use a fresh one
        subnamespace, arg_strings = parser.parse_known_args(arg_strings, subnamespace)
        for key, value in vars(subnamespace).items():
            setattr(namespace, key, value)

As written here, no value (or any False) would use the main parser namespace, as done with existing code.

A 'None' value, would use the a new empty namespace, as proposed in this patch.

Or the user could directly define the namespace, e.g.

    sp = parser.add_subparsers()
    sp.subnamespace = Namespace(foo=4)

That could also be convenient if the user is using a custom Namespace class.

'parse_known_args' could also set this attribute, based on whether the user has given it a namespace or not.
msg230084 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-10-27 18:43
If I understand you correctly, that would mean that if the namespace keyword is not used, we'd have the fixed behavior, but if the namespace keyword is used, we'd have the backward compatible behavior?  If I'm understanding correctly, that sounds like a good solution to me (coupled with backing this out of the maint versions).
msg233041 - (view) Author: (Changaco) Date: 2014-12-23 10:49
The broken patch made it into the 2.7.9 release, causing argparse to silently ignore arguments, cf
msg244768 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2015-06-03 16:29
Another example of this patch causing backward compatibility problems
msg244786 - (view) Author: Rémi Rampin (remram) * Date: 2015-06-03 18:35
To me this is much more than a compatibility problem. The way it worked before made a lot of sense, and just felt like the "correct" solution to accept a flag in multiple places.

Having a --verbose flag is something everybody should consider (Python has a decent builtin logging module), and anybody providing it would definitely want to accept it before and after subcommands (or at least, for every subcommand).

The only way right now is to not only create different arguments with add_argument(), for each parser, but you also need to provide different destination names (and then do something shitty like verbosity = args.verb_main+args.verb_subcommand). This bug makes argparse completely unusable for any real-life application that uses subparsers (in addition to breaking existing programs). And it breaks silently too, simply amazing!

Of course there is very little point in fixing this now. Since this affects multiple released versions of Python, I have to use a work-around anyway (until I can move from argparse to something that won't decide to break someday for the hell of it).
msg274369 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2016-09-04 16:45
In I've demonstrated how a subclass of _SubParsersAction can be used to revert the behavior to pre 2.7.9, passing the main namespace to the subparser.

The only other change is to the parser registry:

    parser.register('action', 'parsers', MyParserAction)

So there's no need to change the file itself.
msg342154 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2019-05-11 03:39
A variation on the problem I reported in

is that a custom Namespace class as documented in

will not be used by the subparsers.  Only the main parser uses that custom Namespace object; the subparsers continue to use the default 'argparse.Namespace()'.
msg350638 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2019-08-27 17:41
How about use a flag(such USING_OUT_NAMESPACE) to identify we use namespace or not?

For example:
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
for key, value in vars(subnamespace).items():
    if USING_OUT_NAMESPACE and not hasattr(namespace, key):
        setattr(namespace, key, value)
msg373665 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2020-07-15 01:49
I just realized if the subparser argument used 


it would not clobber values (default or user) set by the main parser.

(a 'store_true' would have to be replaced with a 'store_const' to do this)
msg405130 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2021-10-27 21:44
A new patch, has clobbered this patch.

It has also exposed the inadequate unittesting for the case(s) where the 'dest' of main namespace, subparser namespace, user provided namespace overlap.
Date User Action Args
2022-04-11 14:57:04adminsetgithub: 53597
2021-10-27 21:44:47paul.j3setmessages: + msg405130
2020-07-15 01:49:48paul.j3setmessages: + msg373665
2019-08-27 17:41:58shihai1991setnosy: + shihai1991
messages: + msg350638
2019-05-11 03:39:33paul.j3setmessages: + msg342154
2016-10-06 15:46:59ned.deilylinkissue27859 superseder
2016-09-04 16:46:44paul.j3setmessages: - msg274336
2016-09-04 16:45:54paul.j3setmessages: + msg274369
2016-09-04 04:24:40paul.j3setmessages: + msg274336
2015-06-03 18:35:56remramsetmessages: + msg244786
2015-06-03 16:35:31smparkessetnosy: + smparkes
2015-06-03 16:29:09paul.j3setmessages: + msg244768
2015-03-27 20:38:31remramsetnosy: + remram
2014-12-23 10:49:08Changacosetnosy: + Changaco
messages: + msg233041
2014-12-06 18:40:34r.david.murraysetresolution: fixed ->
stage: commit review -> needs patch
2014-10-27 18:43:22r.david.murraysetmessages: + msg230084
2014-10-27 02:25:38paul.j3setmessages: + msg230056
2014-10-25 05:03:27r.david.murraysetstage: resolved -> commit review
2014-10-25 05:02:56r.david.murraysetstatus: closed -> open

messages: + msg229983
2014-10-24 23:24:24paul.j3setmessages: + msg229968
2014-10-18 00:08:40r.david.murraysetstatus: open -> closed

versions: + Python 3.4, Python 3.5, - Python 3.2
nosy: + r.david.murray

messages: + msg229620
resolution: fixed
stage: commit review -> resolved
2014-10-18 00:07:35python-devsetnosy: + python-dev
messages: + msg229619
2014-05-29 04:39:15paul.j3setmessages: + msg219332
2014-05-28 05:53:36paul.j3setnosy: + paul.j3
2013-12-27 14:04:14miknsetnosy: + mikn
messages: + msg206993
2012-10-30 13:12:45petri.lehtinensetmessages: + msg174208
stage: needs patch -> commit review
2012-10-30 13:01:47nailorsetfiles: + issue9351.patch
2012-10-30 13:01:39nailorsetfiles: - issue9351.patch
2012-10-30 13:01:36nailorsetfiles: - issue9351.patch
2012-10-30 13:01:17nailorsetfiles: - issue9351.patch
2012-10-30 13:01:12nailorsetfiles: + issue9351.patch
2012-10-30 13:00:45nailorsetfiles: + issue9351.patch

messages: + msg174206
2012-10-29 19:39:38petri.lehtinensetmessages: + msg174147
2012-10-24 21:25:40ezio.melottisetnosy: + ezio.melotti
2012-10-23 12:13:24nailorsetfiles: + issue9351.patch

nosy: + nailor, petri.lehtinen
messages: + msg173602

keywords: + patch
2010-07-23 14:06:50bethardcreate