Issue33109
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2018-03-20 13:41 by wolma, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Pull Requests | |||
---|---|---|---|
URL | Status | Linked | Edit |
PR 6919 | merged | ned.deily, 2018-05-16 19:51 | |
PR 7089 | merged | miss-islington, 2018-05-24 01:56 | |
PR 7609 | merged | ned.deily, 2018-06-11 02:08 | |
PR 7610 | merged | miss-islington, 2018-06-11 02:19 |
Messages (27) | |||
---|---|---|---|
msg314145 - (view) | Author: Wolfgang Maier (wolma) * | Date: 2018-03-20 13:41 | |
I find the True default for 'required' quite cumbersome introduced as a result of issue 26510. With existing parsers it can unnecessarily break compatibility between Python3.x versions only to make porting a bit easier for Python2 users. I think, this late in the life cycle of Python2, within Python3 compatibility should be ranked higher than py2to3 portability. Command line parsing of a package of mine has long used optional subparsers (without me even thinking much about the fact). Now in 3.7, running python3.7 -m MyPackage without arguments (the parser is in __main__.py) I get the ill-formatted error message: __main__.py: error: the following arguments are required: while my code in 3.3 - 3.6 was catching the empty Namespace returned and printed a help message. Because the 'required' keyword argument did not exist in < 3.7 there was no simple way for me to write code that is compatible between all 3.x versions. What I ended up doing now is to check sys.argv before trying to parse things, then print the help message, when that only has a single item, just to keep my existing code working. OTOH, everything would be just fine with a default value of False. Also that truncated error message should be fixed before 3.7 gets released. |
|||
msg314147 - (view) | Author: Anthony Sottile (Anthony Sottile) * | Date: 2018-03-20 15:38 | |
The intention of the change in issue 26510 was to pick the least surprising behaviour for the default value of subparsers -- the compatiblity with the behaviour before the regression was introduced in 3.3 was a nice side-effect. As with the rest of positional arguments in argparse, the positional subparsers were changed to required by default. The main issue addressing the 3.3 regression I believe is https://bugs.python.org/issue9253 and not the one linked. When I revived the patch, I surveyed a number of open source tools using subparsers (~10-20) and they all fell into the following categories: - Used the workaround (part of this SO post: https://stackoverflow.com/a/23354355/812183) (most fell into this category) - crashed with some sort of TypeError (NoneType has no attribute startswith, KeyeError: None, etc.) due to not handling "optional" subparsers - Manually handled when the subparser was `None` to raise an argparse error You can enable a 3.3-3.7 compatible "always optional subparsers" with a similar pattern that was used to manually restore the pre-regression behaviour: subparsers = parser.add_subparsers(...) subparsers.required = False I believe the error message issue is already tracked: https://bugs.python.org/issue29298 |
|||
msg314148 - (view) | Author: Anthony Sottile (Anthony Sottile) * | Date: 2018-03-20 15:43 | |
Grabbed the wrong SO link, I believe this is the one I meant to link to: https://stackoverflow.com/a/18283730/812183 |
|||
msg314152 - (view) | Author: Wolfgang Maier (wolma) * | Date: 2018-03-20 16:25 | |
On 03/20/2018 04:38 PM, Anthony Sottile wrote: > > Anthony Sottile <asottile@umich.edu> added the comment: > > The intention of the change in issue 26510 was to pick the least surprising behaviour for the default value of subparsers -- the compatiblity with the behaviour before the regression was introduced in 3.3 was a nice side-effect. As with the rest of positional arguments in argparse, the positional subparsers were changed to required by default. > Since the 3.3 change happened a long time ago and has been kept through the next three releases, I never considered it a regression, but rather thought the original behavior was an oddity of early Python 3s (I've never written an argparse-based parser in Python 2), so it's interesting to see this in the larger historical context. Overall, I think "least surprising" is in the eye of the beholder here and I want to emphasize that I'm all for your change of adding the keyword argument, just not so convinced about your choice of the default. My main argument for a default of False and against True: having True as the default will only lead people used to the Python 2 and pre-3.3 way of things to think that they have code working for Python 3, when, in fact, that code will break under 3.3-3.6, and, at least, 3.6 will stay in widespread use for a long time (even Ubuntu 18.04 will still ship with it as the default python3, so Python3.6 will outlast the life cycle of Python 2 by a good measure). Conversely, most projects are dropping Python 3.2 support these days or have done so already, so nobody really cares about how things worked in this version (I think it's telling along these lines that your - corrected - SO link dates back from 2013). So I think, it is a rather unnecessary incompatibility you are introducing for project maintainers even if the original issue was a regression. > The main issue addressing the 3.3 regression I believe is https://bugs.python.org/issue9253 and not the one linked. > > When I revived the patch, I surveyed a number of open source tools using subparsers (~10-20) and they all fell into the following categories: > > - Used the workaround (part of this SO post: https://stackoverflow.com/a/23354355/812183) (most fell into this category) > - crashed with some sort of TypeError (NoneType has no attribute startswith, KeyeError: None, etc.) due to not handling "optional" subparsers > - Manually handled when the subparser was `None` to raise an argparse error As yet another option, and similar to the third one on your list, I'm using the set_defaults method of the subparser, and I'm checking whether the corresponding key is present in the Namespace. > > You can enable a 3.3-3.7 compatible "always optional subparsers" with a similar pattern that was used to manually restore the pre-regression behaviour: > > subparsers = parser.add_subparsers(...) > subparsers.required = False > Ah, right! That's a good option. Didn't realize it would work this way, too :) But a still think you should consider my above argument. > I believe the error message issue is already tracked: https://bugs.python.org/issue29298 > I see, that looks as if it would fix this part. It would be great if it could get merged into Python 3.7 still. > ---------- > > _______________________________________ > Python tracker <report@bugs.python.org> > <https://bugs.python.org/issue33109> > _______________________________________ > |
|||
msg314153 - (view) | Author: Anthony Sottile (Anthony Sottile) * | Date: 2018-03-20 16:31 | |
Yeah, I picked the default value `True` because I couldn't actually find a user of subparsers that _wanted_ optional subparsers. ¯\_(ツ)_/¯ |
|||
msg314156 - (view) | Author: Wolfgang Maier (wolma) * | Date: 2018-03-20 16:54 | |
_wants_ is a bit a strong word, but, at least, you can do a bit a nicer job than the default error, like printing a nicely formatted list of subcommands as you would get it with the main parsers help. In fact, in my own code I'm actually catching the missing subparser situation, then calling parse_args again with ['--help'] :) |
|||
msg316275 - (view) | Author: Brian Skinn (bskinn) * | Date: 2018-05-07 16:05 | |
I second Wolfgang's recommendation to change the default back to `False`. I started developing CLI apps &c. in Python ~4yrs ago; I dabbled briefly in 2.7, then switched firmly to Python 3. When I started, I was aimed at supporting 3.3 to 3.5; now I'm specifically supporting 3.4 to 3.6, but starting to test my code against the 3.7b versions. Thus, all I've ever known is the default `required=False` behavior. On my one tool currently using subparsers (https://github.com/bskinn/sphobjinv), I made the subparser choice optional, to allow a `sphobjinv --version` invocation. So, when 3.7 failed that test (https://github.com/bskinn/sphobjinv/blob/6c1f22e40dc3d129485462aec05adbed2ff40ab8/sphobjinv/test/sphobjinv_cli.py#L419-L422), it seemed like a regression to me. All that said, given that the `subparsers.required = False` fix is a clean, single line, I have no beef with a `True` default if that's preferred. However, please include this change in the 3.7 CHANGELOG, at least. |
|||
msg316701 - (view) | Author: Éric Araujo (eric.araujo) * | Date: 2018-05-15 19:18 | |
I’m sorry I don’t have the time to study this and make a judgment call. Bringing this to the release manager’s attention. |
|||
msg316720 - (view) | Author: Gregory P. Smith (gregory.p.smith) * | Date: 2018-05-15 20:41 | |
If the behavior was consistent from 3.3 through 3.6, that is the behavior we should keep going forward in 3.7+ without a deprecation period. (and this does not seem worth deprecating, lets just keep the behavior the same as it was in 3.3-3.6) That 2.7 is different than >=3.3 here isn't important. There are a lot of things that have conditional behavior differences when using 2 and over time that is becoming increasingly irrelevant. Libraries often keep compatibility with both 2.7 and 3.4+ today, but it is less common for a command line tool entry point to need to be compatible with both. |
|||
msg316721 - (view) | Author: Anthony Sottile (Anthony Sottile) * | Date: 2018-05-15 20:46 | |
According to the other bugs, the change in 3.3 was an inadvertent regression. The fact that it didn't get fixed for so long is mostly due to the unmaintained state of argparse in the stdlib. The change in behaviour here is the _fix_ of that regression. Consistency with the rest of the framework to me feels pretty important. subparsers are positional arguments and positional arguments by default are required. |
|||
msg316807 - (view) | Author: Wolfgang Maier (wolma) * | Date: 2018-05-16 14:03 | |
Try to think of it this way: By choosing a default of True, every new project with subparsers that aims for Python <3.7 compatibility will have to take some measures (either overwrite the default or special-case 3.3-3.6). So no matter whether this is the "least surprising" choice, it is an inconvenient one that makes the default almost useless for years to come. In the long term, when support for Python<=3.6 is finally not important anymore, you would get a slightly more consistent API (though I never thought of a subparser as a regular positional argument before this issue), but the price for it seems too high to me. Since backwards compatibility is easy to restore by overwriting the default, I can certainly live with the choice of True, but I think it's not the best one could get out of this new and useful keyword. |
|||
msg316858 - (view) | Author: Ned Deily (ned.deily) * | Date: 2018-05-16 20:06 | |
Several of the core developers here at the PyCon US sprints in Cleveland have discussed this issue. It seems like there legitimate arguments for either behavior. But, while none of us are argparse experts, we all were persuaded by Wolfgang's and Brian's arguments that preserving compatibility with 3.6 (and recent earlier 3.x releases) should be given more weight than attempting to restore 2.7 behavior at this point. We specifically did not get into the finer points of argparse behavior and any other proposed argparse change. Unless someone comes up with a more persuasive argument, I intend to merge this change for 3.7.0rc1 in a few days. |
|||
msg316860 - (view) | Author: Anthony Sottile (Anthony Sottile) * | Date: 2018-05-16 20:12 | |
Considering the huge popularity of these SO questions, I don't think this should be reverted: - https://stackoverflow.com/questions/23349349/argparse-with-required-subparser - https://stackoverflow.com/questions/22990977/why-does-this-argparse-code-behave-differently-between-python-2-and-3 See also: the confusion caused by the 3.3 change: https://bugs.python.org/issue9253 |
|||
msg317319 - (view) | Author: Ned Deily (ned.deily) * | Date: 2018-05-22 18:29 | |
> Considering the huge popularity of these SO questions, I don't think this should be reverted [...] As I understand it (and, again, I make no claim to be an argparse expert), there does not seem to be one absolutely correct answer here; there has to be a tradeoff. If we revert the change in default as in PR 6919, users porting code from 2.7 will continue to run into the unfortunate change in behavior introduced in 3.3. But, with the reversion, those users are no worse off than they were before: the existing workarounds, like those in the cited SO answers, still apply. And it's a one-time annoyance for them, along with all the other changes they may need to make to port to a current Python 3.x. Whereas, if the change is not reverted, then we introduce a new incompatibility to a new class of users, that is, those upgrading from Python 3.3 through 3.6 to 3.7, generating a new set of SO questions, etc. That seems to be making a less-than-ideal situation worse. So, as release manager, I continue to think that the reversion (PR 6919) should go in to 3.7.0. (For 3.8 and beyond, it would be great to have at least one core developer take responsibility for argparse enhancements.) |
|||
msg317321 - (view) | Author: Anthony Sottile (Anthony Sottile) * | Date: 2018-05-22 18:45 | |
Is there then no pathway for actually fixing the bug? aka how can I get `required=True` to be the default. |
|||
msg317325 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * | Date: 2018-05-22 19:02 | |
I tried to use add_subparsers() with required=True and have found it not usable. import argparse parser = argparse.ArgumentParser(prog='PROG') subparsers = parser.add_subparsers(required=True) parser_a = subparsers.add_parser('a') parser_b = subparsers.add_parser('b') parser.parse_args([]) The result: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/serhiy/py/cpython/Lib/argparse.py", line 1745, in parse_args args, argv = self.parse_known_args(args, namespace) File "/home/serhiy/py/cpython/Lib/argparse.py", line 1777, in parse_known_args namespace, args = self._parse_known_args(args, namespace) File "/home/serhiy/py/cpython/Lib/argparse.py", line 2012, in _parse_known_args ', '.join(required_actions)) TypeError: sequence item 0: expected str instance, NoneType found Seems that not only the default value should be changed, but the whole PR 3027 should be reverted. |
|||
msg317326 - (view) | Author: Anthony Sottile (Anthony Sottile) * | Date: 2018-05-22 19:08 | |
That's a separate issue (also a bug introduced by the bad 3.3 patch): https://bugs.python.org/issue29298 I have an open PR to fix it as well but it has not seen review action: https://github.com/python/cpython/pull/3680 |
|||
msg317329 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * | Date: 2018-05-22 19:16 | |
Wouldn't be better to first fix this bug, and only after that add the 'required' parameter? Adding it introduced yet one bug: when pass arguments as positional, the 'help' argument will be swallowed. You can add new parameters only after existing positional parameters. |
|||
msg317331 - (view) | Author: Anthony Sottile (Anthony Sottile) * | Date: 2018-05-22 19:21 | |
The bug is orthogonal, you can trigger it without the `required=` keyword argument via the (currently suggested) monkeypatch workaround which restores the pre-3.3 behaviour: import argparse parser = argparse.ArgumentParser() subp = parser.add_subparsers() subp.add_parser('test') subp.required = True parser.parse_args() $ python3 test.py Traceback (most recent call last): File "test.py", line 7, in <module> parser.parse_args() File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/argparse.py", line 1730, in parse_args args, argv = self.parse_known_args(args, namespace) File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/argparse.py", line 1762, in parse_known_args namespace, args = self._parse_known_args(args, namespace) File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/argparse.py", line 1997, in _parse_known_args ', '.join(required_actions)) TypeError: sequence item 0: expected str instance, NoneType found Also note that when `dest` is specified it works fine: import argparse parser = argparse.ArgumentParser() subp = parser.add_subparsers(dest='cmd') subp.add_parser('test') subp.required = True parser.parse_args() $ python3 test.py usage: test.py [-h] {test} ... test.py: error: the following arguments are required: cmd |
|||
msg317332 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * | Date: 2018-05-22 19:23 | |
Please ignore the last paragraph. It was my mistake, all add_subparsers() parameters are keyword-only, and _SubParsersAction is a privale class. |
|||
msg317499 - (view) | Author: Ned Deily (ned.deily) * | Date: 2018-05-24 01:55 | |
New changeset 8ebf5ceb0f5408d1ebc26c19702ac0762ef5ea04 by Ned Deily in branch 'master': bpo-33109: argparse subparsers are once again not required by default (GH-6919) https://github.com/python/cpython/commit/8ebf5ceb0f5408d1ebc26c19702ac0762ef5ea04 |
|||
msg317511 - (view) | Author: Ned Deily (ned.deily) * | Date: 2018-05-24 02:22 | |
New changeset dd7a255911f364cf521676082a89d4cac307737e by Ned Deily (Miss Islington (bot)) in branch '3.7': bpo-33109: argparse subparsers are once again not required by default (GH-6919) (GH-7089) https://github.com/python/cpython/commit/dd7a255911f364cf521676082a89d4cac307737e |
|||
msg317517 - (view) | Author: Ned Deily (ned.deily) * | Date: 2018-05-24 02:42 | |
> Is there then no pathway for actually fixing the bug? aka how can I get `required=True` to be the default. There may very well be but, unfortunately, dealing with this newly-identified 3.x compatibility issue takes precedence for 3.7.0. In general, for cpython, all other things being equal, the status quo wins and, in this case, that means not breaking 3.6->3.7 compatibility without a good reason. The underlying problem here, IMHO, is that we are essentially an all-volunteer project and at the moment argparse does not have an active core developer to review and champion change requests. If a core developer does want to take up the existing queue of argparse issues, we *might* come to the conclusion that making another incompatible change is the overall right thing to do. But, until that happens, the least bad option is to not make things worse. How to get more core developer interest in argparse issues is a whole 'nother matter and out-of-scope for this issue. Sorry, I wish I had a better answer. Thanks, Anthony and everyone else here, for your input. |
|||
msg319264 - (view) | Author: Ned Deily (ned.deily) * | Date: 2018-06-11 02:17 | |
New changeset ef057bfb06cae0718e6d708061649d2e3983e2ef by Ned Deily in branch 'master': bpo-33109: Remove now-obsolete What's New entry for bpo-26510. (GH-7609) https://github.com/python/cpython/commit/ef057bfb06cae0718e6d708061649d2e3983e2ef |
|||
msg319266 - (view) | Author: Ned Deily (ned.deily) * | Date: 2018-06-11 02:31 | |
New changeset a73399d5963d6b1639d935968f4a8baa868c39d3 by Ned Deily (Miss Islington (bot)) in branch '3.7': bpo-33109: Remove now-obsolete What's New entry for bpo-26510. (GH-7609) (GH-7610) https://github.com/python/cpython/commit/a73399d5963d6b1639d935968f4a8baa868c39d3 |
|||
msg394096 - (view) | Author: Lucas Cimon (Lucas Cimon) * | Date: 2021-05-21 07:02 | |
Reporting a duplicate / superseder with the following bug: parser = argparse.ArgumentParser() subparsers = parser.add_subparsers(required=True) subparsers.add_parser('foo') parser.parse_args() Raising: TypeError: sequence item 0: expected str instance, NoneType found It has already been reported in https://bugs.python.org/issue29298 with an interesting solution by Greg Minshall. |
|||
msg394097 - (view) | Author: Lucas Cimon (Lucas Cimon) * | Date: 2021-05-21 07:22 | |
Sorry, the fix was by Mathias Ettinger: elif isinstance(argument, _SubParsersAction): return '{%s}' % ','.join(map(str, argument.choices)) I submitted a PR with this patch and a corresponding unit test: https://github.com/python/cpython/pull/26278 |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:58 | admin | set | github: 77290 |
2021-05-21 07:22:17 | Lucas Cimon | set | messages: + msg394097 |
2021-05-21 07:02:44 | Lucas Cimon | set | nosy:
+ Lucas Cimon messages: + msg394096 |
2018-06-11 02:31:01 | ned.deily | set | messages: + msg319266 |
2018-06-11 02:19:12 | miss-islington | set | pull_requests: + pull_request7234 |
2018-06-11 02:17:59 | ned.deily | set | messages: + msg319264 |
2018-06-11 02:08:16 | ned.deily | set | pull_requests: + pull_request7232 |
2018-05-24 02:42:55 | ned.deily | set | status: open -> closed priority: release blocker -> messages: + msg317517 resolution: fixed stage: patch review -> resolved |
2018-05-24 02:22:48 | ned.deily | set | messages: + msg317511 |
2018-05-24 01:56:26 | miss-islington | set | stage: commit review -> patch review pull_requests: + pull_request6723 |
2018-05-24 01:55:18 | ned.deily | set | messages: + msg317499 |
2018-05-22 19:23:35 | serhiy.storchaka | set | messages: + msg317332 |
2018-05-22 19:21:53 | Anthony Sottile | set | messages: + msg317331 |
2018-05-22 19:16:37 | serhiy.storchaka | set | messages: + msg317329 |
2018-05-22 19:08:04 | Anthony Sottile | set | messages: + msg317326 |
2018-05-22 19:02:16 | serhiy.storchaka | set | nosy:
+ serhiy.storchaka messages: + msg317325 |
2018-05-22 18:45:50 | Anthony Sottile | set | messages: + msg317321 |
2018-05-22 18:29:39 | ned.deily | set | messages: + msg317319 |
2018-05-16 20:12:41 | Anthony Sottile | set | messages: + msg316860 |
2018-05-16 20:06:06 | ned.deily | set | messages:
+ msg316858 stage: patch review -> commit review |
2018-05-16 19:51:39 | ned.deily | set | keywords:
+ patch stage: patch review pull_requests: + pull_request6587 |
2018-05-16 14:03:31 | wolma | set | messages: + msg316807 |
2018-05-15 20:46:39 | Anthony Sottile | set | messages: + msg316721 |
2018-05-15 20:41:21 | gregory.p.smith | set | nosy:
+ gregory.p.smith messages: + msg316720 |
2018-05-15 19:19:35 | eric.araujo | set | messages: - msg316703 |
2018-05-15 19:19:33 | eric.araujo | set | messages: - msg316702 |
2018-05-15 19:19:14 | eric.araujo | set | messages: + msg316703 |
2018-05-15 19:18:56 | eric.araujo | set | messages: + msg316702 |
2018-05-15 19:18:44 | eric.araujo | set | priority: normal -> release blocker nosy: + ned.deily messages: + msg316701 |
2018-05-07 16:05:19 | bskinn | set | nosy:
+ bskinn messages: + msg316275 |
2018-03-20 16:54:18 | wolma | set | messages: + msg314156 |
2018-03-20 16:31:26 | Anthony Sottile | set | messages: + msg314153 |
2018-03-20 16:25:49 | wolma | set | messages: + msg314152 |
2018-03-20 15:43:10 | Anthony Sottile | set | messages: + msg314148 |
2018-03-20 15:38:02 | Anthony Sottile | set | messages: + msg314147 |
2018-03-20 13:41:59 | wolma | create |