classification
Title: argparse fails with required subparsers, un-named dest, and empty argv
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: Lucas Cimon, Mathias Ettinger, Minshall, eric.araujo, hroncok, iritkatriel, miss-islington, paul.j3, paulie4, petr.viktorin, zachrahan
Priority: normal Keywords: patch

Created on 2017-01-17 14:52 by zachrahan, last changed 2021-07-27 15:42 by petr.viktorin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3680 merged Anthony Sottile, 2017-09-20 21:48
PR 18564 closed python-dev, 2020-02-20 01:39
PR 26278 closed Lucas Cimon, 2021-05-21 07:22
PR 27303 merged miss-islington, 2021-07-23 12:49
PR 27304 merged miss-islington, 2021-07-23 12:49
Messages (9)
msg285646 - (view) Author: (zachrahan) Date: 2017-01-17 14:52
In python 3.6 (and several versions back), using argparse with required subparsers will cause an unhelpful TypeError if the 'dest' parameter is not explicitly specified, and no arguments are provided.

Test case:
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
subparsers.required = True
args = parser.parse_args([])

Observed result:
TypeError: sequence item 0: expected str instance, NoneType found

If the line above is changed to:
subparsers = parser.add_subparsers(dest='function')

Then the following is printed to stderr:
usage: python [-h] {} ...
python: error: the following arguments are required: function

This issue goes back at least several years:
http://stackoverflow.com/questions/23349349/argparse-with-required-subparser/23354355

Though it seems odd to not specify a dest in the add_subparsers line, the pattern is not completely useless. The below works fine without setting a 'dest' in add_subparsers, except when argv is empty:
sub1 = subparsers.add_parser('print')
sub1.set_defaults(function=print)

However, an empty argv produces the unexpected TypeError above. I'm not sure if argparse should provide a more useful exception in this case, or if there is a clean way to do the right thing without a dest specified.
msg285877 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2017-01-20 05:38
http://bugs.python.org/issue9253  argparse: optional subparsers

Initially this bug/issue was a request to allow subparsers to be optional.  But with the change in how required actions are handled, subparsers are now optional by default.

As you learned from the SO question you now have to specify

subparsers.required = True

This is also discussed in my post (and following ones)

http://bugs.python.org/issue9253#msg186387

The default 'dest' is SUPPRESS.  The error you report occurs because the 'required' error mechanism cannot handle that value. The suggest fix is to assign `dest`, even if it is not needed in the Namespace.  For now it is needed for error reporting.

Reviewing my suggested patches, it looks like I generate a 'dest' substitute from the subparser names.  So the 'required' error would look like

python: error: the following arguments are required: {cmd1, cmd2}

I think this issue can be closed with a reference to 9253. Or maybe that issue is too old, long and confusing, and we need a new bug/issue.
msg330133 - (view) Author: Mathias Ettinger (Mathias Ettinger) Date: 2018-11-20 15:37
I was just hit by the very same issue and added the following test into `_get_action_name` to work around it:

    elif isinstance(argument, _SubParsersAction):
        return '{%s}' % ','.join(map(str, argument.choices))

I checked #9253 as referenced by paul j3 and like the `argument.name()` approach as well as it's less specific.

Any chance this can be addressed one way or another?
msg357839 - (view) Author: Greg (Minshall) Date: 2019-12-05 06:00
while waiting for a fix, would it be possible to document in the argparse documentation that the 'dest' parameter is required (at least temporarily) for add_subparsers()?  (somewhere near file:///usr/share/doc/python/html/library/argparse.html#sub-commands)

gratuitous diff:  the pull request from 2017 would probably fix it.  my diffs are here (from: Python 3.8.0 (default, Oct 23 2019, 18:51:26).  (the pull request changes the utility '_get_action_name'; i wasn't sure of side-effects with other callers, so changed nearer the failure location.)
----
*** new/argparse.py     2019-12-05 11:16:37.618985247 +0530
--- old/argparse.py     2019-10-24 00:21:26.000000000 +0530
***************
*** 2017,2030 ****
          for action in self._actions:
              if action not in seen_actions:
                  if action.required:
!                     ra = _get_action_name(action)
!                     if ra is None:
!                         if not action.choices == {}:
!                             choice_strs = [str(choice) for choice in action.choices]
!                             ra = '{%s}' % ','.join(choice_strs)
!                         else:
!                             ra = '<unknown>'
!                     required_actions.append(ra)
                  else:
                      # Convert action default now instead of doing it before
                      # parsing arguments to avoid calling convert functions
--- 2017,2023 ----
          for action in self._actions:
              if action not in seen_actions:
                  if action.required:
!                     required_actions.append(_get_action_name(action))
                  else:
                      # Convert action default now instead of doing it before
                      # parsing arguments to avoid calling convert functions
msg393958 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-05-19 16:44
crash means segfault, not unhandled exception.
msg396654 - (view) Author: Paulie Pena (paulie4) Date: 2021-06-28 17:30
I'd like to second this idea, since it's very confusing without it:
> while waiting for a fix, would it be possible to document in the argparse documentation that the 'dest' parameter is required (at least temporarily) for add_subparsers()?  (somewhere near file:///usr/share/doc/python/html/library/argparse.html#sub-commands)
msg398049 - (view) Author: miss-islington (miss-islington) Date: 2021-07-23 12:49
New changeset 17575f73ce2cb9f3a4eb4cc416c690f9a4e7205c by Anthony Sottile in branch 'main':
bpo-29298: Fix crash with required subparsers without dest (GH-3680)
https://github.com/python/cpython/commit/17575f73ce2cb9f3a4eb4cc416c690f9a4e7205c
msg398051 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-07-23 13:27
New changeset c589992e09d0db7cb47d21d5948929e599fdbb94 by Miss Islington (bot) in branch '3.10':
bpo-29298: Fix crash with required subparsers without dest (GH-3680) (GH-27303)
https://github.com/python/cpython/commit/c589992e09d0db7cb47d21d5948929e599fdbb94
msg398052 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-07-23 13:27
New changeset 097801844c99ea3916bebe1cc761257ea7083d34 by Miss Islington (bot) in branch '3.9':
bpo-29298: Fix crash with required subparsers without dest (GH-3680) (GH-27304)
https://github.com/python/cpython/commit/097801844c99ea3916bebe1cc761257ea7083d34
History
Date User Action Args
2021-07-27 15:42:54petr.viktorinsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-07-23 13:27:25petr.viktorinsetmessages: + msg398052
2021-07-23 13:27:23petr.viktorinsetnosy: + petr.viktorin
messages: + msg398051
2021-07-23 12:49:24miss-islingtonsetmessages: + msg398049
2021-07-23 12:49:21miss-islingtonsetpull_requests: + pull_request25848
2021-07-23 12:49:15miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request25847
2021-06-28 17:30:37paulie4setnosy: + paulie4
messages: + msg396654
2021-05-21 07:22:09Lucas Cimonsetnosy: + Lucas Cimon
pull_requests: + pull_request24884
2021-05-19 16:44:40iritkatrielsetversions: + Python 3.10, Python 3.11, - Python 2.7, Python 3.6, Python 3.7, Python 3.8
nosy: + iritkatriel

messages: + msg393958

type: crash -> behavior
2020-02-20 02:01:47bensokolsettype: behavior -> crash
versions: + Python 3.8, Python 3.9
2020-02-20 01:39:33python-devsetpull_requests: + pull_request17945
2019-12-05 06:00:18Minshallsetnosy: + Minshall
messages: + msg357839
2018-11-20 15:37:01Mathias Ettingersetnosy: + Mathias Ettinger
messages: + msg330133
2018-06-05 13:13:04hroncoksetnosy: + hroncok
2017-09-20 22:04:04eric.araujosetassignee: eric.araujo

nosy: + eric.araujo
versions: + Python 2.7, Python 3.7
2017-09-20 21:48:48Anthony Sottilesetkeywords: + patch
stage: patch review
pull_requests: + pull_request3669
2017-01-20 05:38:54paul.j3setnosy: + paul.j3
messages: + msg285877
2017-01-17 14:52:49zachrahancreate