classification
Title: argparse fails with required subparsers, un-named dest, and empty argv
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: Mathias Ettinger, Minshall, eric.araujo, hroncok, paul.j3, zachrahan
Priority: normal Keywords: patch

Created on 2017-01-17 14:52 by zachrahan, last changed 2019-12-05 06:00 by Minshall.

Pull Requests
URL Status Linked Edit
PR 3680 open Anthony Sottile, 2017-09-20 21:48
Messages (4)
msg285646 - (view) Author: (zachrahan) Date: 2017-01-17 14:52
In python 3.6 (and several versions back), using argparse with required subparsers will cause an unhelpful TypeError if the 'dest' parameter is not explicitly specified, and no arguments are provided.

Test case:
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
subparsers.required = True
args = parser.parse_args([])

Observed result:
TypeError: sequence item 0: expected str instance, NoneType found

If the line above is changed to:
subparsers = parser.add_subparsers(dest='function')

Then the following is printed to stderr:
usage: python [-h] {} ...
python: error: the following arguments are required: function

This issue goes back at least several years:
http://stackoverflow.com/questions/23349349/argparse-with-required-subparser/23354355

Though it seems odd to not specify a dest in the add_subparsers line, the pattern is not completely useless. The below works fine without setting a 'dest' in add_subparsers, except when argv is empty:
sub1 = subparsers.add_parser('print')
sub1.set_defaults(function=print)

However, an empty argv produces the unexpected TypeError above. I'm not sure if argparse should provide a more useful exception in this case, or if there is a clean way to do the right thing without a dest specified.
msg285877 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2017-01-20 05:38
http://bugs.python.org/issue9253  argparse: optional subparsers

Initially this bug/issue was a request to allow subparsers to be optional.  But with the change in how required actions are handled, subparsers are now optional by default.

As you learned from the SO question you now have to specify

subparsers.required = True

This is also discussed in my post (and following ones)

http://bugs.python.org/issue9253#msg186387

The default 'dest' is SUPPRESS.  The error you report occurs because the 'required' error mechanism cannot handle that value. The suggest fix is to assign `dest`, even if it is not needed in the Namespace.  For now it is needed for error reporting.

Reviewing my suggested patches, it looks like I generate a 'dest' substitute from the subparser names.  So the 'required' error would look like

python: error: the following arguments are required: {cmd1, cmd2}

I think this issue can be closed with a reference to 9253. Or maybe that issue is too old, long and confusing, and we need a new bug/issue.
msg330133 - (view) Author: Mathias Ettinger (Mathias Ettinger) Date: 2018-11-20 15:37
I was just hit by the very same issue and added the following test into `_get_action_name` to work around it:

    elif isinstance(argument, _SubParsersAction):
        return '{%s}' % ','.join(map(str, argument.choices))

I checked #9253 as referenced by paul j3 and like the `argument.name()` approach as well as it's less specific.

Any chance this can be addressed one way or another?
msg357839 - (view) Author: Greg (Minshall) Date: 2019-12-05 06:00
while waiting for a fix, would it be possible to document in the argparse documentation that the 'dest' parameter is required (at least temporarily) for add_subparsers()?  (somewhere near file:///usr/share/doc/python/html/library/argparse.html#sub-commands)

gratuitous diff:  the pull request from 2017 would probably fix it.  my diffs are here (from: Python 3.8.0 (default, Oct 23 2019, 18:51:26).  (the pull request changes the utility '_get_action_name'; i wasn't sure of side-effects with other callers, so changed nearer the failure location.)
----
*** new/argparse.py     2019-12-05 11:16:37.618985247 +0530
--- old/argparse.py     2019-10-24 00:21:26.000000000 +0530
***************
*** 2017,2030 ****
          for action in self._actions:
              if action not in seen_actions:
                  if action.required:
!                     ra = _get_action_name(action)
!                     if ra is None:
!                         if not action.choices == {}:
!                             choice_strs = [str(choice) for choice in action.choices]
!                             ra = '{%s}' % ','.join(choice_strs)
!                         else:
!                             ra = '<unknown>'
!                     required_actions.append(ra)
                  else:
                      # Convert action default now instead of doing it before
                      # parsing arguments to avoid calling convert functions
--- 2017,2023 ----
          for action in self._actions:
              if action not in seen_actions:
                  if action.required:
!                     required_actions.append(_get_action_name(action))
                  else:
                      # Convert action default now instead of doing it before
                      # parsing arguments to avoid calling convert functions
History
Date User Action Args
2019-12-05 06:00:18Minshallsetnosy: + Minshall
messages: + msg357839
2018-11-20 15:37:01Mathias Ettingersetnosy: + Mathias Ettinger
messages: + msg330133
2018-06-05 13:13:04hroncoksetnosy: + hroncok
2017-09-20 22:04:04eric.araujosetassignee: eric.araujo

nosy: + eric.araujo
versions: + Python 2.7, Python 3.7
2017-09-20 21:48:48Anthony Sottilesetkeywords: + patch
stage: patch review
pull_requests: + pull_request3669
2017-01-20 05:38:54paul.j3setnosy: + paul.j3
messages: + msg285877
2017-01-17 14:52:49zachrahancreate