This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: argparse: embedded groups may prevent options from being in help output
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Deprecate unsupported nesting of argparse groups
View: 22047
Assigned To: Nosy List: Laszlo.Attila.Toth, iritkatriel, paul.j3
Priority: normal Keywords: patch

Created on 2021-12-12 14:53 by Laszlo.Attila.Toth, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
argparse-add-arg_grp-deprecation.diff Laszlo.Attila.Toth, 2021-12-13 17:53 POC for add_argument_group
Messages (13)
msg408388 - (view) Author: László Attila Tóth (Laszlo.Attila.Toth) * Date: 2021-12-12 14:53
I tried to use the following code where the --db-password is not shown in the --help output (Originally I wanted to use mutually exclusive groups but that feature also works strangely, so I changed them to regular groups).

def register_db_args(parser: argparse.ArgumentParser):
    grp = parser.add_argument_group('Database settings')
    grp.add_argument('--db-config', dest='db_config_file',
                     help='Config file containg all details including password')

    grp.add_argument('--db-host')
    grp.add_argument('--db-port')
    grp.add_argument('--db-user')

    xgrp = grp.add_argument_group()
    xgrp.add_argument('--db-password')
    xgrp.add_argument('--db-password-env')
    xgrp.add_argument('--db-password-file')
msg408389 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-12 14:57
Please complete the bug report: How did you run this function, what output did you get and what output did you expect?
msg408394 - (view) Author: László Attila Tóth (Laszlo.Attila.Toth) * Date: 2021-12-12 16:25
Sorry, these are two bugs in fact. The current one, the help with minmal code:

import argparse

parser = argparse.ArgumentParser()
grp = parser.add_argument_group('Database settings')
grp.add_argument('--db-config')
xgrp = grp.add_argument_group()
xgrp.add_argument('--db-password')
parser.parse_args(['-h'])


The group's help output shows only --db-config option:

Database settings:
  --db-config DB_CONFIG

If I change the xgrp to be mutually exclusive group as:
xgrp = grp.add_mutually_exclusive_group()

then the output is the same as I expect for the previous code, too:

Database settings:
  --db-config DB_CONFIG
  --db-password DB_PASSWORD
msg408396 - (view) Author: László Attila Tóth (Laszlo.Attila.Toth) * Date: 2021-12-12 16:29
And the leading part is the same for both the mutually exclusive and the argument groups:

usage: test1.py [-h] [--db-config DB_CONFIG] [--db-password DB_PASSWORD]

optional arguments:
  -h, --help            show this help message and exit

Database settings:
....
msg408409 - (view) Author: László Attila Tóth (Laszlo.Attila.Toth) * Date: 2021-12-12 20:01
The fix is something like this for _ArgumentGroup, but as the container may not be an _ArgumentGroup, it breaks the tests.

--- Lib/argparse.py
+++ Lib/argparse.py
@@ -1635,9 +1640,13 @@ def __init__(self, container, title=None, description=None, **kwargs):
         self._has_negative_number_optionals = \
             container._has_negative_number_optionals
         self._mutually_exclusive_groups = container._mutually_exclusive_groups
+        self._container = container

     def _add_action(self, action):
-        action = super(_ArgumentGroup, self)._add_action(action)
+        if self.title:
+            action = super(_ArgumentGroup, self)._add_action(action)
+        else:
+            action = self._container._add_action(action)
         self._group_actions.append(action)
msg408411 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-12 23:07
According to the docs it should be

>>> xgrp = parser.add_argument_group()

rather than

>>> xgrp = grp.add_argument_group()


This seems to work:



>>> parser = argparse.ArgumentParser()
>>> grp = parser.add_argument_group('Database settings')
>>> grp.add_argument('--db-config')
_StoreAction(option_strings=['--db-config'], dest='db_config', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)
>>> xgrp = parser.add_argument_group()
>>> xgrp.add_argument('--db-password')
_StoreAction(option_strings=['--db-password'], dest='db_password', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)
>>> parser.parse_args(['-h'])
usage: [-h] [--db-config DB_CONFIG] [--db-password DB_PASSWORD]

options:
  -h, --help            show this help message and exit

Database settings:
  --db-config DB_CONFIG

  --db-password DB_PASSWORD
msg408427 - (view) Author: László Attila Tóth (Laszlo.Attila.Toth) * Date: 2021-12-13 06:48
According to the documentation only the ArgumentParser has add_argument_group option, which is not true, the code allows me to add a subgroup to any group. The complete example is in issue 4608: https://bugs.python.org/issue46058

If add_argument_group shouldn't be used for a regular argument group,
I suggest the following change: _ActionsContainer.add_argument_group should return self if title is not specified and raise error if add_argument_group("...") is called on a group with title.

This is close to the originally intended, documented behaviour.

I'd still allow groups without title in mutually exclusive groups and vice versa.
msg408473 - (view) Author: László Attila Tóth (Laszlo.Attila.Toth) * Date: 2021-12-13 17:53
my idea regarding _ArgumentGroup,add_argument_group is in the attached file. This doesn't solve the complete help output issue, but addresses the incorrectly called _ArgumentGroup.add_argument_group - a warn() call and return self.
As a result the help output for embedded groups (not mutually exclusive groups) work as expected. If the title parameter is specified, this
scenario is not checked, and so on.
msg408515 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-14 07:11
You’re right that the api should not be there. See issue22047.

I don’t think it should be patches to call super/return self. That would just be confusing.
msg408713 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-16 15:34
Nesting argument groups and mutually exclusive groups is now deprecated (see issue22047). Thank you for the bug report.
msg409486 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2022-01-02 02:12
Don't add an argument_group to another argument_group.  While input allows this nesting, the formatting is not designed to handle it.  Nor does the documentation illustrate such nesting.

Nesting a mutually_exclusive_group in an argument_group works because the exclusive_group is not used in formatting the help lines.  The two class of groups have very different purposes.

Note that the add_argument_group() method is defined for the parent _ActionsContainer class.  ArgumentParser inherits from this, as do both of the group classes.  While one could make a case for changing the group's inheritance of this method to Not-implemented, it's only a problem when users, like you, try to push for undocumented usage.

In general argparse has a clean and powerful class structure.  But it doesn't do a lot of checking and pruning, so there loose ends like this.  The Action class and its subclasses is similarly powerful, with enough loose ends to allow adventurous users to hang themselves :).
msg409492 - (view) Author: László Attila Tóth (Laszlo.Attila.Toth) * Date: 2022-01-02 10:57
I don't think I'm adventurous as I try to use its public API. If something is public in the pythonic terms (no underscore before it), but undocumented, that can obviously mean two things: either the documentation is outdated or the API published something it shouldn't.

As I wrote in issue 46058 - which is the same issue as this because of the implementation in argparse.py - I tried to create a group hierarchy in a way that I can pass the responsibility of argument validation to the argument parser. I don't think the good practice is to reimplement the same behavior again and again in dozens of Python programs if it is something that could be handled in an argument parser. My original intent was to provide patch or patches fixing the issue but with the deprecation in issue 22047 this became pointless.

There were a few other opened issues indicating I'm not alone with this expectation.
msg410248 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2022-01-10 19:30
"I tried to create a group hierarchy in a way that I can pass the responsibility of argument validation to the argument parser." 

I looked at your example in:

https://bugs.python.org/issue46058

The many levels of nesting groups was confusing, but now I realize that by nesting argument_groups in a mutually_exlusive_group you were try to implement some sort of any/all group within the xor group.  

This cannot be done within argparse.  Mutually_exclusive only implements a flat xor.  Argument_groups are used for help display, but play no role in parsing.

Some years ago I explored the use of nest parsing groups:

https://bugs.python.org/issue11588
Add "necessarily inclusive" groups to argparse

I was trying to allow for nested groups that implemented all logical options - xor, or, and, not.  Defining such groups wasn't hard.  And I think I got the testing logic working right.  But usage display required too big of a change to the formatter, and left too many loose ends.  So I have let that languish.

Now I recommend focusing on doing the testing after parsing.  Use meaningful defaults where possible, and use `is None` to test whether a users has provided a value or not.
History
Date User Action Args
2022-04-11 14:59:53adminsetgithub: 90215
2022-01-10 19:30:34paul.j3setmessages: + msg410248
2022-01-02 10:57:19Laszlo.Attila.Tothsetmessages: + msg409492
2022-01-02 02:12:21paul.j3setnosy: + paul.j3
messages: + msg409486
2021-12-16 15:34:07iritkatrielsetstatus: open -> closed

messages: + msg408713
stage: resolved
2021-12-14 07:11:28iritkatrielsetsuperseder: Deprecate unsupported nesting of argparse groups
resolution: duplicate
messages: + msg408515
2021-12-13 17:53:17Laszlo.Attila.Tothsetfiles: + argparse-add-arg_grp-deprecation.diff
keywords: + patch
messages: + msg408473
2021-12-13 06:48:31Laszlo.Attila.Tothsetmessages: + msg408427
2021-12-12 23:07:35iritkatrielsetmessages: + msg408411
2021-12-12 20:01:54Laszlo.Attila.Tothsetmessages: + msg408409
2021-12-12 16:29:30Laszlo.Attila.Tothsetmessages: + msg408396
2021-12-12 16:25:28Laszlo.Attila.Tothsetmessages: + msg408394
2021-12-12 14:57:46iritkatrielsetnosy: + iritkatriel
messages: + msg408389
2021-12-12 14:53:18Laszlo.Attila.Tothcreate