Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

argparse: embedded groups may prevent options from being in help output #90215

Closed
LA-Toth mannequin opened this issue Dec 12, 2021 · 13 comments
Closed

argparse: embedded groups may prevent options from being in help output #90215

LA-Toth mannequin opened this issue Dec 12, 2021 · 13 comments
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@LA-Toth
Copy link
Mannequin

LA-Toth mannequin commented Dec 12, 2021

BPO 46057
Nosy @LA-Toth, @iritkatriel
Superseder
  • bpo-22047: Deprecate unsupported nesting of argparse groups
  • Files
  • argparse-add-arg_grp-deprecation.diff: POC for add_argument_group
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-12-16.15:34:07.520>
    created_at = <Date 2021-12-12.14:53:18.620>
    labels = ['type-bug', '3.8', '3.9', '3.10', '3.11', 'library']
    title = 'argparse: embedded groups may prevent options from being in help output'
    updated_at = <Date 2022-01-10.19:30:34.674>
    user = 'https://github.com/LA-Toth'

    bugs.python.org fields:

    activity = <Date 2022-01-10.19:30:34.674>
    actor = 'paul.j3'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-12-16.15:34:07.520>
    closer = 'iritkatriel'
    components = ['Library (Lib)']
    creation = <Date 2021-12-12.14:53:18.620>
    creator = 'Laszlo.Attila.Toth'
    dependencies = []
    files = ['50489']
    hgrepos = []
    issue_num = 46057
    keywords = ['patch']
    message_count = 13.0
    messages = ['408388', '408389', '408394', '408396', '408409', '408411', '408427', '408473', '408515', '408713', '409486', '409492', '410248']
    nosy_count = 3.0
    nosy_names = ['paul.j3', 'Laszlo.Attila.Toth', 'iritkatriel']
    pr_nums = []
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '22047'
    type = 'behavior'
    url = 'https://bugs.python.org/issue46057'
    versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

    @LA-Toth
    Copy link
    Mannequin Author

    LA-Toth mannequin commented Dec 12, 2021

    I tried to use the following code where the --db-password is not shown in the --help output (Originally I wanted to use mutually exclusive groups but that feature also works strangely, so I changed them to regular groups).

    def register_db_args(parser: argparse.ArgumentParser):
        grp = parser.add_argument_group('Database settings')
        grp.add_argument('--db-config', dest='db_config_file',
                         help='Config file containg all details including password')
    
        grp.add_argument('--db-host')
        grp.add_argument('--db-port')
        grp.add_argument('--db-user')
    
        xgrp = grp.add_argument_group()
        xgrp.add_argument('--db-password')
        xgrp.add_argument('--db-password-env')
        xgrp.add_argument('--db-password-file')

    @LA-Toth LA-Toth mannequin added 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Dec 12, 2021
    @iritkatriel
    Copy link
    Member

    Please complete the bug report: How did you run this function, what output did you get and what output did you expect?

    @LA-Toth
    Copy link
    Mannequin Author

    LA-Toth mannequin commented Dec 12, 2021

    Sorry, these are two bugs in fact. The current one, the help with minmal code:

    import argparse
    
    parser = argparse.ArgumentParser()
    grp = parser.add_argument_group('Database settings')
    grp.add_argument('--db-config')
    xgrp = grp.add_argument_group()
    xgrp.add_argument('--db-password')
    parser.parse_args(['-h'])

    The group's help output shows only --db-config option:

    Database settings:
    --db-config DB_CONFIG

    If I change the xgrp to be mutually exclusive group as:
    xgrp = grp.add_mutually_exclusive_group()

    then the output is the same as I expect for the previous code, too:

    Database settings:
    --db-config DB_CONFIG
    --db-password DB_PASSWORD

    @LA-Toth
    Copy link
    Mannequin Author

    LA-Toth mannequin commented Dec 12, 2021

    And the leading part is the same for both the mutually exclusive and the argument groups:

    usage: test1.py [-h] [--db-config DB_CONFIG] [--db-password DB_PASSWORD]

    optional arguments:
    -h, --help show this help message and exit

    Database settings:
    ....

    @LA-Toth
    Copy link
    Mannequin Author

    LA-Toth mannequin commented Dec 12, 2021

    The fix is something like this for _ArgumentGroup, but as the container may not be an _ArgumentGroup, it breaks the tests.

    --- Lib/argparse.py
    +++ Lib/argparse.py
    @@ -1635,9 +1640,13 @@ def __init__(self, container, title=None, description=None, **kwargs):
             self._has_negative_number_optionals = \
                 container._has_negative_number_optionals
             self._mutually_exclusive_groups = container._mutually_exclusive_groups
    +        self._container = container
    
         def _add_action(self, action):
    -        action = super(_ArgumentGroup, self)._add_action(action)
    +        if self.title:
    +            action = super(_ArgumentGroup, self)._add_action(action)
    +        else:
    +            action = self._container._add_action(action)
             self._group_actions.append(action)

    @iritkatriel
    Copy link
    Member

    According to the docs it should be

    >> xgrp = parser.add_argument_group()

    rather than

    >> xgrp = grp.add_argument_group()

    This seems to work:

    >>> parser = argparse.ArgumentParser()
    >>> grp = parser.add_argument_group('Database settings')
    >>> grp.add_argument('--db-config')
    _StoreAction(option_strings=['--db-config'], dest='db_config', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)
    >>> xgrp = parser.add_argument_group()
    >>> xgrp.add_argument('--db-password')
    _StoreAction(option_strings=['--db-password'], dest='db_password', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)
    >>> parser.parse_args(['-h'])
    usage: [-h] [--db-config DB_CONFIG] [--db-password DB_PASSWORD]

    options:
    -h, --help show this help message and exit

    Database settings:
    --db-config DB_CONFIG

    --db-password DB_PASSWORD

    @LA-Toth
    Copy link
    Mannequin Author

    LA-Toth mannequin commented Dec 13, 2021

    According to the documentation only the ArgumentParser has add_argument_group option, which is not true, the code allows me to add a subgroup to any group. The complete example is in bpo-4608: https://bugs.python.org/issue46058

    If add_argument_group shouldn't be used for a regular argument group,
    I suggest the following change: _ActionsContainer.add_argument_group should return self if title is not specified and raise error if add_argument_group("...") is called on a group with title.

    This is close to the originally intended, documented behaviour.

    I'd still allow groups without title in mutually exclusive groups and vice versa.

    @LA-Toth
    Copy link
    Mannequin Author

    LA-Toth mannequin commented Dec 13, 2021

    my idea regarding _ArgumentGroup,add_argument_group is in the attached file. This doesn't solve the complete help output issue, but addresses the incorrectly called _ArgumentGroup.add_argument_group - a warn() call and return self.
    As a result the help output for embedded groups (not mutually exclusive groups) work as expected. If the title parameter is specified, this
    scenario is not checked, and so on.

    @iritkatriel
    Copy link
    Member

    You’re right that the api should not be there. See bpo-22047.

    I don’t think it should be patches to call super/return self. That would just be confusing.

    @iritkatriel
    Copy link
    Member

    Nesting argument groups and mutually exclusive groups is now deprecated (see bpo-22047). Thank you for the bug report.

    @paulj3
    Copy link
    Mannequin

    paulj3 mannequin commented Jan 2, 2022

    Don't add an argument_group to another argument_group. While input allows this nesting, the formatting is not designed to handle it. Nor does the documentation illustrate such nesting.

    Nesting a mutually_exclusive_group in an argument_group works because the exclusive_group is not used in formatting the help lines. The two class of groups have very different purposes.

    Note that the add_argument_group() method is defined for the parent _ActionsContainer class. ArgumentParser inherits from this, as do both of the group classes. While one could make a case for changing the group's inheritance of this method to Not-implemented, it's only a problem when users, like you, try to push for undocumented usage.

    In general argparse has a clean and powerful class structure. But it doesn't do a lot of checking and pruning, so there loose ends like this. The Action class and its subclasses is similarly powerful, with enough loose ends to allow adventurous users to hang themselves :).

    @LA-Toth
    Copy link
    Mannequin Author

    LA-Toth mannequin commented Jan 2, 2022

    I don't think I'm adventurous as I try to use its public API. If something is public in the pythonic terms (no underscore before it), but undocumented, that can obviously mean two things: either the documentation is outdated or the API published something it shouldn't.

    As I wrote in bpo-46058 - which is the same issue as this because of the implementation in argparse.py - I tried to create a group hierarchy in a way that I can pass the responsibility of argument validation to the argument parser. I don't think the good practice is to reimplement the same behavior again and again in dozens of Python programs if it is something that could be handled in an argument parser. My original intent was to provide patch or patches fixing the issue but with the deprecation in bpo-22047 this became pointless.

    There were a few other opened issues indicating I'm not alone with this expectation.

    @paulj3
    Copy link
    Mannequin

    paulj3 mannequin commented Jan 10, 2022

    "I tried to create a group hierarchy in a way that I can pass the responsibility of argument validation to the argument parser."

    I looked at your example in:

    https://bugs.python.org/issue46058

    The many levels of nesting groups was confusing, but now I realize that by nesting argument_groups in a mutually_exlusive_group you were try to implement some sort of any/all group within the xor group.

    This cannot be done within argparse. Mutually_exclusive only implements a flat xor. Argument_groups are used for help display, but play no role in parsing.

    Some years ago I explored the use of nest parsing groups:

    https://bugs.python.org/issue11588
    Add "necessarily inclusive" groups to argparse

    I was trying to allow for nested groups that implemented all logical options - xor, or, and, not. Defining such groups wasn't hard. And I think I got the testing logic working right. But usage display required too big of a change to the formatter, and left too many loose ends. So I have let that languish.

    Now I recommend focusing on doing the testing after parsing. Use meaningful defaults where possible, and use is None to test whether a users has provided a value or not.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant