The argparse module lists required args as optional in the default help message.

If you run the following program (also attached) you'll get the output listed below.

#!/usr/bin/env python

import argparse

parser = argparse.ArgumentParser(
    description = 'Do something'
)
parser.add_argument('--reqarg', '-r', help = 'This is required', required = True)
parser.add_argument('--optarg','-o', help = "This is optional", required = False)
args = parser.parse_args()

$ python argparse-help-says-required-args-are-optional.py -h
usage: argparse-help-says-required-args-are-optional.py [-h] --reqarg REQARG
                                                        [--optarg OPTARG]

Do something

optional arguments:
  -h, --help            show this help message and exit
  --reqarg REQARG, -r REQARG
                        This is required
  --optarg OPTARG, -o OPTARG
                        This is optional
$

It looks to me like reqarg is marked as required, since it's not in brackets. Or am I missing something?

Yeah, the fact that it is listed under the heading "optional arguments:" :)  Guess we need a new section?

Duh. Sorry about that.

Also applies to 3.2.

Yeah, I guess the optional vs. positional isn't the best terminology now that you can have required flag-based arguments. Did you have a word other than "optional" that you'd prefer?

Perhaps you could just label them 'options:'?  After all, even if you have several options you may be required to pick at least one :)

Or "parameters:"?

FWIW, I like the idea of just using the label "options".

If you add a positional parameter by adding:
parser.add_argument('foo')
then the output becomes:

$ python argparse-help-says-required-args-are-optional.py -h
usage: issue9649.py [-h] --reqarg REQARG [--optarg OPTARG] foo

Do something

positional arguments:
  foo

optional arguments:
  -h, --help            show this help message and exit
  --reqarg REQARG, -r REQARG
                        This is required
  --optarg OPTARG, -o OPTARG
                        This is optional

$

So whatever replaces "optional arguments:" needs to read well with "positional arguments:". Maybe just plain "options:" is good enough, but I think a word to replace "optional" (leaving "arguments:") would be better. I just don't have any useful suggestion :)

I guess one possibility might be "flag arguments". It's not great, but I guess it's more accurate.

And I guess the bigger issue to think about is how to add this in a backwards compatible way. I guess we could just add methods like "set_positionals_group_name(name)" and then fiddle with "self._positionals.title" in there. Not sure that's a great solution though - it seems like adding one method to change just this single attribute is overkill and not very general.

In the meantime, here's a workaround:

>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo', required=True)
>>> parser._optionals.title = "flag arguments"
>>> parser.print_help()
usage: PROG [-h] --foo FOO

flag arguments:
  -h, --help  show this help message and exit
  --foo FOO

I can't promise this will continue to work, since it uses the undocumented _optionals attribute, but at least it's a way of getting something like what you want now.

Well, there's also issue 9652, which speaks to having a more general facility, I suppose.  Maybe an exposed dictionary attribute containing the constant strings?

Is this really a behavior bug or doc bug?
Or a feature request for better message customization?

*This* bug is a behavior bug (required flags are mis-labelled as being optional in the help text).  The referenced bug is a feature request, but it may make sense to consider it while fixing this one.

I think this is still really a feature request. We can't just change the text from "optional" - that would silently change a large number of help messages without any warning. So to fix this "bug", we're going to have to add an API to explicitly set the group names - which can only be done as a new feature. People using 2.7 will have to use the workaround using parser._optionals I posted here.

So it strikes me that there already exists an officially supported way to rename your option groups. Just only create your own option groups (never use the default ones) and only put arguments there, e.g.:

------------------------- temp.py --------------------------
parser = argparse.ArgumentParser(description = 'Do something', add_help=False)
flags = parser.add_argument_group('flag arguments')
flags.add_argument('-h', '--help', action='help')
flags.add_argument('--reqarg', '-r', help='This is required', required=True)
flags.add_argument('--optarg','-o', help="This is optional", required=False)
args = parser.parse_args()
------------------------------------------------------------
$ python temp.py --help
usage: temp.py [-h] --reqarg REQARG [--optarg OPTARG]

Do something

flag arguments:
  -h, --help
  --reqarg REQARG, -r REQARG
                        This is required
  --optarg OPTARG, -o OPTARG
                        This is optional
------------------------------------------------------------

The documentation for action='help' needs to be added, as pointed out in Issue# 10772.

So basically, the API for customizing group names is already there. So I'm changing this to a documentation request - there should be an example in the docs showing how to change the default group names as above.

I'm changing the title because I keep seeing duplicates.

Documentation patches still welcome!

How about calling required arguments "required arguments"?

required arguments:
  --reqarg REQARG, -r REQARG
                        This is required
optional arguments:
  -h, --help            show this help message and exit
  --optarg OPTARG, -o OPTARG
                        This is optional

Clear and unambiguous. With this approach the user does not have to bloat the help to state "This is required".

We're having the same discussion over at github regarding argparse4j:
https://github.com/tatsuhiro-t/argparse4j/issues/26#issuecomment-32894297

A new “required arguments” section seems too arbitrary to me. It would clash with the “positional arguments” heading, since those are also required by default.

I would go with the heading “options”, as a noun. That term seems to be well used, at least on Linux and Wikipedia (see https://en.wikipedia.org/wiki/Command-line_option). Other terms are “flag” and “switch”. In this thread I see two arguments against this:

1. Eric Smith prefers to retain the noun “arguments”. How about something like “non-positional arguments” then?

2. Steven Bethard is worried about backwards compatibility. I thought the Python people were happy to make these sort of changes each minor release (e.g. 3.4 to 3.5).

The module’s source code uses the term “optionals” a lot more than this one heading. It would be clearer if this term were dropped, or only used for things that are truly optional. So even if you can’t fix the help output until Python 4, please fix the documentation and the rest of the source code :)

> How about calling required arguments "required arguments"?
> ...
> Clear and unambiguous. With this approach the user does 
> not have to bloat the help to state "This is required".

+1 This is straight-forward, logical, and easy-to-read.

As Steven pointed out, the existing `add_argument_group` mechanism can be used to group required arguments.  For example

    ------------------------- temp.py --------------------------
    parser = argparse.ArgumentParser(description = 'Do something')
    group1 = parser.add_argument_group('required arguments')
    group1.add_argument('--reqarg', '-r', required=True)
    parser.add_argument('--optarg','-o')
    parser.add_argument('foo')
    parser.print_help()
    ------------------------------------------------------------
    usage: ipython [-h] --reqarg REQARG [--optarg OPTARG] foo
    Do something
    positional arguments:
      foo
    optional arguments:
      -h, --help            show this help message and exit
      --optarg OPTARG, -o OPTARG
    required arguments:
      --reqarg REQARG, -r REQARG

Positional 'foo' can also be put in the 'required' group:
    
    group1.add_argument('foo')

    required arguments:
      --reqarg REQARG, -r REQARG
      foo

The distinction between 'positionals' and 'optionals' (or flagged) is essential to the parsing, but it is not necessary for Help Formatting.

I can imagine grouping arguments by 'required/not-required' properties.  It might be worth constructing an alternative HelpFormatter class that regroups the arguments in this way.  Subclassing the HelpFormatter is the established way of adding features to the help display.

The existing HelpFormatter flags 'required' arguments in the usage line with '[]'.  There it is has the added task of flagging Mutually Exclusive Groups in the same way.

It's worth keeping in mind that whether an argument is 'required' or not is determined in 2 different ways.  There is an optional 'required' flag (default False).  But this flag is not allowed for 'positionals'.  Instead with those 'argparse' looks at 'nargs' ('?*' are not required).

The 'required' attribute of an argument (Action) is ignored during 'parse_args' until the end.  At that time it makes an inventory of 'required' arguments that have not been seen, and potentially raises an error.  That testing was changed in a relatively recent patch, and produced an unintended change in whether 'subparsers' were required or not. (I could look up those issues in needed).

I'll think about creating the alternative HelpFormatter.

The attached file shows how the default argument groups could be redefined, using 'required' as the criteria.

I've implemented it as a method that is added to a subclass of ArgumentParser.  This method is invoked after arguments are defined, prior to generating the help.

The help looks something like this:

    usage: alt_grouping.py [-h] [-f FOO] -g GOO pos [baz]

    required:
      pos                required positional
      -g GOO, --goo GOO  required optional

    optional:
      -h, --help         show this help message and exit
      -f FOO, --foo FOO  optional
      baz                optional positional

I was thinking of implementing this as a formatter subclass, but given the way the help is assembled, invoking this method from the parser is simpler.

Here's another possible solution: add a `help_groups` parameter to ArgumentParser.  It is a list of base argument group names.  `parser.add_argument(...)' places the action in one of those groups.

This is a generalization of the current code which creates two groups titled 'positional arguments' and 'optional arguments', and assigns actions based on 'optional strings' (e.g. '-f','--foo').

'help_groups' could have 1, 2, or 3 items.  

1 - just one argument group

2 - the current postional/optional split, but with user chosen names

3 - a 'positional', 'required', and 'optional' split.  

A 4 way split that distinguishes splits positionals between those that allow 0 values and 1 or more, is possible, but probably not that useful.

The changes are in the ArgumentParser.__init__ and _add_action methods.

'subparsers' do not inherit this parameter.  I have not explored how it plays out with 'parents'. 'test_argparse.py' runs fine.

The term "optional arguments" is a poorly formed adjectival suffix of "option". What it's trying to say is "these are kwargs" rather than "these arguments are not compulsory".

I ran into this issue with 3.4 and was looking to file a simple change request:

In early DOS/Linux days we called these "switches". I suggest we simply change the default wording from "optional arguments" to "switches".

To me, the mistake is contrasting 'positional' versus 'optional'.  The proper contrasts are 'positional' versus 'named' or 'keyword'  -- I believe these are mutually exclusive for command lines -- and 'required' versus 'optional.  The two axes (contrasts) are orthogonal. Where are optional positional parameters listed?  If, as I presume, they are listed as 'positional' and given that all keyword arguments are already listed in the so-called 'optional' section, I think we should regard 'optional' as a misspelling of 'keyword'.  That is a word already familiar to python programmers. The change should only be made in default for the same reason we do not correct minor errors in exception messages in bugfix releases.

In unix parlance, they are arguments and options (or, sometimes, flags).  And then required or not required.  So, argparse follows unix precedent here, except that it calls them "optional arguments", because everything is added via add_argument.  Which is why I suggested changing the label to just 'options'.  But I could see using 'switches' instead, that's less ambiguous, and is the term used on Windows (albeit with a different standard syntax).  However, every unix man page uses the term 'options'.

I definitely think 'keywords' is not a good idea.  That's crossing the streams (python parlance versus shell parlance).  argparse is building a bridge to the shell world, and should use its terminology for the bits of shell stuff it is implementing...most especially in the default help display.

Note that, reading the issue history, the argparse maintainer is urging a doc change only (how to fix the help if you run into this issue), not a behavior change.

Personally, 'options' is the first word that comes to mind other than 'arguments' (which is confusing for obvious reasons). But I think we can all agree that any of these candidates are less ambiguous than 'optional arguments'! It's been 4 years, let's put this one to bed!

Here is much larger patch in the spirit of Ryan’s, that fixes the documentation, adjusts the tests, and some of the internal comments and variable names in the source code as well. However if some changes are too controversial, I am happy to simplify it to (say) Ryan’s patch plus the minimum test fixes.

Updated my patch with a “version changed” notice

Is there any interest in my or Ryan’s patches, which change the default heading away from “optional arguments”? Changing the default is my preferred fix, but if others don’t like it (e.g. compatibility concerns), I am happy to work on a documentation patch according to <https://bugs.python.org/issue9694#msg132327>.

I don’t see how adding an extra help groups API is very helpful though. The problem here is that the default help group headings are wrong (or at least misleading to many people).

This is not just a section issue. Please note that using the ArgumentDefaultHelpFormatter:

  argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultHelpFormatter)

The help will also show (default=None) for these required arguments.

This ArgumentDefaultHelpFormatter issue should probably be raised in its own issue.  It applies to 'required' optionals, but any patch would be independent of the issues discussed here.

This class defines a method that adds the '%(default)' string to the help in certain situations.  It already skips required positionals.  So adding a test for 'action.required' should be easy.

    def _get_help_string(self, action):
        help = action.help
        if '%(default)' not in action.help:
            if action.default is not SUPPRESS:
                defaulting_nargs = [OPTIONAL, ZERO_OR_MORE]
                if action.option_strings or action.nargs in defaulting_nargs:
                    help += ' (default: %(default)s)'
        return help

There are 2 easy user fixes.

- a custom HelpFormatter class that implements this fix.  

- 'default=argparse.SUPPRESS' for arguments where you do not want to see the default.  This SUPPRESS is checked else where in the code, but for a required argument I don't think that matters (but it needs testing).

Still applicable to Python 3 AFAIK

I can see changing the group title from 'optional arguments' to 'options' or 'optionals'

    parser._optionals.title
    'optional arguments'

But I don't think there's a need to change references in the code or its comments from 'optionals' to 'options'.  I like the parallelism between 'optionals' and 'positionals'.  The terms are well defined in the code.  During parsing, the 'required' attribute is only used at the end to check for missing arguments.

In Stackoverflow questions I'm tended to talk about 'flagged arguments'.

I still favor encouraging users to define their argument group(s), and making it easier to modify the titles of the two predefined groups.  I don't see enough of a consensus on alternative titles to make more sweeping changes.

I am willing to drop most of the code and comment changes if that moves us closer to consensus. Paul: would you accept changing the heading “optional arguments” to “options” in 3.6?

I thought we were close to consensus to use “options”, but maybe that is my own bias showing. Re-reading through this thread, here is a summary of the people that seem to prefer renaming the “optional arguments” heading:

Steven: “flag arguments”, but concerned about affecting existing help messages
David: options, switches
Eric: replace adjective “optional” but leave “. . . arguments”
Ben (OP): options
Martin Panter: options, flags, switches, or non-positional arguments
Oliver: switches
Terry: keyword arguments
Ryan: options, or other suggestions

Martin d'Anjou and Raymond seem to prefer splitting out a third group, “required arguments”, although it is not clear how that would tie in with the “positional arguments” group.

I think making it easier to modify the headings is worthwhile only if we deprecated the old heading in favour of using a new default name in the future. To me the problem is the default behaviour, not that argparse is inflexible.

Another option might be to fix Issue 10529 (explain using gettext with argparse), and then use that to translate the "optional arguments" string. But I don’t know how to do this.

Another thought is if we added a notice to Porting to Python 3.6 in What’s New, that might ease concerns about changing the heading.

Posting argparse_option.v2.patch, which is minimally complete version of Ryan’s patch. I have dropped all the nonessential code and documentation tweaks. I also added a What’s New entry. I’d like to know if people think this is the right direction to move in.

I'd like to note that this also happens with a required mutually exclusive group:

group = parser.add_mutually_exclusive_group(required=True)

The arguments in the group are listed under “optional arguments:”.

I'm guessing the mechanism is the same.

Is there anything that can be done to help this issue move along? I just ran into it just now

Mark,

Have you tried defining your own Argument Group?  If that didn't work, what fix do you want?  Why?

I have just run into the same issue here: https://bugs.python.org/issue38950

- I prefer Terry J. Reedy's "keyword arguments" as it is clear and consistent with "positional arguments".
- But Steven Bethard 's "flag arguments" looks fine since it is well known to shell users.
- Martin Panter's "options" looks okay since it is the standard name in GNU Coreutils (https://www.gnu.org/software/coreutils/manual/coreutils.html#Common-options). However I don't like it very much as it is still ambiguous: "options", like "optional arguments", still suggests something that is non required. And "options" is less consistent with "positional arguments" (nobody seems to have suggested "option arguments").
- Oliver Smith's "switches" does not look okay because it is not general enough since it is commonly restricted to Boolean arguments.

Anyway, the first 3 solutions are better than the current "optional arguments". What is blocking the approval of Martin Panter's PR?

It seems the discussion has so far revolved around 'optional' arguments with `required=True`.

What about the other way around?

While trying to set `required=False` for a positional argument raises an exception, it is still possible to make the positional argument effectively optional by `nargs='?'` (it is then printed in brackets in usage message).

Based on the attached survey of practices, I propose a minimal edit to the help display.  Instead of "optional arguments:", we say "the options are as follows:".

The use of the word "option" is dominant is in the CLI world, followed by "action" and "switch".  The noun form "option" doesn't seem to cause the same confusion that arises in the adjective form "optional arguments" which strongly implies "not required".

For the documentation, I suggest adding a sentence or two in the introduction to explain the terminology used throughout the rest of the argparse docs.

Since any chance to the help output will likely break tests, marking this as 3.10 only.

I also like Eric'c suggestion of just using "options:" instead of "optional arguments".

Since this change will break tests that rely matching help output exactly, I  would like to hear if there are any objections to replacing "optional arguments" with "options".

The words "switch" or "flag" don't work as well because they imply on/off and don't encompass option that take arguments.

I wouldn't let breaking these tests deter you from improving the output. I think using "options" is an improvement.

New changeset 41b223d29cdfeb1f222c12c3abaccc3bc128f5e7 by Raymond Hettinger in branch 'master':
bpo-9694: Fix misleading phrase "optional arguments" (GH-23858)
https://github.com/python/cpython/commit/41b223d29cdfeb1f222c12c3abaccc3bc128f5e7

Coudl this please be mentioned on https://docs.python.org/3.10/whatsnew/3.10.html ?

At least two packages fail tests because of the change (ipython and sphinxcontrib-autoprogram).

Since this issue is closed it might be a good idea to open a new one with this problem.  And if possible identify the failed tests.  

We forgot to allow for the fact that working code/tests might be checking for specific help messages, checks the will fail when this group label is changed.

https://bugs.python.org/issue42870