I have some Python code which takes advantage of the ability to prepopulate the argparse Namespace before passing it to argparse. This allows me to read sensitive settings such as usernames and passwords from DBs, environment variables, etc. in addition to the CLI for security so they don't appear as process arguments anyone could see.

Some of these scripts have usernames and passwords as required arguments since they cannot function without them. Unfortunately you hit this bug when you load them into the namespace yourself from some secure place of your choice and then pass them in:

    args = parser.parse_args(argv, namespace)

The following argparse code which doesn't respect that they were present already and throws a fatal error. In addition, the parse_known_args function is 246 lines of code which uses ugly nested cheater functions, and the seen_actions variable is in the stack of the function not a member variable, so there is no way you can cleanly override or work around the behavior with a subclass, either by replacing the function (too massive) or by fixing up the seen_actions variable (can't get to it on the stack from outside).

So, I suggest that this code should to be fixed so that it will respect any existing values in the Namespace if they are present:

        # make sure all required actions were present, and convert defaults.
        for action in self._actions:
            if action not in seen_actions:
                if action.required:
                    name = _get_action_name(action)
                    self.error(_('argument %s is required') % name)

Can you test on 3.6?  This might have been fixed and not back-ported.

What issue might have changed this behavior?  I'm not aware of any that were trying to change either the `seen_actions` and the 'required' tests, not any dealing with pre-existing values in Namespace.

The current handling of defaults complicates respecting pre-existing values.  Defaults are added to the namespace at the start of parsing (provided they aren't already present - that respects pre-existing values).  At the end of parsing, the same block that tests for required actions also tests if the defaults in the Namespace need to be converted with the `type` function.

So the code at the end of _parse_known_args has limited ability to distinguish between values in the Namespace that  were preloaded, were loaded as defaults, or were loaded during parsing.

I agree that _parse_known_args is complicated and difficult to customize.  But it's been like that from the start.  And the handling of defaults is also complicated (and made worse by the 'delayed evaluation' change some years ago).

For another bug/issue I'd like to make seen_actions (or seen_non_default_actions) available to the user for testing.  But that isn't easy in a backward compatible fashion.

I think the best choice is to pre-load non-required Actions only.  Preloading should be seen as an alternative way of setting defaults, not a way of getting around the 'required' test.  And if that isn't enough, do some of your own 'required' tests after parsing.

One refactoring that I'd like to see is to move all the tests at the end of _parse_know_known_args to its caller, parse_known_args.  It would have to return more values than the current namespace and extras, in particular the see_actions and seen_non_default_actions.

This would have, I think, several benefits:

- it would make easier to customize the testing as proposed by Matthew.  It probably would also make it easier to implement the nested-logical-groups that I propose in http://bugs.python.org/issue11588.

- It would put the '# Convert action default now instead of doing it before parsing arguments' step at the same code level as when the defaults were first loaded.

Currently defaults are set at the start of parse_known_args, but evaluated at the end of _parse_known_args.  That asymmetry has bugged me for some time.  I'm not aware of any bug/issues that have arisen from that, but still it doesn't look right.

Last time the 'required' tests were revised, it introduced an error in the handling of subparsers.  http://bugs.python.org/issue9253

Subparsers used to be required (default for positionals).  But now they are optional unless you add an explicit 'subparsers.required = True' statement.

http://bugs.python.org/issue26394
Have argparse provide ability to require a fallback value be present

is a related issue - getting 'required' values from a config file.

http://bugs.python.org/issue27859
argparse - subparsers does not retain namespace

may complicate this issue.  Due to changes in http://bugs.python.org/issue9351, a user defined namespace is not passed to the subparsers.  They get a fresh namespace, which is then copied onto the main namespace.

Another pre-existing namespace issue

http://bugs.python.org/issue28734

When positional nargs='?' or '*', the default (or []) overwrites the namespace value.  That's because the posiitonals are always 'seen' (by an empty string), and `get_values` has special handling.  This action interacts with `take_action` and the handling of mutually_exclusive_groups.