classification
Title: Argparse incorrectly handles '--' as argument to option
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.9, Python 3.8, Python 3.7, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: bethard, eric.araujo, jssfr, maker, martin.panter, paul.j3, r.david.murray, rhettinger, shihai1991
Priority: high Keywords: needs review, patch

Created on 2012-03-18 14:29 by maker, last changed 2019-10-07 08:54 by shihai1991.

Files
File name Uploaded Description Edit
foo.py maker, 2012-03-18 14:29
issue14364.test.patch maker, 2013-04-19 08:12 review
dbldash.patch paul.j3, 2016-12-17 17:52 review
Pull Requests
URL Status Linked Edit
PR 15714 open shihai1991, 2019-09-06 05:58
Messages (16)
msg156254 - (view) Author: Michele Orrù (maker) * Date: 2012-03-18 14:29
http://docs.python.org/library/argparse.html#arguments-containing 
The attached file shows different behaviours when using '--' immediately after an optional argument.

tumbolandia:cpython maker$ python foo.py --test=-- foo 
[]
tumbolandia:cpython maker$ python foo.py --test -- foo 
usage: foo.py [-h] [-t TEST] [yuri]
foo.py: error: argument -t/--test: expected 1 argument(s)

The same is for single-dash arguments.

tumbolandia:cpython maker$ python foo.py -t -- foo
usage: foo.py [-h] [-t TEST] [yuri]
foo.py: error: argument -t/--test: expected 1 argument(s)
tumbolandia:cpython maker$ python foo.py -t-- foo
[]

Obviously argparse should return an error in both cases.
The  bug is probably due to Lib/argparser.py:2211
msg156256 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-18 14:45
It does look like there's anomalous behavior here of some sort, but I'd expect --test=-- to result in test="--", myself, rather than an error.  My intuition is that '--' would need to be preceded by a space to function as the 'end of options' marker.  Because of that, I've never tried the above scenario with a unix command, so maybe my intuition is wrong :)

Just for reference (the code paths may be different), there is another open issue about -- parsing, issue 13922.
msg156275 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2012-03-18 21:31
I just tried this with grep's "-e" and "--regexp":

$ cat > temp.txt
a--b
cdef
$ grep -e-- -v temp.txt
cdef
$ grep --regexp=-- -v temp.txt
cdef
$ grep -e -- -v temp.txt
cdef
$ grep --regexp -- -v temp.txt
cdef

And with diff's "-I" and "--ignore-matching-lines":

$ cat > temp2.txt
cdef
a--b
$ diff temp.txt temp2.txt
1d0
< a--b
2a2
> a--b
$ diff -I-- temp.txt temp2.txt 
$ diff -I -- temp.txt temp2.txt 
$ diff --ignore-matching-lines -- temp.txt temp2.txt 
$ diff --ignore-matching-lines=-- temp.txt temp2.txt 
$

Note though that for options that don't take an argument, the "--" is just removed:

$ grep -v -- a temp.txt 
cdef
$ diff -i -- temp.txt temp2.txt
1d0
< a--b
2a2
> a--b

So I guess the unix rule is: if an option that takes an argument is followed by "--", use that as the option's argument and continue parsing as usual. If an option that takes no argument is followed by "--", then delete the "--" and treat all following flags as positional arguments.

Argparse can't follow this directly, because then people who are using "--" to signal the end of an option with nargs="*" would start getting "--" included in that list. (And I know people have used "--" this way for a while.)

I guess my preference is what R. David Murray suggests: "--" when part of an argument (i.e. not separated by spaces) is treated like any other characters, and only a lone "--" signals the end of options (and is ignored otherwise). That would mean that both "--test=--" and "-t--" in your example would give you ["--"], and the other errors would stay as you saw them.
msg156293 - (view) Author: Michele Orrù (maker) * Date: 2012-03-18 23:40
+1 also for me. 
I will try to work for a patch in the next days. :)
msg187283 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2013-04-18 19:27
@Michele could you provide a patch for this?
msg187344 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2013-04-19 06:20
The patch that I  recently submitted for http://bugs.python.org/issue13922
appears to solve this issue.  It only removes the '--' that marked the end of options.  

With:

parser = argparse.ArgumentParser()
parser.add_argument('-f','--foo')
print(parser.parse_args(['-f--']))
print(parser.parse_args(['--foo=--']))
print(parser.parse_args(['-f', '--']))

I get:

Namespace(foo='--')
Namespace(foo='--')
usage: foodash.py [-h] [-f FOO]
foodash.py: error: argument -f/--foo: expected one argument
msg187346 - (view) Author: Michele Orrù (maker) * Date: 2013-04-19 06:51
wow, I was just writing the unittests, thanks paul. 
Shall I continue? I don't see any test case on tip.
msg187349 - (view) Author: Michele Orrù (maker) * Date: 2013-04-19 08:12
Yes, http://bugs.python.org/file29845/dbldash.patch seems to fix this.
Attaching the unittests, and noisying on your issue.
msg262545 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-28 01:49
In Python 3.5, this does not seem fixed. Issue 13922 is marked as resolved and fixed, but Pauls’s patch has not actually been committed.

>>> ap = ArgumentParser()
>>> ap.add_argument("-t", "--test", type=str, nargs=1)
_StoreAction(option_strings=['-t', '--test'], dest='test', nargs=1, const=None, default=None, type=<class 'str'>, choices=None, help=None, metavar=None)
>>> ap.add_argument('yuri', nargs='?')
_StoreAction(option_strings=[], dest='yuri', nargs='?', const=None, default=None, type=None, choices=None, help=None, metavar=None)
>>> ap.parse_args(["--test=--", "foo"])
Namespace(test=[], yuri='foo')
>>> ap.parse_args(["--test", "--", "foo"])
usage: [-h] [-t TEST] [yuri]
: error: argument -t/--test: expected 1 argument
argparse.ArgumentError: argument -t/--test: expected 1 argument

During handling of the above exception, another exception occurred:

__main__.SystemExit: 2
>>> ap.parse_args(["-t--", "foo"])
Namespace(test=[], yuri='foo')
>>> ap.parse_args(["-t", "--", "foo"])
usage: [-h] [-t TEST] [yuri]
: error: argument -t/--test: expected 1 argument
argparse.ArgumentError: argument -t/--test: expected 1 argument

During handling of the above exception, another exception occurred:

__main__.SystemExit: 2
msg262558 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2016-03-28 05:55
I made a mistake of trying to add to or refine a closed patch.  Maybe I need to move the dbldash.patch over here for more formal consideration.
msg283501 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2016-12-17 17:52
I've copied 'dbldash.patch' over from http://bugs.python.org/issue13922.  I mistakenly added it to a closed issue.

The patch is old, and should be revised.  It is also missing tests for this '--opt=--' case, even though it solves it.   

I also reference this patch in http://bugs.python.org/issue9571, which tries to refine dbldash handling for PARSER, REMAINDER nargs.

I'm raising the priority on this because the issue has a come up a couple of times on Stackoverflow:

http://stackoverflow.com/questions/40685320/python-argparse-with-as-the-value
msg351230 - (view) Author: hai shi (shihai1991) * Date: 2019-09-06 05:44
some test cases which paul provided looks doesn't keep compatible.

In TestDoubleDashRemoval:
# output in my env is Namespace(cmd='cmd', foo=None, rest=['--', '--foo'])
('-- cmd -- -- --foo', NS(cmd='cmd', foo=None, rest=['--', '--', '--foo']))

# output in my env is Namespace(cmd='--', foo='1', rest=['1', '2'])
('-f1 -- -- 1 -- 2', NS(cmd='--', foo='1', rest=['1', '--', '2']))

# output in my env is Namespace(cmd='--foo', foo=None, rest=['--bar', '2'])
('-- --foo -- --bar 2', NS(cmd='--foo', foo=None, rest=['--', '--bar', '2']))

# output in my env is Namespace(cmd='cmd', foo=None, rest=['--foo', '-f2'])
('cmd -- --foo -- -f2', NS(cmd='cmd', foo=None, rest=['--foo', '--', '-f2']))


In TestDoubleDashRemoval1:
# output in my env is Namespace(cmd='cmd', foo='1', rest=['-f2', '3'])
('-f1 -- cmd -- -f2 3', NS(cmd='cmd', foo='1', rest=['--', '-f2', '3']))

my python's version is: 2.7.5, 3.6.8
msg353865 - (view) Author: Jonas Schäfer (jssfr) Date: 2019-10-03 16:19
Since I have been adversely affected by this bug ([1]), I looked at the patches.

I combined issue14364.test.patch (which adds test cases for --foo=--) and dbldash.patch in my local working tree and that seems to resolve the issue (tests pass if and only if I apply dbldash.patch and my reproducer from [1]) passes too).

The patches do not contain any type of metainformation, so I’m not comfortable with submitting this as a PR. I am also not at all familiar with the process of managing the changelog etc.

This should serve as a confirmation that this issue can be resolved with the patches. Someone more familiar than me with the process could take up the task to get this merged.

Note: As you can see in [1], this issue has already caused data loss.

   [1]: https://github.com/borgbackup/borg/issues/4769
msg354001 - (view) Author: hai shi (shihai1991) * Date: 2019-10-05 10:43
Hi, Jonas. Thanks for your report.
Let us wait core team's discuss;)
msg354049 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-10-06 21:22
Paul and David, is the current PR complete in your opinion?  Are there known cases where the PR would break existing, working code?

Also there needs to be doc entry that is clear on both new and old strategy for distinguishing options from arguments when a dash or dashdash is present.  The PR's Misc/NEWS entry, "Fix behavior of argparse when '--' as argument to option", is insufficient.
msg354066 - (view) Author: hai shi (shihai1991) * Date: 2019-10-07 08:54
> The PR's Misc/NEWS entry, "Fix behavior of argparse when '--' as argument to option", is insufficient.

Thanks for your comment, Raymond. I would continue update the desc.
History
Date User Action Args
2019-10-07 08:54:41shihai1991setmessages: + msg354066
2019-10-06 21:22:36rhettingersetassignee: rhettinger
messages: + msg354049
versions: + Python 3.8, Python 3.9, - Python 3.6
2019-10-05 10:43:35shihai1991setmessages: + msg354001
2019-10-03 16:19:59jssfrsetnosy: + jssfr
messages: + msg353865
2019-09-06 05:58:28shihai1991setnosy: + rhettinger
type: behavior
2019-09-06 05:58:00shihai1991setstage: needs patch -> patch review
pull_requests: + pull_request15369
2019-09-06 05:44:55shihai1991setnosy: + shihai1991
messages: + msg351230
2016-12-17 17:52:20paul.j3setfiles: + dbldash.patch
priority: normal -> high
versions: + Python 3.6, Python 3.7, - Python 3.2, Python 3.3
messages: + msg283501

keywords: + needs review
2016-03-28 05:55:28paul.j3setmessages: + msg262558
2016-03-28 01:49:32martin.pantersetnosy: + martin.panter

messages: + msg262545
title: Argparse incorrectly handles '--' -> Argparse incorrectly handles '--' as argument to option
2014-02-03 18:32:34BreamoreBoysetnosy: - BreamoreBoy
2013-04-19 08:12:09makersetfiles: + issue14364.test.patch
keywords: + patch
messages: + msg187349
2013-04-19 06:51:42makersetmessages: + msg187346
2013-04-19 06:20:51paul.j3setmessages: + msg187344
2013-04-19 05:09:46paul.j3setnosy: + paul.j3
2013-04-18 19:27:49BreamoreBoysetnosy: + BreamoreBoy
messages: + msg187283
2012-03-24 15:21:54eric.araujosetnosy: + eric.araujo
2012-03-18 23:40:46makersetmessages: + msg156293
2012-03-18 21:31:48bethardsetmessages: + msg156275
2012-03-18 14:45:00r.david.murraysetversions: + Python 3.2, Python 3.3, - Python 3.1
nosy: + r.david.murray, bethard

messages: + msg156256

components: + Library (Lib)
stage: needs patch
2012-03-18 14:29:12makercreate