classification
Title: Argparse incorrectly handles '--' as argument to option
Type: Stage: needs patch
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: bethard, eric.araujo, maker, martin.panter, paul.j3, r.david.murray
Priority: high Keywords: needs review, patch

Created on 2012-03-18 14:29 by maker, last changed 2016-12-17 17:52 by paul.j3.

Files
File name Uploaded Description Edit
foo.py maker, 2012-03-18 14:29
issue14364.test.patch maker, 2013-04-19 08:12 review
dbldash.patch paul.j3, 2016-12-17 17:52 review
Messages (11)
msg156254 - (view) Author: Michele Orrù (maker) * Date: 2012-03-18 14:29
http://docs.python.org/library/argparse.html#arguments-containing 
The attached file shows different behaviours when using '--' immediately after an optional argument.

tumbolandia:cpython maker$ python foo.py --test=-- foo 
[]
tumbolandia:cpython maker$ python foo.py --test -- foo 
usage: foo.py [-h] [-t TEST] [yuri]
foo.py: error: argument -t/--test: expected 1 argument(s)

The same is for single-dash arguments.

tumbolandia:cpython maker$ python foo.py -t -- foo
usage: foo.py [-h] [-t TEST] [yuri]
foo.py: error: argument -t/--test: expected 1 argument(s)
tumbolandia:cpython maker$ python foo.py -t-- foo
[]

Obviously argparse should return an error in both cases.
The  bug is probably due to Lib/argparser.py:2211
msg156256 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-18 14:45
It does look like there's anomalous behavior here of some sort, but I'd expect --test=-- to result in test="--", myself, rather than an error.  My intuition is that '--' would need to be preceded by a space to function as the 'end of options' marker.  Because of that, I've never tried the above scenario with a unix command, so maybe my intuition is wrong :)

Just for reference (the code paths may be different), there is another open issue about -- parsing, issue 13922.
msg156275 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2012-03-18 21:31
I just tried this with grep's "-e" and "--regexp":

$ cat > temp.txt
a--b
cdef
$ grep -e-- -v temp.txt
cdef
$ grep --regexp=-- -v temp.txt
cdef
$ grep -e -- -v temp.txt
cdef
$ grep --regexp -- -v temp.txt
cdef

And with diff's "-I" and "--ignore-matching-lines":

$ cat > temp2.txt
cdef
a--b
$ diff temp.txt temp2.txt
1d0
< a--b
2a2
> a--b
$ diff -I-- temp.txt temp2.txt 
$ diff -I -- temp.txt temp2.txt 
$ diff --ignore-matching-lines -- temp.txt temp2.txt 
$ diff --ignore-matching-lines=-- temp.txt temp2.txt 
$

Note though that for options that don't take an argument, the "--" is just removed:

$ grep -v -- a temp.txt 
cdef
$ diff -i -- temp.txt temp2.txt
1d0
< a--b
2a2
> a--b

So I guess the unix rule is: if an option that takes an argument is followed by "--", use that as the option's argument and continue parsing as usual. If an option that takes no argument is followed by "--", then delete the "--" and treat all following flags as positional arguments.

Argparse can't follow this directly, because then people who are using "--" to signal the end of an option with nargs="*" would start getting "--" included in that list. (And I know people have used "--" this way for a while.)

I guess my preference is what R. David Murray suggests: "--" when part of an argument (i.e. not separated by spaces) is treated like any other characters, and only a lone "--" signals the end of options (and is ignored otherwise). That would mean that both "--test=--" and "-t--" in your example would give you ["--"], and the other errors would stay as you saw them.
msg156293 - (view) Author: Michele Orrù (maker) * Date: 2012-03-18 23:40
+1 also for me. 
I will try to work for a patch in the next days. :)
msg187283 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2013-04-18 19:27
@Michele could you provide a patch for this?
msg187344 - (view) Author: paul j3 (paul.j3) * Date: 2013-04-19 06:20
The patch that I  recently submitted for http://bugs.python.org/issue13922
appears to solve this issue.  It only removes the '--' that marked the end of options.  

With:

parser = argparse.ArgumentParser()
parser.add_argument('-f','--foo')
print(parser.parse_args(['-f--']))
print(parser.parse_args(['--foo=--']))
print(parser.parse_args(['-f', '--']))

I get:

Namespace(foo='--')
Namespace(foo='--')
usage: foodash.py [-h] [-f FOO]
foodash.py: error: argument -f/--foo: expected one argument
msg187346 - (view) Author: Michele Orrù (maker) * Date: 2013-04-19 06:51
wow, I was just writing the unittests, thanks paul. 
Shall I continue? I don't see any test case on tip.
msg187349 - (view) Author: Michele Orrù (maker) * Date: 2013-04-19 08:12
Yes, http://bugs.python.org/file29845/dbldash.patch seems to fix this.
Attaching the unittests, and noisying on your issue.
msg262545 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-28 01:49
In Python 3.5, this does not seem fixed. Issue 13922 is marked as resolved and fixed, but Pauls’s patch has not actually been committed.

>>> ap = ArgumentParser()
>>> ap.add_argument("-t", "--test", type=str, nargs=1)
_StoreAction(option_strings=['-t', '--test'], dest='test', nargs=1, const=None, default=None, type=<class 'str'>, choices=None, help=None, metavar=None)
>>> ap.add_argument('yuri', nargs='?')
_StoreAction(option_strings=[], dest='yuri', nargs='?', const=None, default=None, type=None, choices=None, help=None, metavar=None)
>>> ap.parse_args(["--test=--", "foo"])
Namespace(test=[], yuri='foo')
>>> ap.parse_args(["--test", "--", "foo"])
usage: [-h] [-t TEST] [yuri]
: error: argument -t/--test: expected 1 argument
argparse.ArgumentError: argument -t/--test: expected 1 argument

During handling of the above exception, another exception occurred:

__main__.SystemExit: 2
>>> ap.parse_args(["-t--", "foo"])
Namespace(test=[], yuri='foo')
>>> ap.parse_args(["-t", "--", "foo"])
usage: [-h] [-t TEST] [yuri]
: error: argument -t/--test: expected 1 argument
argparse.ArgumentError: argument -t/--test: expected 1 argument

During handling of the above exception, another exception occurred:

__main__.SystemExit: 2
msg262558 - (view) Author: paul j3 (paul.j3) * Date: 2016-03-28 05:55
I made a mistake of trying to add to or refine a closed patch.  Maybe I need to move the dbldash.patch over here for more formal consideration.
msg283501 - (view) Author: paul j3 (paul.j3) * Date: 2016-12-17 17:52
I've copied 'dbldash.patch' over from http://bugs.python.org/issue13922.  I mistakenly added it to a closed issue.

The patch is old, and should be revised.  It is also missing tests for this '--opt=--' case, even though it solves it.   

I also reference this patch in http://bugs.python.org/issue9571, which tries to refine dbldash handling for PARSER, REMAINDER nargs.

I'm raising the priority on this because the issue has a come up a couple of times on Stackoverflow:

http://stackoverflow.com/questions/40685320/python-argparse-with-as-the-value
History
Date User Action Args
2016-12-17 17:52:20paul.j3setfiles: + dbldash.patch
priority: normal -> high
versions: + Python 3.6, Python 3.7, - Python 3.2, Python 3.3
messages: + msg283501

keywords: + needs review
2016-03-28 05:55:28paul.j3setmessages: + msg262558
2016-03-28 01:49:32martin.pantersetnosy: + martin.panter

messages: + msg262545
title: Argparse incorrectly handles '--' -> Argparse incorrectly handles '--' as argument to option
2014-02-03 18:32:34BreamoreBoysetnosy: - BreamoreBoy
2013-04-19 08:12:09makersetfiles: + issue14364.test.patch
keywords: + patch
messages: + msg187349
2013-04-19 06:51:42makersetmessages: + msg187346
2013-04-19 06:20:51paul.j3setmessages: + msg187344
2013-04-19 05:09:46paul.j3setnosy: + paul.j3
2013-04-18 19:27:49BreamoreBoysetnosy: + BreamoreBoy
messages: + msg187283
2012-03-24 15:21:54eric.araujosetnosy: + eric.araujo
2012-03-18 23:40:46makersetmessages: + msg156293
2012-03-18 21:31:48bethardsetmessages: + msg156275
2012-03-18 14:45:00r.david.murraysetversions: + Python 3.2, Python 3.3, - Python 3.1
nosy: + r.david.murray, bethard

messages: + msg156256

components: + Library (Lib)
stage: needs patch
2012-03-18 14:29:12makercreate