classification
Title: test.regrtest does not support multiple -x flags
Type: behavior Stage:
Components: Tests Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: bethard, cstratak, larry, ned.deily, python-dev, r.david.murray, serhiy.storchaka, vstinner, xiang.zhang
Priority: normal Keywords: patch

Created on 2016-10-10 17:37 by cstratak, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
regrtest_cli.patch vstinner, 2016-10-11 14:35 review
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Messages (16)
msg278435 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2016-10-10 17:37
Until python3.6.0a04 it was possible to pass multiple times the -x option at regrtest.

However since the 3.6.0b1 when trying the same thing I get an error. Test names are random.

python3 -m test.regrtest -x test_venv -x test_gdb
usage: python -m test [options] [test_name1 [test_name2 ...]]
python path/to/Lib/test/regrtest.py [options] [test_name1 [test_name2 ...]]
regrtest.py: error: unrecognized arguments: test_gdb

If however I omit the second -x, the tests run fine and correctly exclude the specific tests.

Was the change intentional or a side effect from something else?
msg278439 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-10 18:19
Seems this is related to some changes in argparse.
msg278440 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-10-10 18:52
That sounds like a regression in argparse that we'd better track down.
msg278458 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016-10-11 05:01
This regression is introduced in 4df2d43e995d. 

And actually to skip multiple tests, you should use python3 -m test.regrtest -x test_venv test_gdb. Although Charalampos's example could work, it actually treats -x as a test.
msg278459 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016-10-11 05:10
So should we still consider this a regression? Actually -x is used as a hint flag and should not be used as -x arg pair. The right usage could still work now.
msg278460 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-11 05:47
This issue is not specific to -x, but affects all options. Following lines work:

    ./python -m test.regrtest -v test_unary test_binop
    ./python -m test.regrtest -u all test_unary test_binop
    ./python -m test.regrtest test_unary test_binop -v
    ./python -m test.regrtest test_unary test_binop -u all

But following lines don't work:

    ./python -m test.regrtest test_unary -v test_binop
    ./python -m test.regrtest test_unary -u all test_binop
msg278461 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016-10-11 05:57
>   ./python -m test.regrtest test_unary -v test_binop
>   ./python -m test.regrtest test_unary -u all test_binop

These also fail in 3.6.0a4. You can only specify options after args now. test_binop is not valid option. Isn't this the expected behaviour?
msg278464 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-11 06:18
Indeed, previously -x in the middle of test names worked only by accident, because it was interpreted as the name of excluded test. Now it stops positional arguments.

Current behavior is explainable, but surprising. This part of argparse is not clear and can cause unexpected effects. I afraid that argparse is incorrectly used in tarfile.
msg278480 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-10-11 12:40
OK, so this isn't a regression in argparse itself?  I thought you meant it was, which is why I set it to release blocker.
msg278482 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-11 13:16
No, this is not a regression in argparse itself, nor in regrtest (as Xiang pointed, original example is incorrect use). But I'm not sure about the support of options after positional arguments. This is dangerous feature. I'm going to investigate this further and find whether all is correct or needed some code or documentation changes.
msg278487 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-10-11 13:36
Xiang: "This regression is introduced in 4df2d43e995d."

Sorry, my commit message is very short.

I made this change because I was very annoying of getting "unknown test -m" and "unknown test test_access" when using the -m option to filter tests:

   ./python -m test test_os -m test_access

It's annoying to have to type:

   ./python -m test -m test_access test_os

By the way, it seems weird to have two consecutive -m with two different meanings ;-)

I didn't expect any regression, since argparse is smart to parse arguments. If something is wrong, I wouldn't call it a bug in argparse, but a bug in regrtest, how regrtest uses argparse.
msg278490 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-10-11 14:35
Attached patch should fix the issue.

I didn't know that argparse was so strict: "arg1 -v arg2" is not supported by default, see the issue #14191, whereas it works well with optparse.

The patch uses parse_known_args() as a workaround, and then manually checks for unknown options.
msg278813 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-17 16:15
New changeset af06d9616c29 by Victor Stinner in branch '3.5':
Issue #28409: regrtest: fix the parser of command line arguments.
https://hg.python.org/cpython/rev/af06d9616c29

New changeset 26249f82c15d by Victor Stinner in branch '3.6':
Merge 3.6: Issue #28409: regrtest: fix the parser of command line arguments.
https://hg.python.org/cpython/rev/26249f82c15d

New changeset 9f03b3dbb929 by Victor Stinner in branch 'default':
Merge 3.7: Issue #28409: regrtest: fix the parser of command line arguments.
https://hg.python.org/cpython/rev/9f03b3dbb929
msg278814 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-10-17 16:19
Thanks for the bug report Charalampos! It's now fixed.

--

> Until python3.6.0a04 it was possible to pass multiple times the -x option at regrtest.

To be clear: the regrtest parser of command line argument was broken since Python 2.7 at least.

"./python -m test.regrtest -v test_binop -v test_unaryop" on Python 2.7 tries to run "-v" test...

I fixed Python 3.5, 3.6 and 3.7, but not Python 2.7. Python 3 uses argparse, whereas Python 2 uses getopt. I'm not interested to try to fix the Python 2 code using getopt.
msg278825 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-17 18:19
What is the difference between using parse_known_args() and nargs=argparse.REMAINDER?
msg278826 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-10-17 19:04
Serhiy Storchaka added the comment:
> What is the difference between using parse_known_args() and
nargs=argparse.REMAINDER?

Using REMAINDER, args become ["b", "-c", "d"] for -a b -c d.

I would be happy to not use parse_known_args(), but I didn't find how.
History
Date User Action Args
2017-03-31 16:36:17dstufftsetpull_requests: + pull_request910
2016-10-17 19:04:42vstinnersetmessages: + msg278826
2016-10-17 18:19:10serhiy.storchakasetmessages: + msg278825
2016-10-17 16:19:22vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg278814
2016-10-17 16:15:34python-devsetnosy: + python-dev
messages: + msg278813
2016-10-11 14:35:03vstinnersetfiles: + regrtest_cli.patch
keywords: + patch
messages: + msg278490
2016-10-11 13:36:15vstinnersetmessages: + msg278487
2016-10-11 13:16:12serhiy.storchakasetpriority: release blocker -> normal

messages: + msg278482
stage: needs patch ->
2016-10-11 12:40:16r.david.murraysetmessages: + msg278480
2016-10-11 06:18:46serhiy.storchakasetmessages: + msg278464
2016-10-11 05:57:00xiang.zhangsetmessages: + msg278461
2016-10-11 05:47:54serhiy.storchakasetmessages: + msg278460
2016-10-11 05:10:59xiang.zhangsetmessages: + msg278459
2016-10-11 05:01:48xiang.zhangsetnosy: + xiang.zhang
messages: + msg278458
2016-10-10 18:52:55r.david.murraysetpriority: normal -> release blocker
nosy: + ned.deily, r.david.murray, larry
messages: + msg278440

2016-10-10 18:19:05serhiy.storchakasetnosy: + serhiy.storchaka, bethard

messages: + msg278439
versions: + Python 3.5
2016-10-10 18:03:42serhiy.storchakasetnosy: + vstinner
stage: needs patch
type: behavior

versions: + Python 3.7
2016-10-10 17:37:52cstratakcreate