classification
Title: RFC: issue a warning in regrtest when no tests have been executed?
Type: Stage: resolved
Components: Tests Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: miss-islington, pablogsal, serhiy.storchaka, terry.reedy, vstinner
Priority: normal Keywords: patch

Created on 2018-07-30 10:48 by vstinner, last changed 2018-12-12 17:22 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 10150 merged pablogsal, 2018-10-27 12:28
PR 10800 merged vstinner, 2018-11-29 17:23
PR 10801 merged vstinner, 2018-11-29 17:40
PR 10802 merged miss-islington, 2018-11-29 20:18
PR 11132 open vstinner, 2018-12-12 17:22
Messages (9)
msg322667 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-30 10:48
Python test runner regrtest has many options to select tests: -u (resources), -m and --matchfile, -x (exclude), etc. Sometimes, I do a mistake and I run 0 test, but I still get a success.

Maybe regrtest should be modified to fail with an error when no test is run? At least, it would help to understand my mistake if regrtest could emit a warning in such case.

Example:
---
$ ./python  -W error  -m test test_asyncio -m test_set_default_executor_deprecation_warnings 
Run tests sequentially
0:00:00 load avg: 0.27 [1/1] test_asyncio

== Tests result: SUCCESS ==

1 test OK.

Total duration: 266 ms
Tests result: SUCCESS
---

Success ok, but is there a typo in the test method?

Another example:
---
$ ./python  -W error  -m test test_asyncio -m test_no_such_test -v
== CPython 3.8.0a0 (heads/pr/8533:50c0499950, Jul 30 2018, 12:41:53) [GCC 8.1.1 20180712 (Red Hat 8.1.1-5)]
== Linux-4.17.6-200.fc28.x86_64-x86_64-with-glibc2.26 little-endian
== cwd: /home/vstinner/prog/python/master/build/test_python_5505
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 0.31 [1/1] test_asyncio

----------------------------------------------------------------------
Ran 0 tests in 0.001s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 318 ms
Tests result: SUCCESS
---

Success but no test have been executed. It's not obvious at all without the verbose mode:
---
$ ./python  -W error  -m test test_asyncio -m test_no_such_test 
Run tests sequentially
0:00:00 load avg: 0.31 [1/1] test_asyncio

== Tests result: SUCCESS ==

1 test OK.

Total duration: 260 ms
Tests result: SUCCESS
---
msg322684 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-30 17:51
Do we need to output OK and SUCCESS many times? I would prefer shorter and totally different from the normal output in case of no tests ran.
msg322783 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-31 14:28
> I would prefer shorter and totally different from the normal output in case of no tests ran.

That's basically the whole purpose of the issue, yep ;-)
msg323087 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-08-03 21:56
A test file that runs no test functions should pass on the buildbots.  (If it imports the target file, it is not completely empty.  And I can think of a use case for WIP PRs that intentionally do not run anything.)  

But I also would like a visually different display, having been fooled more than once. For one thing, one can omit '(unittest.testCase)' from test class definitions ;-).
msg330705 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-29 17:17
New changeset 9724348b43a9005a449ba532ccd3c6726f031097 by Victor Stinner (Pablo Galindo) in branch 'master':
bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10150)
https://github.com/python/cpython/commit/9724348b43a9005a449ba532ccd3c6726f031097
msg330720 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-29 20:14
New changeset 36003003f26d0c30fc15ec4dc3b0d7697dff908e by Victor Stinner in branch '2.7':
bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10801)
https://github.com/python/cpython/commit/36003003f26d0c30fc15ec4dc3b0d7697dff908e
msg330722 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-29 20:15
New changeset 8a73cac618a050f4e74eb38ff43e48d9957a6dec by Victor Stinner in branch '3.7':
[3.7] bpo-34279: Synchronize regrtest with master (GH-10800)
https://github.com/python/cpython/commit/8a73cac618a050f4e74eb38ff43e48d9957a6dec
msg330725 - (view) Author: miss-islington (miss-islington) Date: 2018-11-29 20:39
New changeset 43d812692f9207520e1169ff88cd8d6c59cc4804 by Miss Islington (bot) in branch '3.6':
[3.7] bpo-34279: Synchronize regrtest with master (GH-10800)
https://github.com/python/cpython/commit/43d812692f9207520e1169ff88cd8d6c59cc4804
msg330728 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-29 21:58
Thanks Pablo! I merged your change in 2.7, 3.6, 3.7 and master branches.

To reuse the example of my first message, we now get:

vstinner@apu$ ./python  -W error -m test test_asyncio -m test_no_such_test 
Run tests sequentially
0:00:00 load avg: 1.21 [1/1] test_asyncio
test_asyncio run no tests

== Tests result: NO TEST RUN ==

1 test run no tests:
    test_asyncio

Total duration: 347 ms
Tests result: NO TEST RUN

vstinner@apu$ echo $?
0


So the exit status is still a success (code 0), but "Tests result: NO TEST RUN" is an hint that no test has been executed.

If we run two files, but one file runs no test, the output now contains "1 test run no tests: test_sys":

$ ./python -m test test_os test_sys -m test_access -v
== CPython 3.8.0a0 (heads/platform_popen:4fcc412d3d, Nov 29 2018, 21:21:53) [GCC 8.2.1 20181105 (Red Hat 8.2.1-5)]
== Linux-4.19.3-300.fc29.x86_64-x86_64-with-glibc2.28 little-endian
== cwd: /home/vstinner/prog/python/master/build/test_python_18420
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 1.05 [1/2] test_os
test_access (test.test_os.FileTests) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK
0:00:00 load avg: 1.05 [2/2] test_sys

----------------------------------------------------------------------

Ran 0 tests in 0.000s

OK
test_sys run no tests

== Tests result: SUCCESS ==

1 test OK.

1 test run no tests:
    test_sys

Total duration: 240 ms
Tests result: SUCCESS
History
Date User Action Args
2018-12-12 17:22:01vstinnersetpull_requests: + pull_request10363
2018-11-29 21:58:19vstinnersetstatus: open -> closed
versions: + Python 2.7, Python 3.6, Python 3.7
messages: + msg330728

resolution: fixed
stage: patch review -> resolved
2018-11-29 20:39:08miss-islingtonsetnosy: + miss-islington
messages: + msg330725
2018-11-29 20:18:45miss-islingtonsetpull_requests: + pull_request10049
2018-11-29 20:15:04vstinnersetmessages: + msg330722
2018-11-29 20:14:47vstinnersetmessages: + msg330720
2018-11-29 17:40:01vstinnersetpull_requests: + pull_request10048
2018-11-29 17:23:25vstinnersetpull_requests: + pull_request10046
2018-11-29 17:17:48vstinnersetmessages: + msg330705
2018-10-27 12:28:18pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request9477
2018-08-03 21:56:35terry.reedysetnosy: + terry.reedy
messages: + msg323087
2018-08-02 02:37:03ppperrysettitle: RFC: issue a warning in regrtest when no test have been executed? -> RFC: issue a warning in regrtest when no tests have been executed?
2018-07-31 14:28:32vstinnersetmessages: + msg322783
2018-07-30 17:51:55serhiy.storchakasetmessages: + msg322684
2018-07-30 10:48:27vstinnercreate