I would like to continue to backport enhancements of regrtest from master to 2.7. In master, regrtest has so many super useful features!

This week, Serhiy and me backported some basic features, but we introduced regressions. I now would like to backport test_regrtest to reduce risk of breaking things (like buildbots and the Python CI).

I backported test_regrtest from master to 2.7. I removed:

* ParseArgsTestCase: regrtest.py of 2.7 has no API to easily write unit tests for argument parsing
* test_huntrleaks_fd_leak(): this features doesn't exist in regrtest 2.7
* test_coverage(): regrtest 2.7 argument parser doesn't support --coverage, moreover this is an issue with the test summary
* check_executed_tests() doesn't check the final list "Tests results: xxx" since it doesn't exist in Python 2.7
* test_list_tests(): --list-tests feature doesn't exist in 2.7
* test_fromlist() only supports the most basic format: "test_xxx"
* test_slowest_interrupted(): the test failed and I was too lazy to try to fix it, this test is fragile
* test_wait(): no --wait feature in 2.7

I made minor changes (ex: super()) to port the code from Python 3.7 to 2.7. I removed code using faulthandler, since faulthandler doesn't exist in 2.7.

test_slowest() was renamed to test_slow(), since regrtest 2.7 has the cmdline option --slow, not --slowest.

I will try to fix test_coverage() later, after test_regrtest is added to 2.7.

test_crashed() uses "import ctypes; ctypes.string_at(0)" instead of "import faulthandler; faulthandler._sigsegv()", I hope that ctypes.string_at(0) is enough to crash on all platforms :-D Otherwise, I may add something to _testcapi.

I may try to backport/fix other issues later to fix other removed tests.

New changeset d2aff607199a24182714404777c80c7f180a35af by Victor Stinner in branch '2.7':
[2.7] bpo-30283: Backport test_regrtest from master to 2.7 (#1513)
https://github.com/python/cpython/commit/d2aff607199a24182714404777c80c7f180a35af

Oh, test_regrtest of Python 3.5 only tests regrtest command line parser, it doesn't contain functional tests. We may also backport functional tests for 3.5.

PR 1516 is the real stuff. I would like to enhance regrtest output in 2.7 to be able to debug buildbots stuck somewhere.

Example:
---
filecmp
[398/402] test_setcomps
[399/402] test_array
[400/402] test_tools
[24166 refs]
(...)
[24167 refs]
recursedown('@test_33891_tmp')
[401/402] test_trace

command timed out: 1200 seconds without output running ['make', 'buildbottest', 'TESTOPTS=-j2 -j4', 'TESTPYTHONOPTS=', 'TESTTIMEOUT=900'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1628.312249
---
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%202.7/builds/73/steps/test/logs/stdio

I chose to not backport "loadavg: 1.73" feature (bpo-30263) in PR 1516. I still consider this feature as experimental, not sure that it's really useful :-)

test_regrtest failed on "x86 Windows XP VS9.0 2.7" buildbot, but passed on "x86 Windows XP 2.7" buildbot?

---
[279/402/1] test_regrtest
test test_regrtest failed -- multiple errors occurred; run in verbose mode for details
---
http://buildbot.python.org/all/builders/x86%20Windows%20XP%20VS9.0%202.7/builds/140/steps/test/logs/stdio

The test isn't rerun in verbose mode. Another bug?

> The test isn't rerun in verbose mode. Another bug?

Yes, I opened bpo-30313.

New changeset 453a6857023a0a2c714772cda8717ecf03de64f5 by Victor Stinner in branch '2.7':
bpo-30283: Backport regrtest features from master to 2.7 (#1516)
https://github.com/python/cpython/commit/453a6857023a0a2c714772cda8717ecf03de64f5

I merged a first serie of enhancements for regrtest output in the 2.7 branch. Let's see how things are going, if it helps to debug stuck buildbots.

For Python 3.5, I'm less motivated to backport enhancements since Python 3.5 has faulthandler which is used in regrtest to kill a stuck test after 15 min with a nice Python traceback. But maybe I will do the similar backports, since I'm now trying to watching buildbots of all Python branches, not only matser ;-)

Oh, test_regrtest still fails on x86 Windows XP VS9.0 2.7.

0:16:29 [283/402/1] test_regrtest failed
test test_regrtest failed -- multiple errors occurred; run in verbose mode for details

> Oh, test_regrtest still fails on x86 Windows XP VS9.0 2.7.

PCbuild/rt.bat doesn't work on VS9.0 since VS9.0 creates python(_d).exe in PC/VS9.0/{amd64,x86}/, not in PCbuild/{amd64,x86}/.

My new https://github.com/python/cpython/pull/1541 patch serie now starts with a fix for this issue.

New changeset a5bb62436e25614276ac4b8e252a87f3fcc946cd by Victor Stinner in branch '2.7':
[2.7] bpo-30283: regrtest: backport test_slow_interrupted() and test_coverage() (#1541)
https://github.com/python/cpython/commit/a5bb62436e25614276ac4b8e252a87f3fcc946cd

-    @unittest.skipUnless(sysconfig.is_python_build(),
-                         'test.bat script is not installed')

I had to remove this decorator from test_tools_buildbot_test() since sysconfig.is_python_build() is false on Python 2.7 built with VS 2008 my Windows VM, whereas I didn't install Python!? I don't know why.

Moreover, the decorator was missing on test_pcbuild_rt(), likely a mistake in the backport from master.

> I had to remove this decorator from test_tools_buildbot_test() since sysconfig.is_python_build() is false on Python 2.7 built with VS 2008 my Windows VM, whereas I didn't install Python!? I don't know why.

I opened bpo-30342 for this bug.

> Oh, test_regrtest still fails on x86 Windows XP VS9.0 2.7.

With my third commit a5bb62436e25614276ac4b8e252a87f3fcc946cd, test_regrtest pass again on x86 Windows XP VS9.0 2.7 buildbot:

http://buildbot.python.org/all/builders/x86%20Windows%20XP%20VS9.0%202.7/builds/146

Python 2.7 now has a nice test_regrtest and its regrtest now has many new features backported from master. test_regrtest pass on all 2.7 buildbots, I close the issue.

I created bpo-30383 to backport regrtest enhancements to Python 3.5.

Could you add a Misc/NEWS entry about backported regrtest features?

Serhiy Storchaka: "Could you add a Misc/NEWS entry about backported regrtest features?"

Oh, I don't really consider that regrtest is part of "Python", but sure, I can document my changes in NEWS. I reopen the issue to not forget.

New changeset 305f333a447bc8cff62eece16fab7fe4a9bef71c by Victor Stinner in branch '2.7':
bpo-30283: Add NEWS entry for backported regrtest (#2398)
https://github.com/python/cpython/commit/305f333a447bc8cff62eece16fab7fe4a9bef71c

I backported almost all regrtest features from master to 2.7. It should ease the development on Python 2.7 ;-)

I added a NEWS entry, thanks Serhiy for the reminder. Hopefully, with NEWS.d/ and blurb, we will get much less conflicts on NEWS entries now ;-)

I'm aware of a last bug: "./python -m test -R 3:3 test_regrtest" hangs on Windows, but I'm quite sure that it comes test_crashed() which opens a popup and I'm working on a fix. Anyway, I already opened a specific issue for that: bpo-30705.

It was a long journey (2 months), but I'm now quite happy to get a much features-full regrtest on Python 2.7!

Spoiler: You can also expect my bisect tool in next weeks in Python 2.7 as well ;-)