classification
Title: [2.7] Backport test_regrtest (partially) on Python 2.7
Type: Stage: resolved
Components: Tests Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: vstinner Nosy List: serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2017-05-05 16:23 by vstinner, last changed 2017-06-28 00:52 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1513 merged vstinner, 2017-05-09 10:58
PR 1516 merged vstinner, 2017-05-09 13:10
PR 1541 merged vstinner, 2017-05-10 23:50
PR 2398 merged vstinner, 2017-06-26 09:28
Messages (20)
msg293120 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-05 16:23
I would like to continue to backport enhancements of regrtest from master to 2.7. In master, regrtest has so many super useful features!

This week, Serhiy and me backported some basic features, but we introduced regressions. I now would like to backport test_regrtest to reduce risk of breaking things (like buildbots and the Python CI).
msg293298 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-09 11:09
I backported test_regrtest from master to 2.7. I removed:

* ParseArgsTestCase: regrtest.py of 2.7 has no API to easily write unit tests for argument parsing
* test_huntrleaks_fd_leak(): this features doesn't exist in regrtest 2.7
* test_coverage(): regrtest 2.7 argument parser doesn't support --coverage, moreover this is an issue with the test summary
* check_executed_tests() doesn't check the final list "Tests results: xxx" since it doesn't exist in Python 2.7
* test_list_tests(): --list-tests feature doesn't exist in 2.7
* test_fromlist() only supports the most basic format: "test_xxx"
* test_slowest_interrupted(): the test failed and I was too lazy to try to fix it, this test is fragile
* test_wait(): no --wait feature in 2.7

I made minor changes (ex: super()) to port the code from Python 3.7 to 2.7. I removed code using faulthandler, since faulthandler doesn't exist in 2.7.

test_slowest() was renamed to test_slow(), since regrtest 2.7 has the cmdline option --slow, not --slowest.

I will try to fix test_coverage() later, after test_regrtest is added to 2.7.

test_crashed() uses "import ctypes; ctypes.string_at(0)" instead of "import faulthandler; faulthandler._sigsegv()", I hope that ctypes.string_at(0) is enough to crash on all platforms :-D Otherwise, I may add something to _testcapi.

I may try to backport/fix other issues later to fix other removed tests.
msg293302 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-09 11:57
New changeset d2aff607199a24182714404777c80c7f180a35af by Victor Stinner in branch '2.7':
[2.7] bpo-30283: Backport test_regrtest from master to 2.7 (#1513)
https://github.com/python/cpython/commit/d2aff607199a24182714404777c80c7f180a35af
msg293303 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-09 12:13
Oh, test_regrtest of Python 3.5 only tests regrtest command line parser, it doesn't contain functional tests. We may also backport functional tests for 3.5.
msg293310 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-09 13:34
PR 1516 is the real stuff. I would like to enhance regrtest output in 2.7 to be able to debug buildbots stuck somewhere.

Example:
---
filecmp
[398/402] test_setcomps
[399/402] test_array
[400/402] test_tools
[24166 refs]
(...)
[24167 refs]
recursedown('@test_33891_tmp')
[401/402] test_trace

command timed out: 1200 seconds without output running ['make', 'buildbottest', 'TESTOPTS=-j2 -j4', 'TESTPYTHONOPTS=', 'TESTTIMEOUT=900'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1628.312249
---
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%202.7/builds/73/steps/test/logs/stdio
msg293311 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-09 13:36
I chose to not backport "loadavg: 1.73" feature (bpo-30263) in PR 1516. I still consider this feature as experimental, not sure that it's really useful :-)
msg293312 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-09 13:39
test_regrtest failed on "x86 Windows XP VS9.0 2.7" buildbot, but passed on "x86 Windows XP 2.7" buildbot?

---
[279/402/1] test_regrtest
test test_regrtest failed -- multiple errors occurred; run in verbose mode for details
---
http://buildbot.python.org/all/builders/x86%20Windows%20XP%20VS9.0%202.7/builds/140/steps/test/logs/stdio

The test isn't rerun in verbose mode. Another bug?
msg293314 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-09 14:00
> The test isn't rerun in verbose mode. Another bug?

Yes, I opened bpo-30313.
msg293317 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-09 15:06
New changeset 453a6857023a0a2c714772cda8717ecf03de64f5 by Victor Stinner in branch '2.7':
bpo-30283: Backport regrtest features from master to 2.7 (#1516)
https://github.com/python/cpython/commit/453a6857023a0a2c714772cda8717ecf03de64f5
msg293318 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-09 15:08
I merged a first serie of enhancements for regrtest output in the 2.7 branch. Let's see how things are going, if it helps to debug stuck buildbots.

For Python 3.5, I'm less motivated to backport enhancements since Python 3.5 has faulthandler which is used in regrtest to kill a stuck test after 15 min with a nice Python traceback. But maybe I will do the similar backports, since I'm now trying to watching buildbots of all Python branches, not only matser ;-)
msg293469 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-11 00:47
Oh, test_regrtest still fails on x86 Windows XP VS9.0 2.7.

0:16:29 [283/402/1] test_regrtest failed
test test_regrtest failed -- multiple errors occurred; run in verbose mode for details
msg293476 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-11 09:13
> Oh, test_regrtest still fails on x86 Windows XP VS9.0 2.7.

PCbuild/rt.bat doesn't work on VS9.0 since VS9.0 creates python(_d).exe in PC/VS9.0/{amd64,x86}/, not in PCbuild/{amd64,x86}/.

My new https://github.com/python/cpython/pull/1541 patch serie now starts with a fix for this issue.
msg293479 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-11 09:30
New changeset a5bb62436e25614276ac4b8e252a87f3fcc946cd by Victor Stinner in branch '2.7':
[2.7] bpo-30283: regrtest: backport test_slow_interrupted() and test_coverage() (#1541)
https://github.com/python/cpython/commit/a5bb62436e25614276ac4b8e252a87f3fcc946cd
msg293483 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-11 09:41
-    @unittest.skipUnless(sysconfig.is_python_build(),
-                         'test.bat script is not installed')

I had to remove this decorator from test_tools_buildbot_test() since sysconfig.is_python_build() is false on Python 2.7 built with VS 2008 my Windows VM, whereas I didn't install Python!? I don't know why.

Moreover, the decorator was missing on test_pcbuild_rt(), likely a mistake in the backport from master.
msg293489 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-11 11:15
> I had to remove this decorator from test_tools_buildbot_test() since sysconfig.is_python_build() is false on Python 2.7 built with VS 2008 my Windows VM, whereas I didn't install Python!? I don't know why.

I opened bpo-30342 for this bug.

> Oh, test_regrtest still fails on x86 Windows XP VS9.0 2.7.

With my third commit a5bb62436e25614276ac4b8e252a87f3fcc946cd, test_regrtest pass again on x86 Windows XP VS9.0 2.7 buildbot:

http://buildbot.python.org/all/builders/x86%20Windows%20XP%20VS9.0%202.7/builds/146
msg293828 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-17 00:22
Python 2.7 now has a nice test_regrtest and its regrtest now has many new features backported from master. test_regrtest pass on all 2.7 buildbots, I close the issue.

I created bpo-30383 to backport regrtest enhancements to Python 3.5.
msg293840 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-17 05:55
Could you add a Misc/NEWS entry about backported regrtest features?
msg293866 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-17 18:15
Serhiy Storchaka: "Could you add a Misc/NEWS entry about backported regrtest features?"

Oh, I don't really consider that regrtest is part of "Python", but sure, I can document my changes in NEWS. I reopen the issue to not forget.
msg297014 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-27 06:42
New changeset 305f333a447bc8cff62eece16fab7fe4a9bef71c by Victor Stinner in branch '2.7':
bpo-30283: Add NEWS entry for backported regrtest (#2398)
https://github.com/python/cpython/commit/305f333a447bc8cff62eece16fab7fe4a9bef71c
msg297024 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-27 12:02
I backported almost all regrtest features from master to 2.7. It should ease the development on Python 2.7 ;-)

I added a NEWS entry, thanks Serhiy for the reminder. Hopefully, with NEWS.d/ and blurb, we will get much less conflicts on NEWS entries now ;-)

I'm aware of a last bug: "./python -m test -R 3:3 test_regrtest" hangs on Windows, but I'm quite sure that it comes test_crashed() which opens a popup and I'm working on a fix. Anyway, I already opened a specific issue for that: bpo-30705.

It was a long journey (2 months), but I'm now quite happy to get a much features-full regrtest on Python 2.7!

Spoiler: You can also expect my bisect tool in next weeks in Python 2.7 as well ;-)
History
Date User Action Args
2017-06-28 00:52:08vstinnersetstatus: open -> closed
resolution: fixed
2017-06-27 12:02:24vstinnersetmessages: + msg297024
2017-06-27 11:57:16vstinnersetpull_requests: - pull_request2486
2017-06-27 11:53:33vstinnersetpull_requests: + pull_request2486
2017-06-27 06:42:03vstinnersetmessages: + msg297014
2017-06-26 09:28:32vstinnersetpull_requests: + pull_request2445
2017-05-17 18:15:13vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg293866
2017-05-17 05:55:43serhiy.storchakasetmessages: + msg293840
2017-05-17 00:22:51vstinnersetstatus: open -> closed
versions: - Python 3.5
messages: + msg293828

resolution: fixed
stage: resolved
2017-05-11 11:15:48vstinnersetmessages: + msg293489
2017-05-11 09:41:30vstinnersetmessages: + msg293483
2017-05-11 09:30:25vstinnersetmessages: + msg293479
2017-05-11 09:13:41vstinnersetmessages: + msg293476
2017-05-11 00:47:28vstinnersetmessages: + msg293469
2017-05-10 23:50:26vstinnersetpull_requests: + pull_request1640
2017-05-09 15:08:20vstinnersetmessages: + msg293318
2017-05-09 15:06:37vstinnersetmessages: + msg293317
2017-05-09 14:00:30vstinnersetmessages: + msg293314
2017-05-09 13:39:43vstinnersetmessages: + msg293312
2017-05-09 13:36:21vstinnersetmessages: + msg293311
2017-05-09 13:34:45vstinnersetmessages: + msg293310
2017-05-09 13:10:27vstinnersetpull_requests: + pull_request1616
2017-05-09 12:13:21vstinnersetmessages: + msg293303
versions: + Python 3.5
2017-05-09 11:57:22vstinnersetmessages: + msg293302
2017-05-09 11:09:36vstinnersetmessages: + msg293298
2017-05-09 10:58:11vstinnersetpull_requests: + pull_request1613
2017-05-05 16:23:16vstinnersetassignee: vstinner
2017-05-05 16:23:08vstinnercreate