[2.7] Backport test_regrtest (partially) on Python 2.7 #74469

vstinner · 2017-05-05T16:23:09Z

BPO	30283
Nosy	@vstinner, @serhiy-storchaka
PRs	[2.7] bpo-30283: Backport test_regrtest from master to 2.7 #1513 bpo-30283: Backport regrtest features from master to 2.7 #1516 [2.7] bpo-30283: regrtest: backport test_slow_interrupted() and test_coverage() #1541 [2.7] bpo-30283: Add NEWS entry for backported regrtest #2398

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2017-06-28.00:52:08.481>
created_at = <Date 2017-05-05.16:23:08.993>
labels = ['tests']
title = '[2.7] Backport test_regrtest (partially) on Python 2.7'
updated_at = <Date 2017-06-28.00:52:08.480>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-06-28.00:52:08.480>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2017-06-28.00:52:08.481>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-05-05.16:23:08.993>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 30283
keywords = []
message_count = 20.0
messages = ['293120', '293298', '293302', '293303', '293310', '293311', '293312', '293314', '293317', '293318', '293469', '293476', '293479', '293483', '293489', '293828', '293840', '293866', '297014', '297024']
nosy_count = 2.0
nosy_names = ['vstinner', 'serhiy.storchaka']
pr_nums = ['1513', '1516', '1541', '2398']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue30283'
versions = ['Python 2.7']

vstinner · 2017-05-05T16:23:09Z

I would like to continue to backport enhancements of regrtest from master to 2.7. In master, regrtest has so many super useful features!

This week, Serhiy and me backported some basic features, but we introduced regressions. I now would like to backport test_regrtest to reduce risk of breaking things (like buildbots and the Python CI).

vstinner · 2017-05-09T11:09:36Z

I backported test_regrtest from master to 2.7. I removed:

ParseArgsTestCase: regrtest.py of 2.7 has no API to easily write unit tests for argument parsing
test_huntrleaks_fd_leak(): this features doesn't exist in regrtest 2.7
test_coverage(): regrtest 2.7 argument parser doesn't support --coverage, moreover this is an issue with the test summary
check_executed_tests() doesn't check the final list "Tests results: xxx" since it doesn't exist in Python 2.7
test_list_tests(): --list-tests feature doesn't exist in 2.7
test_fromlist() only supports the most basic format: "test_xxx"
test_slowest_interrupted(): the test failed and I was too lazy to try to fix it, this test is fragile
test_wait(): no --wait feature in 2.7

I made minor changes (ex: super()) to port the code from Python 3.7 to 2.7. I removed code using faulthandler, since faulthandler doesn't exist in 2.7.

test_slowest() was renamed to test_slow(), since regrtest 2.7 has the cmdline option --slow, not --slowest.

I will try to fix test_coverage() later, after test_regrtest is added to 2.7.

test_crashed() uses "import ctypes; ctypes.string_at(0)" instead of "import faulthandler; faulthandler._sigsegv()", I hope that ctypes.string_at(0) is enough to crash on all platforms :-D Otherwise, I may add something to _testcapi.

I may try to backport/fix other issues later to fix other removed tests.

vstinner · 2017-05-09T11:57:22Z

New changeset d2aff60 by Victor Stinner in branch '2.7':
[2.7] bpo-30283: Backport test_regrtest from master to 2.7 (bpo-1513)
d2aff60

vstinner · 2017-05-09T12:13:21Z

Oh, test_regrtest of Python 3.5 only tests regrtest command line parser, it doesn't contain functional tests. We may also backport functional tests for 3.5.

vstinner · 2017-05-09T13:34:45Z

PR 1516 is the real stuff. I would like to enhance regrtest output in 2.7 to be able to debug buildbots stuck somewhere.

Example:
---
filecmp
[398/402] test_setcomps
[399/402] test_array
[400/402] test_tools
[24166 refs]
(...)
[24167 refs]
recursedown('@test_33891_tmp')
[401/402] test_trace

command timed out: 1200 seconds without output running ['make', 'buildbottest', 'TESTOPTS=-j2 -j4', 'TESTPYTHONOPTS=', 'TESTTIMEOUT=900'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1628.312249
---
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%202.7/builds/73/steps/test/logs/stdio

vstinner · 2017-05-09T13:36:21Z

I chose to not backport "loadavg: 1.73" feature (bpo-30263) in PR 1516. I still consider this feature as experimental, not sure that it's really useful :-)

vstinner · 2017-05-09T13:39:43Z

test_regrtest failed on "x86 Windows XP VS9.0 2.7" buildbot, but passed on "x86 Windows XP 2.7" buildbot?

---
[279/402/1] test_regrtest
test test_regrtest failed -- multiple errors occurred; run in verbose mode for details
---
http://buildbot.python.org/all/builders/x86%20Windows%20XP%20VS9.0%202.7/builds/140/steps/test/logs/stdio

The test isn't rerun in verbose mode. Another bug?

vstinner · 2017-05-09T14:00:30Z

The test isn't rerun in verbose mode. Another bug?

Yes, I opened bpo-30313.

vstinner · 2017-05-09T15:06:37Z

New changeset 453a685 by Victor Stinner in branch '2.7':
bpo-30283: Backport regrtest features from master to 2.7 (bpo-1516)
453a685

vstinner · 2017-05-09T15:08:20Z

I merged a first serie of enhancements for regrtest output in the 2.7 branch. Let's see how things are going, if it helps to debug stuck buildbots.

For Python 3.5, I'm less motivated to backport enhancements since Python 3.5 has faulthandler which is used in regrtest to kill a stuck test after 15 min with a nice Python traceback. But maybe I will do the similar backports, since I'm now trying to watching buildbots of all Python branches, not only matser ;-)

vstinner · 2017-05-11T00:47:28Z

Oh, test_regrtest still fails on x86 Windows XP VS9.0 2.7.

0:16:29 [283/402/1] test_regrtest failed
test test_regrtest failed -- multiple errors occurred; run in verbose mode for details

vstinner · 2017-05-11T09:13:42Z

Oh, test_regrtest still fails on x86 Windows XP VS9.0 2.7.

PCbuild/rt.bat doesn't work on VS9.0 since VS9.0 creates python(_d).exe in PC/VS9.0/{amd64,x86}/, not in PCbuild/{amd64,x86}/.

My new #1541 patch serie now starts with a fix for this issue.

vstinner · 2017-05-11T09:30:26Z

New changeset a5bb624 by Victor Stinner in branch '2.7':
[2.7] bpo-30283: regrtest: backport test_slow_interrupted() and test_coverage() (bpo-1541)
a5bb624

vstinner · 2017-05-11T09:41:30Z

@unittest.skipUnless(sysconfig.is_python_build(),

                    'test.bat script is not installed')

I had to remove this decorator from test_tools_buildbot_test() since sysconfig.is_python_build() is false on Python 2.7 built with VS 2008 my Windows VM, whereas I didn't install Python!? I don't know why.

Moreover, the decorator was missing on test_pcbuild_rt(), likely a mistake in the backport from master.

vstinner · 2017-05-11T11:15:48Z

I had to remove this decorator from test_tools_buildbot_test() since sysconfig.is_python_build() is false on Python 2.7 built with VS 2008 my Windows VM, whereas I didn't install Python!? I don't know why.

I opened bpo-30342 for this bug.

Oh, test_regrtest still fails on x86 Windows XP VS9.0 2.7.

With my third commit a5bb624, test_regrtest pass again on x86 Windows XP VS9.0 2.7 buildbot:

http://buildbot.python.org/all/builders/x86%20Windows%20XP%20VS9.0%202.7/builds/146

vstinner · 2017-05-17T00:22:51Z

Python 2.7 now has a nice test_regrtest and its regrtest now has many new features backported from master. test_regrtest pass on all 2.7 buildbots, I close the issue.

I created bpo-30383 to backport regrtest enhancements to Python 3.5.

serhiy-storchaka · 2017-05-17T05:55:44Z

Could you add a Misc/NEWS entry about backported regrtest features?

vstinner · 2017-05-17T18:15:14Z

Serhiy Storchaka: "Could you add a Misc/NEWS entry about backported regrtest features?"

Oh, I don't really consider that regrtest is part of "Python", but sure, I can document my changes in NEWS. I reopen the issue to not forget.

vstinner · 2017-06-27T06:42:03Z

New changeset 305f333 by Victor Stinner in branch '2.7':
bpo-30283: Add NEWS entry for backported regrtest (bpo-2398)
305f333

vstinner · 2017-06-27T12:02:24Z

I backported almost all regrtest features from master to 2.7. It should ease the development on Python 2.7 ;-)

I added a NEWS entry, thanks Serhiy for the reminder. Hopefully, with NEWS.d/ and blurb, we will get much less conflicts on NEWS entries now ;-)

I'm aware of a last bug: "./python -m test -R 3:3 test_regrtest" hangs on Windows, but I'm quite sure that it comes test_crashed() which opens a popup and I'm working on a fix. Anyway, I already opened a specific issue for that: bpo-30705.

It was a long journey (2 months), but I'm now quite happy to get a much features-full regrtest on Python 2.7!

Spoiler: You can also expect my bisect tool in next weeks in Python 2.7 as well ;-)

vstinner added the tests Tests in the Lib/test dir label May 5, 2017

vstinner self-assigned this May 5, 2017

vstinner closed this as completed May 17, 2017

vstinner reopened this May 17, 2017

vstinner closed this as completed Jun 28, 2017

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[2.7] Backport test_regrtest (partially) on Python 2.7 #74469

[2.7] Backport test_regrtest (partially) on Python 2.7 #74469

vstinner commented May 5, 2017

vstinner commented May 5, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 11, 2017

vstinner commented May 11, 2017

vstinner commented May 11, 2017

vstinner commented May 11, 2017

vstinner commented May 11, 2017

vstinner commented May 17, 2017

serhiy-storchaka commented May 17, 2017

vstinner commented May 17, 2017

vstinner commented Jun 27, 2017

vstinner commented Jun 27, 2017

[2.7] Backport test_regrtest (partially) on Python 2.7 #74469

[2.7] Backport test_regrtest (partially) on Python 2.7 #74469

Comments

vstinner commented May 5, 2017

vstinner commented May 5, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 9, 2017

vstinner commented May 11, 2017

vstinner commented May 11, 2017

vstinner commented May 11, 2017

vstinner commented May 11, 2017

vstinner commented May 11, 2017

vstinner commented May 17, 2017

serhiy-storchaka commented May 17, 2017

vstinner commented May 17, 2017

vstinner commented Jun 27, 2017

vstinner commented Jun 27, 2017