classification
Title: regrtest: Add --fail-env-changed option
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: vstinner
Priority: normal Keywords:

Created on 2017-06-26 12:17 by vstinner, last changed 2017-07-05 13:44 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
suppress_crash.py vstinner, 2017-06-26 22:47
Pull Requests
URL Status Linked Edit
PR 2402 merged vstinner, 2017-06-26 12:19
PR 2405 merged vstinner, 2017-06-26 13:00
PR 2410 merged vstinner, 2017-06-26 15:29
PR 2411 merged vstinner, 2017-06-26 15:33
PR 2412 merged vstinner, 2017-06-26 16:15
PR 2423 merged vstinner, 2017-06-26 22:40
PR 2441 merged vstinner, 2017-06-27 13:50
PR 2442 merged vstinner, 2017-06-27 14:15
PR 2444 merged vstinner, 2017-06-27 14:49
PR 2521 closed vstinner, 2017-06-30 21:13
PR 2536 merged vstinner, 2017-07-03 08:47
PR 2539 merged vstinner, 2017-07-03 10:40
PR 2540 merged vstinner, 2017-07-03 10:57
PR 2541 merged vstinner, 2017-07-03 11:30
Messages (21)
msg296883 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 12:17
The be able to debug bpo-30280 using my bisection script:
https://github.com/haypo/misc/blob/205fea00d509ddd1cffa30c5c6f2da012f3ec54e/python/bisect_test.py

I would like to be able to mark tests which alter the environment as failed, using an explicit option.

It would be nice to be able to enable this option by default, or even remove the option and just make the behaviour as the default, but it would require to fix some issues first (like bpo-30280).
msg296885 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 12:20
I implemented the feature:
https://github.com/python/cpython/pull/2402

My PR changes also the exit code from 1 to 2 (tests failed) or 3 (interrupted), to be able to distinguish how regrtest failed. It should also help to bisect failing tests.
msg296902 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 14:22
I updated my PR to use the exit code 130 when regrtest is interrupted. Serhiy Storchaka found a reference:

"On Linux scripts terminated with Ctrl-C return exit code 130 (http://tldp.org/LDP/abs/html/exitcodes.html)."
msg296908 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 15:23
New changeset cdee3f14f7f4c995e7eedb0bf6a67e260c739f7d by Victor Stinner in branch 'master':
bpo-30764: test_subprocess uses SuppressCrashReport (#2405)
https://github.com/python/cpython/commit/cdee3f14f7f4c995e7eedb0bf6a67e260c739f7d
msg296911 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 16:05
New changeset 849b062a82ca2f09e33259d34067faba196c9e23 by Victor Stinner in branch '3.5':
bpo-30764: test_subprocess uses SuppressCrashReport (#2405) (#2411)
https://github.com/python/cpython/commit/849b062a82ca2f09e33259d34067faba196c9e23
msg296913 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 16:33
New changeset 63f54c68936d648c70ca411661e4208329edcf26 by Victor Stinner in branch 'master':
bpo-30764: regrtest: add --fail-env-changed option (#2402)
https://github.com/python/cpython/commit/63f54c68936d648c70ca411661e4208329edcf26
msg296958 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 22:00
New changeset 9ad50d94599aed0c37beaf78948ec271c8aa3881 by Victor Stinner in branch '3.6':
bpo-30764: test_subprocess uses SuppressCrashReport (#2405) (#2410)
https://github.com/python/cpython/commit/9ad50d94599aed0c37beaf78948ec271c8aa3881
msg296960 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 22:00
New changeset 2097b9e0ef32ab7a0d745edc0f707c615780c006 by Victor Stinner in branch '2.7':
[2.7] bpo-30764: test_subprocess uses SuppressCrashReport (#2405) (#2412)
https://github.com/python/cpython/commit/2097b9e0ef32ab7a0d745edc0f707c615780c006
msg296965 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 22:47
I tried to implement test.support.SuppressCrashReport on Windows using only ctypes for Python 2.7, but I failed: see attached suppress_crash.py. So I created https://github.com/python/cpython/pull/2423 which adds new functions and constants to the _testcapi module. IMHO it's more reliabe to do that, but it adds more C code.
msg296966 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 22:50
Test failure on x86-64 Sierra 2.7 buildbot:

http://buildbot.python.org/all/builders/x86-64%20Sierra%202.7/builds/111/steps/test/logs/stdio

test_shell_string_with_spaces (test.test_subprocess.CommandsWithSpaces) ... test test_subprocess failed -- Traceback (most recent call last):
  File "/Users/buildbot/buildarea/2.7.billenstein-sierra/build/Lib/test/test_subprocess.py", line 1302, in test_child_terminated_in_stopped_state
    self.assertLess(returncode, 0)  # signal death, likely SIGSEGV.
AssertionError: 1 not less than 0

skipped 'mswindows only'

======================================================================
FAIL: test_child_terminated_in_stopped_state (test.test_subprocess.POSIXProcessTestCase)
Test wait() behavior when waitpid returns WIFSTOPPED; issue29335.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/2.7.billenstein-sierra/build/Lib/test/test_subprocess.py", line 1302, in test_child_terminated_in_stopped_state
    self.assertLess(returncode, 0)  # signal death, likely SIGSEGV.
AssertionError: 1 not less than 0
msg296972 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 23:14
Oops, I used the wrong bpo number for the test_subprocess fixes: the right bpo is bpo-30448!
msg297028 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-27 12:31
New changeset 95a91df0363f88722ea2860d9d42a0b04b3f8be2 by Victor Stinner in branch '2.7':
bpo-30764: Windows support.SuppressCrashReport (#2423)
https://github.com/python/cpython/commit/95a91df0363f88722ea2860d9d42a0b04b3f8be2
msg297036 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-27 14:04
New changeset 35d2ca2b94a6ff29e763ddb7727166f0592edfa2 by Victor Stinner in branch '3.6':
[3.6] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2441)
https://github.com/python/cpython/commit/35d2ca2b94a6ff29e763ddb7727166f0592edfa2
msg297043 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-27 14:35
New changeset de1850bb03f8225cbff85f437b6e972bf9b68c2a by Victor Stinner in branch '3.5':
[3.5] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2442)
https://github.com/python/cpython/commit/de1850bb03f8225cbff85f437b6e972bf9b68c2a
msg297046 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-27 14:56
New changeset fea98bfcff6ccf9142daa97677fe86c1fdf8e63e by Victor Stinner in branch '2.7':
[2.7] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2444)
https://github.com/python/cpython/commit/fea98bfcff6ccf9142daa97677fe86c1fdf8e63e
msg297075 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-27 21:37
The initial issue, the "regrtest: Add --fail-env-changed option" feature, has been implemented in 2.7, 3.5; 3.6 and master branches. So I close this issue.

Again, this issue got hijacked by the test_subprocess change, bpo-30448, because I wrote the wrong bpo number in my commits ;-)
msg297549 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 09:16
New changeset 5e87592fd12e0b7c41edc11d4885ed7298d5063b by Victor Stinner in branch 'master':
bpo-30764: Fix regrtest --fail-env-changed --forever (#2536)
https://github.com/python/cpython/commit/5e87592fd12e0b7c41edc11d4885ed7298d5063b
msg297558 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 10:50
New changeset 4132adb4b8f90893d8f19ff46edff56f189faca0 by Victor Stinner in branch '3.6':
bpo-30764: Fix regrtest --fail-env-changed --forever (#2536) (#2539)
https://github.com/python/cpython/commit/4132adb4b8f90893d8f19ff46edff56f189faca0
msg297567 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 11:17
New changeset 0b1210739d12c26e4a161ffd494bd572d49b2483 by Victor Stinner in branch '3.5':
[3.5] bpo-29512, bpo-30764, bpo-30776: Backport regrtest enhancements from 3.6 to 3.5 (#2540)
https://github.com/python/cpython/commit/0b1210739d12c26e4a161ffd494bd572d49b2483
msg297577 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 13:07
New changeset d7955b8196578306e9d86f6c61c9cb3ee72edab0 by Victor Stinner in branch '2.7':
[2.7] bpo-29512, bpo-30764: Backport regrtest enhancements from 3.5 to 2.7 (#2541)
https://github.com/python/cpython/commit/d7955b8196578306e9d86f6c61c9cb3ee72edab0
msg297744 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-05 13:44
New changeset 8207c17486baece8ed0ac42d9f8d69ecec4ba7e4 by Victor Stinner in branch 'master':
Revert "bpo-30822: Fix testing of datetime module." (#2588)
https://github.com/python/cpython/commit/8207c17486baece8ed0ac42d9f8d69ecec4ba7e4
History
Date User Action Args
2017-07-05 13:44:55vstinnersetmessages: + msg297744
2017-07-03 13:07:56vstinnersetmessages: + msg297577
2017-07-03 11:30:57vstinnersetpull_requests: + pull_request2610
2017-07-03 11:17:00vstinnersetmessages: + msg297567
2017-07-03 10:57:21vstinnersetpull_requests: + pull_request2607
2017-07-03 10:50:38vstinnersetmessages: + msg297558
2017-07-03 10:40:24vstinnersetpull_requests: + pull_request2605
2017-07-03 09:16:03vstinnersetmessages: + msg297549
2017-07-03 08:47:17vstinnersetpull_requests: + pull_request2603
2017-06-30 21:13:42vstinnersetpull_requests: + pull_request2589
2017-06-27 21:37:48vstinnersetstatus: open -> closed
versions: + Python 2.7, Python 3.5, Python 3.6
messages: + msg297075

resolution: fixed
stage: resolved
2017-06-27 14:56:46vstinnersetmessages: + msg297046
2017-06-27 14:49:30vstinnersetpull_requests: + pull_request2499
2017-06-27 14:35:21vstinnersetmessages: + msg297043
2017-06-27 14:15:35vstinnersetpull_requests: + pull_request2495
2017-06-27 14:04:18vstinnersetmessages: + msg297036
2017-06-27 13:50:38vstinnersetpull_requests: + pull_request2492
2017-06-27 12:31:42vstinnersetmessages: + msg297028
2017-06-26 23:14:21vstinnersetmessages: + msg296972
2017-06-26 22:50:09vstinnersetmessages: + msg296966
2017-06-26 22:47:20vstinnersetfiles: + suppress_crash.py

messages: + msg296965
2017-06-26 22:40:32vstinnersetpull_requests: + pull_request2474
2017-06-26 22:00:54vstinnersetmessages: + msg296960
2017-06-26 22:00:36vstinnersetmessages: + msg296958
2017-06-26 16:33:21vstinnersetmessages: + msg296913
2017-06-26 16:15:43vstinnersetpull_requests: + pull_request2462
2017-06-26 16:05:23vstinnersetmessages: + msg296911
2017-06-26 15:33:58vstinnersetpull_requests: + pull_request2460
2017-06-26 15:29:23vstinnersetpull_requests: + pull_request2458
2017-06-26 15:23:05vstinnersetmessages: + msg296908
2017-06-26 14:22:18vstinnersetmessages: + msg296902
2017-06-26 13:00:30vstinnersetpull_requests: + pull_request2452
2017-06-26 12:20:25vstinnersetmessages: + msg296885
2017-06-26 12:19:23vstinnersetpull_requests: + pull_request2449
2017-06-26 12:17:09vstinnercreate