Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

regrtest: Add --fail-env-changed option #74949

Closed
vstinner opened this issue Jun 26, 2017 · 21 comments
Closed

regrtest: Add --fail-env-changed option #74949

vstinner opened this issue Jun 26, 2017 · 21 comments
Labels
3.7 (EOL) end of life tests Tests in the Lib/test dir type-feature A feature request or enhancement

Comments

@vstinner
Copy link
Member

BPO 30764
Nosy @vstinner
PRs
  • bpo-30764: regrtest: add --fail-env-changed option #2402
  • bpo-30764: test_subprocess uses SuppressCrashReport #2405
  • [3.6] bpo-30764: test_subprocess uses SuppressCrashReport (#2405) #2410
  • [3.5] bpo-30764: test_subprocess uses SuppressCrashReport (#2405) #2411
  • [2.7] bpo-30764: test_subprocess uses SuppressCrashReport (#2405) #2412
  • bpo-30764: Windows support.SuppressCrashReport #2423
  • [3.6] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master #2441
  • [3.5] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master #2442
  • [2.7] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master #2444
  • [2.7] bpo-30764: Fix test_subprocess on macOS #2521
  • bpo-30764: Fix regrtest --fail-env-changed --forever #2536
  • [3.6] bpo-30764: Fix regrtest --fail-env-changed --forever (#2536) #2539
  • [3.5] bpo-29512, bpo-30764, bpo-30776: Backport regrtest enhancements from 3.6 to 3.5 #2540
  • [2.7] bpo-29512, bpo-30764: Backport regrtest enhancements from 3.5 to 2.7 #2541
    		•
    Files
  • suppress_crash.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-06-27.21:37:48.617>
created_at = <Date 2017-06-26.12:17:09.833>
labels = ['3.7', 'type-feature', 'tests']
title = 'regrtest: Add --fail-env-changed option'
updated_at = <Date 2017-07-05.13:44:55.955>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2017-07-05.13:44:55.955>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-06-27.21:37:48.617>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-06-26.12:17:09.833>
creator = 'vstinner'
dependencies = []
files = ['46976']
hgrepos = []
issue_num = 30764
keywords = []
message_count = 21.0
messages = ['296883', '296885', '296902', '296908', '296911', '296913', '296958', '296960', '296965', '296966', '296972', '297028', '297036', '297043', '297046', '297075', '297549', '297558', '297567', '297577', '297744']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['2402', '2405', '2410', '2411', '2412', '2423', '2441', '2442', '2444', '2521', '2536', '2539', '2540', '2541']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue30764'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']
    @vstinner
    Copy link
    Member Author

    The be able to debug bpo-30280 using my bisection script:
    https://github.com/haypo/misc/blob/205fea00d509ddd1cffa30c5c6f2da012f3ec54e/python/bisect_test.py

    I would like to be able to mark tests which alter the environment as failed, using an explicit option.

    It would be nice to be able to enable this option by default, or even remove the option and just make the behaviour as the default, but it would require to fix some issues first (like bpo-30280).

    @vstinner vstinner added 3.7 (EOL) end of life tests Tests in the Lib/test dir type-feature A feature request or enhancement labels Jun 26, 2017
    @vstinner
    Copy link
    Member Author

    I implemented the feature:
    #2402

    My PR changes also the exit code from 1 to 2 (tests failed) or 3 (interrupted), to be able to distinguish how regrtest failed. It should also help to bisect failing tests.

    @vstinner
    Copy link
    Member Author

    I updated my PR to use the exit code 130 when regrtest is interrupted. Serhiy Storchaka found a reference:

    "On Linux scripts terminated with Ctrl-C return exit code 130 (http://tldp.org/LDP/abs/html/exitcodes.html)."

    @vstinner
    Copy link
    Member Author

    New changeset cdee3f1 by Victor Stinner in branch 'master':
    bpo-30764: test_subprocess uses SuppressCrashReport (bpo-2405)
    cdee3f1

    @vstinner
    Copy link
    Member Author

    New changeset 849b062 by Victor Stinner in branch '3.5':
    bpo-30764: test_subprocess uses SuppressCrashReport (bpo-2405) (bpo-2411)
    849b062

    @vstinner
    Copy link
    Member Author

    New changeset 63f54c6 by Victor Stinner in branch 'master':
    bpo-30764: regrtest: add --fail-env-changed option (bpo-2402)
    63f54c6

    @vstinner
    Copy link
    Member Author

    New changeset 9ad50d9 by Victor Stinner in branch '3.6':
    bpo-30764: test_subprocess uses SuppressCrashReport (bpo-2405) (bpo-2410)
    9ad50d9

    @vstinner
    Copy link
    Member Author

    New changeset 2097b9e by Victor Stinner in branch '2.7':
    [2.7] bpo-30764: test_subprocess uses SuppressCrashReport (bpo-2405) (bpo-2412)
    2097b9e

    @vstinner
    Copy link
    Member Author

    I tried to implement test.support.SuppressCrashReport on Windows using only ctypes for Python 2.7, but I failed: see attached suppress_crash.py. So I created #2423 which adds new functions and constants to the _testcapi module. IMHO it's more reliabe to do that, but it adds more C code.

    @vstinner
    Copy link
    Member Author

    Test failure on x86-64 Sierra 2.7 buildbot:

    http://buildbot.python.org/all/builders/x86-64%20Sierra%202.7/builds/111/steps/test/logs/stdio

    test_shell_string_with_spaces (test.test_subprocess.CommandsWithSpaces) ... test test_subprocess failed -- Traceback (most recent call last):
  File "/Users/buildbot/buildarea/2.7.billenstein-sierra/build/Lib/test/test_subprocess.py", line 1302, in test_child_terminated_in_stopped_state
    self.assertLess(returncode, 0)  # signal death, likely SIGSEGV.
AssertionError: 1 not less than 0

    skipped 'mswindows only'

    ======================================================================
    FAIL: test_child_terminated_in_stopped_state (test.test_subprocess.POSIXProcessTestCase)
    Test wait() behavior when waitpid returns WIFSTOPPED; bpo-29335.
    ----------------------------------------------------------------------

    Traceback (most recent call last):
  File "/Users/buildbot/buildarea/2.7.billenstein-sierra/build/Lib/test/test_subprocess.py", line 1302, in test_child_terminated_in_stopped_state
    self.assertLess(returncode, 0)  # signal death, likely SIGSEGV.
AssertionError: 1 not less than 0

    @vstinner
    Copy link
    Member Author

    Oops, I used the wrong bpo number for the test_subprocess fixes: the right bpo is bpo-30448!

    @vstinner
    Copy link
    Member Author

    New changeset 95a91df by Victor Stinner in branch '2.7':
    bpo-30764: Windows support.SuppressCrashReport (bpo-2423)
    95a91df

    @vstinner
    Copy link
    Member Author

    New changeset 35d2ca2 by Victor Stinner in branch '3.6':
    [3.6] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (bpo-2441)
    35d2ca2

    @vstinner
    Copy link
    Member Author

    New changeset de1850b by Victor Stinner in branch '3.5':
    [3.5] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (bpo-2442)
    de1850b

    @vstinner
    Copy link
    Member Author

    New changeset fea98bf by Victor Stinner in branch '2.7':
    [2.7] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (bpo-2444)
    fea98bf

    @vstinner
    Copy link
    Member Author

    The initial issue, the "regrtest: Add --fail-env-changed option" feature, has been implemented in 2.7, 3.5; 3.6 and master branches. So I close this issue.

    Again, this issue got hijacked by the test_subprocess change, bpo-30448, because I wrote the wrong bpo number in my commits ;-)

    @vstinner
    Copy link
    Member Author

    vstinner commented Jul 3, 2017

    New changeset 5e87592 by Victor Stinner in branch 'master':
    bpo-30764: Fix regrtest --fail-env-changed --forever (bpo-2536)
    5e87592

    @vstinner
    Copy link
    Member Author

    vstinner commented Jul 3, 2017

    New changeset 4132adb by Victor Stinner in branch '3.6':
    bpo-30764: Fix regrtest --fail-env-changed --forever (bpo-2536) (bpo-2539)
    4132adb

    @vstinner
    Copy link
    Member Author

    vstinner commented Jul 3, 2017

    New changeset 0b12107 by Victor Stinner in branch '3.5':
    [3.5] bpo-29512, bpo-30764, bpo-30776: Backport regrtest enhancements from 3.6 to 3.5 (bpo-2540)
    0b12107

    @vstinner
    Copy link
    Member Author

    vstinner commented Jul 3, 2017

    New changeset d7955b8 by Victor Stinner in branch '2.7':
    [2.7] bpo-29512, bpo-30764: Backport regrtest enhancements from 3.5 to 2.7 (bpo-2541)
    d7955b8

    @vstinner
    Copy link
    Member Author

    vstinner commented Jul 5, 2017

    New changeset 8207c17 by Victor Stinner in branch 'master':
    Revert "bpo-30822: Fix testing of datetime module." (bpo-2588)
    8207c17

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life tests Tests in the Lib/test dir type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @vstinner