Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test suite skips failing tests when setUp[Class] fails #90681

Closed
sobolevn opened this issue Jan 25, 2022 · 3 comments
Closed

Test suite skips failing tests when setUp[Class] fails #90681

sobolevn opened this issue Jan 25, 2022 · 3 comments
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@sobolevn
Copy link
Member

BPO 46523
Nosy @gvanrossum, @ezio-melotti, @sobolevn, @Fidget-Spinner
PRs
  • bpo-46523: fix tests rerun when setUp[Class|Module] fails #30895
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2022-01-25.15:42:34.695>
labels = ['type-bug', 'tests', '3.9', '3.10', '3.11']
title = 'Test suite skips failing tests when setUp[Class] fails'
updated_at = <Date 2022-01-25.20:48:08.548>
user = 'https://github.com/sobolevn'

    bugs.python.org fields:

    activity = <Date 2022-01-25.20:48:08.548>
actor = 'sobolevn'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2022-01-25.15:42:34.695>
creator = 'sobolevn'
dependencies = []
files = []
hgrepos = []
issue_num = 46523
keywords = ['patch']
message_count = 2.0
messages = ['411620', '411625']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'ezio.melotti', 'sobolevn', 'kj']
pr_nums = ['30895']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46523'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
    @sobolevn
    Copy link
    Member Author

    Here's what happened. We had an error in test_typing.py, which was silently ignored.

     ======================================================================
ERROR: setUpClass (test.test_typing.NewTypeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\test_typing.py", line 3917, in setUpClass
    UserId = NewType('UserId', int)
NameError: name 'NewType' is not defined

----------------------------------------------------------------------
Ran 396 tests in 0.085s

FAILED (errors=1, skipped=1)
test test_typing failed

    Link: https://github.com/python/cpython/runs/4902363883?check_suite_focus=true

    But, later the suite runner tried to rerun it:

    0:09:12 load avg: 6.37 Re-running failed tests in verbose mode
0:09:12 load avg: 6.37 Re-running test_typing in verbose mode (matching: setUpClass)

1 re-run test:
    test_typing

1 test run no tests:
    test_typing

    And since nothing matched setUpClass - no tests were executed and the CI went green instead of red.

    What can we do?

    1. Only schedule real test_ item to be rerun, fail for everything else
    2. Convert setupClass failure into the whole class rerun
    3. Other options?

    I would like to work on this, when we will decide which way is best.

    @sobolevn sobolevn added 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Jan 25, 2022
    @Fidget-Spinner
    Copy link
    Member

    I vote for option 2.

    @Fidget-Spinner Fidget-Spinner changed the title Test suite skips failing tests Test suite skips failing tests when setUp[Class] fails Jan 25, 2022
    @Fidget-Spinner Fidget-Spinner changed the title Test suite skips failing tests Test suite skips failing tests when setUp[Class] fails Jan 25, 2022
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @sobolevn sobolevn mentioned this issue Jul 5, 2022
    Merged
    @furkanonder
    Copy link
    Sponsor Contributor

    @sobolevn The issue seems to have been solved. We can close the issue.

    @sobolevn sobolevn closed this as completed May 9, 2023
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @sobolevn @furkanonder @Fidget-Spinner