Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_pdb fails when only some tests are run #88126

Closed
vstinner opened this issue Apr 27, 2021 · 10 comments
Closed

test_pdb fails when only some tests are run #88126

vstinner opened this issue Apr 27, 2021 · 10 comments
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

BPO 43960
Nosy @vstinner, @miss-islington, @iritkatriel
PRs
  • bpo-43960: reset global breakpoint state at the beginning of a pdb te… #25673
  • [3.9] bpo-43960: test_pdb resets breakpoints (GH-25673) #25681
  • [3.8] bpo-43960: test_pdb resets breakpoints (GH-25673) #25682
  • [3.9] bpo-43960: reset breakpoints to make tests deterministic (GH-25673) #25691
  • [3.8] bpo-43960: test_pdb resets breakpoints to make tests deterministic (GH-25691) #25692
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-04-28.15:27:55.157>
    created_at = <Date 2021-04-27.21:01:37.636>
    labels = ['3.8', 'tests', '3.9', '3.10']
    title = 'test_pdb fails when only some tests are run'
    updated_at = <Date 2021-04-28.15:43:32.273>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-04-28.15:43:32.273>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-04-28.15:27:55.157>
    closer = 'vstinner'
    components = ['Tests']
    creation = <Date 2021-04-27.21:01:37.636>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 43960
    keywords = ['patch']
    message_count = 10.0
    messages = ['392131', '392144', '392146', '392181', '392186', '392188', '392201', '392217', '392225', '392226']
    nosy_count = 3.0
    nosy_names = ['vstinner', 'miss-islington', 'iritkatriel']
    pr_nums = ['25673', '25681', '25682', '25691', '25692']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue43960'
    versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    @vstinner
    Copy link
    Member Author

    test_pdb fails with the following commands. I tested on Linux, same/similar failure on Windows.

    See also bpo-41914: "test_pdb fails".

    $ cat bisect 
    test.test_pdb.PdbTestCase.test_run_module
    test.test_pdb.test_next_until_return_at_return_event
    test.test_pdb.test_pdb_next_command_in_generator_for_loop
    
    $ ./python -m test test_pdb  --matchfile=bisect -v
    == CPython 3.10.0a7+ (heads/master:6bd9288b80, Apr 27 2021, 22:16:25) [GCC 11.0.1 20210324 (Red Hat 11.0.1-0)]
    == Linux-5.11.15-300.fc34.x86_64-x86_64-with-glibc2.33 little-endian
    == cwd: /home/vstinner/python/master/build/test_python_7720æ
    == CPU count: 8
    == encodings: locale=UTF-8, FS=utf-8
    0:00:00 load avg: 1.31 Run tests sequentially
    0:00:00 load avg: 1.31 [1/1] test_pdb
    test_run_module (test.test_pdb.PdbTestCase) ... ok
    test_next_until_return_at_return_event (test.test_pdb)
    Doctest: test.test_pdb.test_next_until_return_at_return_event ... ok
    test_pdb_next_command_in_generator_for_loop (test.test_pdb)
    Doctest: test.test_pdb.test_pdb_next_command_in_generator_for_loop ... FAIL

    ======================================================================
    FAIL: test_pdb_next_command_in_generator_for_loop (test.test_pdb)
    Doctest: test.test_pdb.test_pdb_next_command_in_generator_for_loop
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/vstinner/python/master/Lib/doctest.py", line 2205, in runTest
        raise self.failureException(self.format_failure(new.getvalue()))
    AssertionError: Failed doctest test for test.test_pdb.test_pdb_next_command_in_generator_for_loop
      File "/home/vstinner/python/master/Lib/test/test_pdb.py", line 1182, in test_pdb_next_command_in_generator_for_loop

    File "/home/vstinner/python/master/Lib/test/test_pdb.py", line 1195, in test.test_pdb.test_pdb_next_command_in_generator_for_loop
    Failed example:
    with PdbTestInput(['break test_gen',
    'continue',
    'next',
    'next',
    'next',
    'continue']):
    test_function()
    Expected:
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(3)test_function()
    -> for i in test_gen():
    (Pdb) break test_gen
    Breakpoint 1 at <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>:1
    (Pdb) continue
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>(2)test_gen()
    -> yield 0
    (Pdb) next
    value 0
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>(3)test_gen()
    -> return 1
    (Pdb) next
    Internal StopIteration: 1
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(3)test_function()
    -> for i in test_gen():
    (Pdb) next
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(5)test_function()
    -> x = 123
    (Pdb) continue
    Got:
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(3)test_function()
    -> for i in test_gen():
    (Pdb) break test_gen
    Breakpoint 2 at <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>:1
    (Pdb) continue
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>(2)test_gen()
    -> yield 0
    (Pdb) next
    value 0
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>(3)test_gen()
    -> return 1
    (Pdb) next
    Internal StopIteration: 1
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(3)test_function()
    -> for i in test_gen():
    (Pdb) next
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(5)test_function()
    -> x = 123
    (Pdb) continue

    ----------------------------------------------------------------------

    Ran 3 tests in 0.186s

    FAILED (failures=1)
    test test_pdb failed
    test_pdb failed

    == Tests result: FAILURE ==

    1 test failed:
    test_pdb

    Total duration: 480 ms
    Tests result: FAILURE

    @vstinner vstinner added 3.10 only security fixes tests Tests in the Lib/test dir labels Apr 27, 2021
    @iritkatriel
    Copy link
    Member

    I have a fix, I'll make a PR in a bit.

    The issue here is the breakpoint number:

    Breakpoint 1 at <doctest ...

    vs

    Breakpoint 2 at <doctest ...

    The breakpoint number is global state, so tests can impact each other.

    I recently added a reset_Breakpoint() function that can be used to clear the global state at the beginning of the test: https://github.com/python/cpython/pull/21989/files

    So I will make this test use it.

    @iritkatriel
    Copy link
    Member

    I scanned the file and I think this was the only test where this was missing. test_pdb_continue_in_bottomframe doesn't have a reset, but it doesn't expect a specific breakpoint number, it does:

    Breakpoint ... at <doctest test

    So with this patch I think the tests shouldn't interfere with each other anymore.

    @vstinner
    Copy link
    Member Author

    New changeset 21b02b5 by Irit Katriel in branch 'master':
    bpo-43960: test_pdb resets breakpoints (GH-25673)
    21b02b5

    @vstinner
    Copy link
    Member Author

    Irit: the automated backported failed, test_pdb fails with:

    NameError: name 'reset_Breakpoint' is not defined

    Do you want to try to backport manually the change to Python 3.9 (and then I can automate the backport to 3.8).

    test_pdb on Python 3.8 and 3.9 are also affected by this issue.

    @iritkatriel
    Copy link
    Member

    We didn't backport PR21989 because it was changing the way breakpoints hare maintained, so reset_Breakpoint is not there before 3.10. I can add that to the test file manually.

    @vstinner
    Copy link
    Member Author

    We didn't backport PR21989 because it was changing the way breakpoints hare maintained, so reset_Breakpoint is not there before 3.10. I can add that to the test file manually.

    Ah yes, it sounds reasonable to me to add reset_Breakpoint() to test_pdb with the bdb clearBreakpoints() code from master.

    @vstinner
    Copy link
    Member Author

    New changeset 2dc6b17 by Irit Katriel in branch '3.9':
    bpo-43960: test_pdb resets breakpoints to make tests deterministic (GH-25691)
    2dc6b17

    @vstinner vstinner added 3.8 only security fixes 3.9 only security fixes labels Apr 28, 2021
    @vstinner
    Copy link
    Member Author

    New changeset b52cc7c by Miss Islington (bot) in branch '3.8':
    bpo-43960: test_pdb resets breakpoints to make tests deterministic (GH-25691) (GH-25692)
    b52cc7c

    @vstinner
    Copy link
    Member Author

    Thanks for the fix Irit!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes tests Tests in the Lib/test dir
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants