Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.10: Under some trivial circunstances, GIL not released #88811

Closed
jcea opened this issue Jul 15, 2021 · 11 comments
Closed

Python 3.10: Under some trivial circunstances, GIL not released #88811

jcea opened this issue Jul 15, 2021 · 11 comments
Labels
3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) release-blocker

Comments

@jcea
Copy link
Member

jcea commented Jul 15, 2021

BPO 44645
Nosy @jcea, @ambv, @markshannon, @pablogsal, @miss-islington
PRs
  • bpo-44645: Check for interrupts on any potentially backwards edge. #27167
  • [3.10] bpo-44645: Check for interrupts on any potentially backwards edge. #27183
  • Revert "bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)" #27194
  • [3.10] Revert "bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)" (GH-27194) #27195
  • bpo-44645: Check for interrupts on any potentially backwards edge #27216
  • [3.10] bpo-44645: Check for interrupts on any potentially backwards edge (GH-27216) #27235
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-07-31.23:21:29.452>
    created_at = <Date 2021-07-15.13:26:07.952>
    labels = ['interpreter-core', '3.10', 'release-blocker']
    title = 'Python 3.10: Under some trivial circunstances, GIL not released'
    updated_at = <Date 2021-07-31.23:21:29.452>
    user = 'https://github.com/jcea'

    bugs.python.org fields:

    activity = <Date 2021-07-31.23:21:29.452>
    actor = 'pablogsal'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-07-31.23:21:29.452>
    closer = 'pablogsal'
    components = ['Interpreter Core']
    creation = <Date 2021-07-15.13:26:07.952>
    creator = 'jcea'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 44645
    keywords = ['patch', '3.10regression']
    message_count = 11.0
    messages = ['397549', '397554', '397609', '397610', '397639', '397641', '397642', '397643', '397719', '397782', '397789']
    nosy_count = 5.0
    nosy_names = ['jcea', 'lukasz.langa', 'Mark.Shannon', 'pablogsal', 'miss-islington']
    pr_nums = ['27167', '27183', '27194', '27195', '27216', '27235']
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue44645'
    versions = ['Python 3.10']

    @jcea
    Copy link
    Member Author

    jcea commented Jul 15, 2021

    After #18334, a "while condition: pass" in a thread will preclude releasing the GIL, hanging the program with CPU at 100%.

    Trivial to reproduce:

    """
    #!/usr/bin/env python3.10

    import threading
    import time
    
    def worker():
        while cont:
            pass
    
    def main():
        global cont
        cont = True
        t = threading.Thread(target=worker)
        t.start()
        time.sleep(1)
        cont = False
        t.join()
    
    if __name__ == '__main__':
        main()
    """

    @jcea jcea added 3.10 only security fixes release-blocker interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Jul 15, 2021
    @markshannon
    Copy link
    Member

    #18334 assumes that since all loops will contain a backwards edge, checking for interrupts on JUMP_ABSOLUTE should be sufficient.

    However, #23743 changed the back edges in while statements. So worker compiles to:

    2 0 LOAD_GLOBAL 0 (cont)
    2 POP_JUMP_IF_FALSE 7 (to 14)

    3 >> 4 NOP

    2 6 LOAD_GLOBAL 0 (cont)
    8 POP_JUMP_IF_TRUE 2 (to 4)
    10 LOAD_CONST 0 (None)
    12 RETURN_VALUE
    >> 14 LOAD_CONST 0 (None)
    16 RETURN_VALUE

    which has no CALL or JUMP_ABSOLUTE instructions.

    @markshannon
    Copy link
    Member

    New changeset 000e70a by Mark Shannon in branch 'main':
    bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)
    000e70a

    @pablogsal
    Copy link
    Member

    New changeset 0e349ea by Mark Shannon in branch '3.10':
    [3.10] bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167) (GH-27183)
    0e349ea

    @pablogsal
    Copy link
    Member

    PR 27167 hangs or fails. in the refleak buildbots, so I am going to proceed to revert

    @pablogsal pablogsal reopened this Jul 16, 2021
    @pablogsal
    Copy link
    Member

    To reproduce the problem:

    $./python -m test test_threading -R : -v

    ...

    test_original_excepthook (test.test_threading.ExceptHookTests) ... ok
    test_system_exit (test.test_threading.ExceptHookTests) ... ok
    test_can_interrupt_tight_loops (test.test_threading.InterruptMainTests) ...

    (HANGS here or some time fails like in:

    ======================================================================
    FAIL: test_can_interrupt_tight_loops (test.test_threading.InterruptMainTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/buildbot/buildarea/pull_request.cstratak-RHEL7-ppc64le.refleak/build/Lib/test/support/threading_helper.py", line 62, in decorator
        return func(*args)
               ^^^^^^^^^^^
      File "/home/buildbot/buildarea/pull_request.cstratak-RHEL7-ppc64le.refleak/build/Lib/test/test_threading.py", line 1630, in test_can_interrupt_tight_loops
        self.assertNotEqual(iterations, 0)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AssertionError: 0 == 0

    https://buildbot.python.org/all/#/builders/411/builds/92

    @ambv
    Copy link
    Contributor

    ambv commented Jul 16, 2021

    New changeset c90c591 by Pablo Galindo Salgado in branch 'main':
    Revert "bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)" (bpo-27194)
    c90c591

    @ambv
    Copy link
    Contributor

    ambv commented Jul 16, 2021

    New changeset 42a5514 by Miss Islington (bot) in branch '3.10':
    Revert "bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)" (GH-27194) (bpo-27195)
    42a5514

    @markshannon
    Copy link
    Member

    Can we at least include the fix, until we can come up with a better test?

    That test fails on 3.9 as well as 3.10 and main.

    With a 3.9 build and the test in bpo-27194:

    ./python -m test test_threading -R :
    0:00:00 load avg: 0.95 Run tests sequentially
    0:00:00 load avg: 0.95 [1/1] test_threading
    beginning 9 repetitions
    123456789
    ...test test_threading failed -- Traceback (most recent call last):
      File "/home/mark/repos/cpython/Lib/test/test_threading.py", line 1479, in test_can_interrupt_tight_loops
        self.assertNotEqual(iterations, 0)
    AssertionError: 0 == 0

    I don't know what the underlying issue is.

    Can you observe a failure without the -R flag?

    @markshannon
    Copy link
    Member

    New changeset d09c134 by Mark Shannon in branch 'main':
    bpo-44645: Check for interrupts on any potentially backwards edge (GH-27216)
    d09c134

    @markshannon
    Copy link
    Member

    New changeset 37bdd22 by Miss Islington (bot) in branch '3.10':
    bpo-44645: Check for interrupts on any potentially backwards edge (GH-27216) (GH-27235)
    37bdd22

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) release-blocker
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants