classification
Title: Python 3.10: Under some trivial circunstances, GIL not released
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, jcea, lukasz.langa, miss-islington, pablogsal
Priority: release blocker Keywords: 3.10regression, patch

Created on 2021-07-15 13:26 by jcea, last changed 2021-07-31 23:21 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27167 merged Mark.Shannon, 2021-07-15 15:42
PR 27183 merged Mark.Shannon, 2021-07-16 10:05
PR 27194 merged pablogsal, 2021-07-16 16:29
PR 27195 merged miss-islington, 2021-07-16 17:05
PR 27216 merged Mark.Shannon, 2021-07-17 12:03
PR 27235 merged miss-islington, 2021-07-19 10:10
Messages (11)
msg397549 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2021-07-15 13:26
After https://github.com/python/cpython/pull/18334, a "while condition: pass" in a thread will preclude releasing the GIL, hanging the program with CPU at 100%.

Trivial to reproduce:

"""
#!/usr/bin/env python3.10

import threading
import time

def worker():
    while cont:
        pass

def main():
    global cont
    cont = True
    t = threading.Thread(target=worker)
    t.start()
    time.sleep(1)
    cont = False
    t.join()

if __name__ == '__main__':
    main()
"""
msg397554 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-07-15 14:12
https://github.com/python/cpython/pull/18334 assumes that since all loops will contain a backwards edge, checking for interrupts on JUMP_ABSOLUTE should be sufficient.

However, https://github.com/python/cpython/pull/23743 changed the back edges in while statements. So `worker` compiles to:

  2           0 LOAD_GLOBAL              0 (cont)
              2 POP_JUMP_IF_FALSE        7 (to 14)

  3     >>    4 NOP

  2           6 LOAD_GLOBAL              0 (cont)
              8 POP_JUMP_IF_TRUE         2 (to 4)
             10 LOAD_CONST               0 (None)
             12 RETURN_VALUE
        >>   14 LOAD_CONST               0 (None)
             16 RETURN_VALUE

which has no CALL or JUMP_ABSOLUTE instructions.
msg397609 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-07-16 09:59
New changeset 000e70ad5246732fcbd27cf59268185cbd5ad734 by Mark Shannon in branch 'main':
bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)
https://github.com/python/cpython/commit/000e70ad5246732fcbd27cf59268185cbd5ad734
msg397610 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-07-16 10:48
New changeset 0e349ea5541104c76cafc173bfcfef8de872f96f by Mark Shannon in branch '3.10':
[3.10] bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167) (GH-27183)
https://github.com/python/cpython/commit/0e349ea5541104c76cafc173bfcfef8de872f96f
msg397639 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-07-16 16:27
PR 27167 hangs or fails. in the refleak buildbots, so I am going to proceed to revert
msg397641 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-07-16 16:33
To reproduce the problem:

$./python -m test test_threading -R : -v

...

test_original_excepthook (test.test_threading.ExceptHookTests) ... ok
test_system_exit (test.test_threading.ExceptHookTests) ... ok
test_can_interrupt_tight_loops (test.test_threading.InterruptMainTests) ...

(HANGS here or some time fails like in: 

======================================================================
FAIL: test_can_interrupt_tight_loops (test.test_threading.InterruptMainTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/pull_request.cstratak-RHEL7-ppc64le.refleak/build/Lib/test/support/threading_helper.py", line 62, in decorator
    return func(*args)
           ^^^^^^^^^^^
  File "/home/buildbot/buildarea/pull_request.cstratak-RHEL7-ppc64le.refleak/build/Lib/test/test_threading.py", line 1630, in test_can_interrupt_tight_loops
    self.assertNotEqual(iterations, 0)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 0 == 0
----------------------------------------------------------------------

https://buildbot.python.org/all/#/builders/411/builds/92
msg397642 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-16 17:05
New changeset c90c591e5158ab7b531dcd6e2a5f00bc70ba7637 by Pablo Galindo Salgado in branch 'main':
Revert "bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)" (#27194)
https://github.com/python/cpython/commit/c90c591e5158ab7b531dcd6e2a5f00bc70ba7637
msg397643 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-16 17:30
New changeset 42a5514cca6b4542f6b492b5652e337f15a89227 by Miss Islington (bot) in branch '3.10':
Revert "bpo-44645: Check for interrupts on any potentially backwards edge. (GH-27167)" (GH-27194) (#27195)
https://github.com/python/cpython/commit/42a5514cca6b4542f6b492b5652e337f15a89227
msg397719 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-07-17 10:17
Can we at least include the fix, until we can come up with a better test?

That test fails on 3.9 as well as 3.10 and main.

With a 3.9 build and the test in #27194:

./python -m test test_threading -R :
0:00:00 load avg: 0.95 Run tests sequentially
0:00:00 load avg: 0.95 [1/1] test_threading
beginning 9 repetitions
123456789
...test test_threading failed -- Traceback (most recent call last):
  File "/home/mark/repos/cpython/Lib/test/test_threading.py", line 1479, in test_can_interrupt_tight_loops
    self.assertNotEqual(iterations, 0)
AssertionError: 0 == 0



I don't know what the underlying issue is.

Can you observe a failure without the -R flag?
msg397782 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-07-19 10:10
New changeset d09c13417890427f829e3df297beb0e27133f8f4 by Mark Shannon in branch 'main':
bpo-44645: Check for interrupts on any potentially backwards edge (GH-27216)
https://github.com/python/cpython/commit/d09c13417890427f829e3df297beb0e27133f8f4
msg397789 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-07-19 11:16
New changeset 37bdd2221ce3607a81d5d7fafc4603d95ca3e8cb by Miss Islington (bot) in branch '3.10':
bpo-44645: Check for interrupts on any potentially backwards edge (GH-27216) (GH-27235)
https://github.com/python/cpython/commit/37bdd2221ce3607a81d5d7fafc4603d95ca3e8cb
History
Date User Action Args
2021-07-31 23:21:29pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-07-19 11:16:14Mark.Shannonsetmessages: + msg397789
2021-07-19 10:10:49Mark.Shannonsetmessages: + msg397782
2021-07-19 10:10:33miss-islingtonsetpull_requests: + pull_request25783
2021-07-17 12:03:05Mark.Shannonsetpull_requests: + pull_request25755
2021-07-17 10:17:13Mark.Shannonsetmessages: + msg397719
2021-07-16 17:30:04lukasz.langasetmessages: + msg397643
2021-07-16 17:05:57miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request25730
2021-07-16 17:05:54lukasz.langasetnosy: + lukasz.langa
messages: + msg397642
2021-07-16 16:33:20pablogsalsetmessages: + msg397641
2021-07-16 16:32:19pablogsalsetmessages: - msg397640
2021-07-16 16:31:50pablogsalsetmessages: + msg397640
2021-07-16 16:29:40pablogsalsetstage: resolved -> patch review
pull_requests: + pull_request25729
2021-07-16 16:27:20pablogsalsetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg397639
2021-07-16 10:49:00pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-07-16 10:48:50pablogsalsetmessages: + msg397610
2021-07-16 10:05:54Mark.Shannonsetpull_requests: + pull_request25720
2021-07-16 09:59:35Mark.Shannonsetmessages: + msg397609
2021-07-15 15:42:10Mark.Shannonsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request25703
2021-07-15 14:12:33Mark.Shannonsetmessages: + msg397554
2021-07-15 13:27:01jceasetcomponents: + Interpreter Core
stage: needs patch
2021-07-15 13:26:07jceacreate