classification
Title: concurrent.futures race condition
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: 0x0L, davin, gregory.p.smith, iritkatriel, miss-islington, pitrou, python-dev, shihai1991
Priority: normal Keywords: patch

Created on 2021-08-26 20:09 by 0x0L, last changed 2021-09-20 18:54 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 28007 merged python-dev, 2021-08-27 21:14
PR 28480 merged miss-islington, 2021-09-20 18:30
PR 28481 merged miss-islington, 2021-09-20 18:30
Messages (13)
msg400378 - (view) Author: 0x0L (0x0L) * Date: 2021-08-26 20:09
The following code can sometimes hang up


import random                                                         
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor    
from time import sleep    
                        
def worker():           
    with ProcessPoolExecutor() as pool:      
        r = list(pool.map(sleep, [0.01] * 8))       
                                                    
                                                   
if __name__ == '__main__':                         
    pool = ThreadPoolExecutor()     
                                   
    i = 0                          
    while True:                    
        if random.random() < 0.9:                    
            pool.submit(sleep, 0.001)      
        else:                             
            r = pool.submit(worker)         
            r = r.result()                
        i += 1                           
        print('alive', i)


It's a bit hard to trigger that way but with some luck and many restarts it'll eventually freeze as r.result() never returns.

The backtrace from a child process shows that the child is stuck in Lib/concurrent/futures/thread.py:_python_exit waiting for _global_shutdown_lock.

The fork happened while the lock was already grabbed i.e. while executing ThreadPoolExecutor.submit
msg400380 - (view) Author: 0x0L (0x0L) * Date: 2021-08-26 20:55
A more direct way to reproduce


from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from time import sleep

def worker():
    with ProcessPoolExecutor() as pool:
        r = list(pool.map(sleep, [0.01] * 8))


def submit(pool):
    pool.submit(submit, pool)


if __name__ == '__main__':
    pool = ThreadPoolExecutor(2)
    pool.submit(submit, pool)

    i = 0
    while True:
        r = pool.submit(worker)
        r = r.result()
        print(i)
        i += 1
msg400444 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-08-27 22:17
I'm seeing this (intermittently) on main branch, could be related?

iritkatriel@Irits-MBP cpython-1 % ./python.exe Lib/test/test_importlib/partial/pool_in_threads.py

Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 58, in _launch
    self.pid = util.spawnv_passfds(spawn.get_executable(),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/util.py", line 450, in spawnv_passfds
    errpipe_read, errpipe_write = os.pipe()
                                  ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 54, in _launch
    child_r, parent_w = os.pipe()
                        ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
OSError: [Errno 24] Too many open files
iritkatriel@Irits-MBP cpython-1 % /Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 59 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-2xx2cl25': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-8ru9f83a': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vn_5pkc4': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-2l68znmb': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-5onb3wo5': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-8yle2uvb': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-n0jgz2b_': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-18ikpx7f': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ikdrzqcc': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-1cecwdl7': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
msg400446 - (view) Author: 0x0L (0x0L) * Date: 2021-08-27 23:31
I don't think so: this issue would only arise when using fork on linux and not spawn (win/osx) like it seems your system is doing
msg400471 - (view) Author: 0x0L (0x0L) * Date: 2021-08-28 14:22
Simplifying the reproducing example a bit more:


from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from time import sleep

def submit(pool):
    pool.submit(submit, pool)


if __name__ == '__main__':
    pool = ThreadPoolExecutor(1)
    pool.submit(submit, pool)

    while True:
        with ProcessPoolExecutor() as workers:
            print('WORK')
            workers.submit(sleep, 0.01).result()
            print('DONE')
        print('OK')
msg400608 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-08-30 14:08
I don't understand anything about multiprocessing, nor its API, but some people seem to find it useful :-D I remove myself from the nosy list ;-)
msg400690 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2021-08-31 05:04
Is it a defined behavior?
I got this sentence from pep-3148: Deadlock can occur when the callable associated with a Future waits on the results of another Future.
msg400717 - (view) Author: 0x0L (0x0L) * Date: 2021-08-31 12:04
In the last reproducing example, one can drop the .result() and just submit the task: we're not waiting on any result explicitly. It's the shutdown of the ProcessPoolExecutor that waits forever for the children to exit.

Generally speaking it's probably a bad idea to mix multi-thread and forks. See https://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them
msg400724 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2021-08-31 12:23
As the multiprocessing doc says (https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods):
"""Note that safely forking a multithreaded process is problematic."""

The reproducer is trivially fixed by adding a call to `multiprocessing.set_start_method("forkserver")` (you can also replace "forkserver" with "spawn")
msg402267 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-09-20 18:30
New changeset 0bfa1106acfcddc03590e1f5d6789dbad3affe70 by nullptr in branch 'main':
bpo-45021: Fix a hang in forked children (GH-28007)
https://github.com/python/cpython/commit/0bfa1106acfcddc03590e1f5d6789dbad3affe70
msg402270 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-09-20 18:34
I expect the unittest for this might causes hangs on some platforms in the future as it mixes fork and threads which is a posix-nono.  If so, we should just disable it on all but specific known-to-pass build configs as a future PR.

3.9 and 3.10 PRs will automerge.
msg402272 - (view) Author: miss-islington (miss-islington) Date: 2021-09-20 18:52
New changeset d0d83a94314402366e04e7ea2638f809510eb830 by Miss Islington (bot) in branch '3.10':
[3.10] bpo-45021: Fix a hang in forked children (GH-28007) (GH-28480)
https://github.com/python/cpython/commit/d0d83a94314402366e04e7ea2638f809510eb830
msg402273 - (view) Author: miss-islington (miss-islington) Date: 2021-09-20 18:54
New changeset b06c3b364720ce8e8dfb74dfa24434e067ac89ba by Miss Islington (bot) in branch '3.9':
[3.9] bpo-45021: Fix a hang in forked children (GH-28007) (GH-28481)
https://github.com/python/cpython/commit/b06c3b364720ce8e8dfb74dfa24434e067ac89ba
History
Date User Action Args
2021-09-20 18:54:28miss-islingtonsetmessages: + msg402273
2021-09-20 18:52:31miss-islingtonsetmessages: + msg402272
2021-09-20 18:34:44gregory.p.smithsetstatus: open -> closed
messages: + msg402270

assignee: gregory.p.smith
resolution: fixed
stage: patch review -> resolved
2021-09-20 18:30:33miss-islingtonsetpull_requests: + pull_request26881
2021-09-20 18:30:29miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26880
2021-09-20 18:30:26gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg402267
2021-09-15 13:25:33vstinnersettitle: Race condition in thread.py -> concurrent.futures race condition
2021-08-31 12:23:50pitrousetversions: + Python 3.10, Python 3.11
2021-08-31 12:23:44pitrousetmessages: + msg400724
2021-08-31 12:04:100x0Lsetmessages: + msg400717
2021-08-31 05:04:01shihai1991setnosy: + shihai1991
messages: + msg400690
2021-08-30 14:14:20vstinnersetnosy: - vstinner
2021-08-30 14:08:22vstinnersetnosy: pitrou, vstinner, python-dev, davin, iritkatriel, 0x0L
messages: + msg400608
2021-08-30 02:05:14rhettingersetnosy: + davin
2021-08-28 14:22:160x0Lsetmessages: + msg400471
2021-08-27 23:31:490x0Lsetmessages: + msg400446
2021-08-27 22:17:41iritkatrielsetnosy: + iritkatriel
messages: + msg400444
2021-08-27 22:13:55iritkatrielsetnosy: + pitrou, vstinner
2021-08-27 21:14:30python-devsetkeywords: + patch
nosy: + python-dev

pull_requests: + pull_request26449
stage: patch review
2021-08-26 20:55:480x0Lsetmessages: + msg400380
2021-08-26 20:14:460x0Lsettype: behavior
2021-08-26 20:09:170x0Lcreate