concurrent.futures race condition #89184

0x0L · 2021-08-26T20:09:17Z

BPO	45021
Nosy	@gpshead, @pitrou, @applio, @miss-islington, @shihai1991, @iritkatriel, @0x0L
PRs	bpo-45021: Fix a hang in forked children #28007 [3.10] bpo-45021: Fix a hang in forked children (GH-28007) #28480 [3.9] bpo-45021: Fix a hang in forked children (GH-28007) #28481

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2021-09-20.18:34:44.066>
created_at = <Date 2021-08-26.20:09:17.066>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'concurrent.futures race condition'
updated_at = <Date 2021-09-20.18:54:28.786>
user = 'https://github.com/0x0L'

bugs.python.org fields:

activity = <Date 2021-09-20.18:54:28.786>
actor = 'miss-islington'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2021-09-20.18:34:44.066>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2021-08-26.20:09:17.066>
creator = '0x0L'
dependencies = []
files = []
hgrepos = []
issue_num = 45021
keywords = ['patch']
message_count = 13.0
messages = ['400378', '400380', '400444', '400446', '400471', '400608', '400690', '400717', '400724', '402267', '402270', '402272', '402273']
nosy_count = 8.0
nosy_names = ['gregory.p.smith', 'pitrou', 'python-dev', 'davin', 'miss-islington', 'shihai1991', 'iritkatriel', '0x0L']
pr_nums = ['28007', '28480', '28481']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45021'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

0x0L · 2021-08-26T20:09:17Z

The following code can sometimes hang up

import random                                                         
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor    
from time import sleep    
                        
def worker():           
    with ProcessPoolExecutor() as pool:      
        r = list(pool.map(sleep, [0.01] * 8))       
                                                    
                                                   
if __name__ == '__main__':                         
    pool = ThreadPoolExecutor()     
                                   
    i = 0                          
    while True:                    
        if random.random() < 0.9:                    
            pool.submit(sleep, 0.001)      
        else:                             
            r = pool.submit(worker)         
            r = r.result()                
        i += 1                           
        print('alive', i)

It's a bit hard to trigger that way but with some luck and many restarts it'll eventually freeze as r.result() never returns.

The backtrace from a child process shows that the child is stuck in Lib/concurrent/futures/thread.py:_python_exit waiting for _global_shutdown_lock.

The fork happened while the lock was already grabbed i.e. while executing ThreadPoolExecutor.submit

0x0L · 2021-08-26T20:55:48Z

A more direct way to reproduce

from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from time import sleep

def worker():
    with ProcessPoolExecutor() as pool:
        r = list(pool.map(sleep, [0.01] * 8))


def submit(pool):
    pool.submit(submit, pool)


if __name__ == '__main__':
    pool = ThreadPoolExecutor(2)
    pool.submit(submit, pool)

    i = 0
    while True:
        r = pool.submit(worker)
        r = r.result()
        print(i)
        i += 1

iritkatriel · 2021-08-27T22:17:42Z

I'm seeing this (intermittently) on main branch, could be related?

iritkatriel@Irits-MBP cpython-1 % ./python.exe Lib/test/test_importlib/partial/pool_in_threads.py

Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 58, in _launch
    self.pid = util.spawnv_passfds(spawn.get_executable(),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/util.py", line 450, in spawnv_passfds
    errpipe_read, errpipe_write = os.pipe()
                                  ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 54, in _launch
    child_r, parent_w = os.pipe()
                        ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
OSError: [Errno 24] Too many open files
iritkatriel@Irits-MBP cpython-1 % /Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 59 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-2xx2cl25': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-8ru9f83a': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vn_5pkc4': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-2l68znmb': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-5onb3wo5': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-8yle2uvb': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-n0jgz2b_': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-18ikpx7f': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ikdrzqcc': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-1cecwdl7': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))

0x0L · 2021-08-27T23:31:50Z

I don't think so: this issue would only arise when using fork on linux and not spawn (win/osx) like it seems your system is doing

0x0L · 2021-08-28T14:22:16Z

Simplifying the reproducing example a bit more:

from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from time import sleep

def submit(pool):
    pool.submit(submit, pool)


if __name__ == '__main__':
    pool = ThreadPoolExecutor(1)
    pool.submit(submit, pool)

    while True:
        with ProcessPoolExecutor() as workers:
            print('WORK')
            workers.submit(sleep, 0.01).result()
            print('DONE')
        print('OK')

vstinner · 2021-08-30T14:08:23Z

I don't understand anything about multiprocessing, nor its API, but some people seem to find it useful :-D I remove myself from the nosy list ;-)

shihai1991 · 2021-08-31T05:04:01Z

Is it a defined behavior?
I got this sentence from pep-3148: Deadlock can occur when the callable associated with a Future waits on the results of another Future.

0x0L · 2021-08-31T12:04:11Z

In the last reproducing example, one can drop the .result() and just submit the task: we're not waiting on any result explicitly. It's the shutdown of the ProcessPoolExecutor that waits forever for the children to exit.

Generally speaking it's probably a bad idea to mix multi-thread and forks. See https://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them

pitrou · 2021-08-31T12:23:45Z

As the multiprocessing doc says (https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods):
"""Note that safely forking a multithreaded process is problematic."""

The reproducer is trivially fixed by adding a call to multiprocessing.set_start_method("forkserver") (you can also replace "forkserver" with "spawn")

gpshead · 2021-09-20T18:30:27Z

New changeset 0bfa110 by nullptr in branch 'main':
bpo-45021: Fix a hang in forked children (GH-28007)
0bfa110

gpshead · 2021-09-20T18:34:44Z

I expect the unittest for this might causes hangs on some platforms in the future as it mixes fork and threads which is a posix-nono. If so, we should just disable it on all but specific known-to-pass build configs as a future PR.

3.9 and 3.10 PRs will automerge.

miss-islington · 2021-09-20T18:52:32Z

New changeset d0d83a9 by Miss Islington (bot) in branch '3.10':
[3.10] bpo-45021: Fix a hang in forked children (GH-28007) (GH-28480)
d0d83a9

miss-islington · 2021-09-20T18:54:29Z

New changeset b06c3b3 by Miss Islington (bot) in branch '3.9':
[3.9] bpo-45021: Fix a hang in forked children (GH-28007) (GH-28481)
b06c3b3

0x0L mannequin added 3.9 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Aug 26, 2021

pitrou added 3.10 only security fixes 3.11 only security fixes labels Aug 31, 2021

vstinner changed the title ~~Race condition in thread.py~~ concurrent.futures race condition Sep 15, 2021

gpshead closed this as completed Sep 20, 2021

gpshead self-assigned this Sep 20, 2021

gpshead closed this as completed Sep 20, 2021

gpshead self-assigned this Sep 20, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

concurrent.futures race condition #89184

concurrent.futures race condition #89184

0x0L mannequin commented Aug 26, 2021

0x0L mannequin commented Aug 26, 2021

0x0L mannequin commented Aug 26, 2021

iritkatriel commented Aug 27, 2021

0x0L mannequin commented Aug 27, 2021

0x0L mannequin commented Aug 28, 2021

vstinner commented Aug 30, 2021

shihai1991 commented Aug 31, 2021

0x0L mannequin commented Aug 31, 2021

pitrou commented Aug 31, 2021

gpshead commented Sep 20, 2021

gpshead commented Sep 20, 2021

miss-islington commented Sep 20, 2021

miss-islington commented Sep 20, 2021

concurrent.futures race condition #89184

concurrent.futures race condition #89184

Comments

0x0L mannequin commented Aug 26, 2021

0x0L mannequin commented Aug 26, 2021

0x0L mannequin commented Aug 26, 2021

iritkatriel commented Aug 27, 2021

0x0L mannequin commented Aug 27, 2021

0x0L mannequin commented Aug 28, 2021

vstinner commented Aug 30, 2021

shihai1991 commented Aug 31, 2021

0x0L mannequin commented Aug 31, 2021

pitrou commented Aug 31, 2021

gpshead commented Sep 20, 2021

gpshead commented Sep 20, 2021

miss-islington commented Sep 20, 2021

miss-islington commented Sep 20, 2021