The following code can sometimes hang up


import random                                                         
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor    
from time import sleep    
                        
def worker():           
    with ProcessPoolExecutor() as pool:      
        r = list(pool.map(sleep, [0.01] * 8))       
                                                    
                                                   
if __name__ == '__main__':                         
    pool = ThreadPoolExecutor()     
                                   
    i = 0                          
    while True:                    
        if random.random() < 0.9:                    
            pool.submit(sleep, 0.001)      
        else:                             
            r = pool.submit(worker)         
            r = r.result()                
        i += 1                           
        print('alive', i)


It's a bit hard to trigger that way but with some luck and many restarts it'll eventually freeze as r.result() never returns.

The backtrace from a child process shows that the child is stuck in Lib/concurrent/futures/thread.py:_python_exit waiting for _global_shutdown_lock.

The fork happened while the lock was already grabbed i.e. while executing ThreadPoolExecutor.submit

A more direct way to reproduce


from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from time import sleep

def worker():
    with ProcessPoolExecutor() as pool:
        r = list(pool.map(sleep, [0.01] * 8))


def submit(pool):
    pool.submit(submit, pool)


if __name__ == '__main__':
    pool = ThreadPoolExecutor(2)
    pool.submit(submit, pool)

    i = 0
    while True:
        r = pool.submit(worker)
        r = r.result()
        print(i)
        i += 1

I'm seeing this (intermittently) on main branch, could be related?

iritkatriel@Irits-MBP cpython-1 % ./python.exe Lib/test/test_importlib/partial/pool_in_threads.py

Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 58, in _launch
    self.pid = util.spawnv_passfds(spawn.get_executable(),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/util.py", line 450, in spawnv_passfds
    errpipe_read, errpipe_write = os.pipe()
                                  ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/popen_spawn_posix.py", line 54, in _launch
    child_r, parent_w = os.pipe()
                        ^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
OSError: [Errno 24] Too many open files
iritkatriel@Irits-MBP cpython-1 % /Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 59 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-2xx2cl25': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-8ru9f83a': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vn_5pkc4': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-2l68znmb': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-5onb3wo5': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-8yle2uvb': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-n0jgz2b_': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-18ikpx7f': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ikdrzqcc': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-1cecwdl7': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))

I don't think so: this issue would only arise when using fork on linux and not spawn (win/osx) like it seems your system is doing

Simplifying the reproducing example a bit more:


from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from time import sleep

def submit(pool):
    pool.submit(submit, pool)


if __name__ == '__main__':
    pool = ThreadPoolExecutor(1)
    pool.submit(submit, pool)

    while True:
        with ProcessPoolExecutor() as workers:
            print('WORK')
            workers.submit(sleep, 0.01).result()
            print('DONE')
        print('OK')

I don't understand anything about multiprocessing, nor its API, but some people seem to find it useful :-D I remove myself from the nosy list ;-)

Is it a defined behavior?
I got this sentence from pep-3148: Deadlock can occur when the callable associated with a Future waits on the results of another Future.

In the last reproducing example, one can drop the .result() and just submit the task: we're not waiting on any result explicitly. It's the shutdown of the ProcessPoolExecutor that waits forever for the children to exit.

Generally speaking it's probably a bad idea to mix multi-thread and forks. See https://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them

As the multiprocessing doc says (https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods):
"""Note that safely forking a multithreaded process is problematic."""

The reproducer is trivially fixed by adding a call to `multiprocessing.set_start_method("forkserver")` (you can also replace "forkserver" with "spawn")

New changeset 0bfa1106acfcddc03590e1f5d6789dbad3affe70 by nullptr in branch 'main':
bpo-45021: Fix a hang in forked children (GH-28007)
https://github.com/python/cpython/commit/0bfa1106acfcddc03590e1f5d6789dbad3affe70

I expect the unittest for this might causes hangs on some platforms in the future as it mixes fork and threads which is a posix-nono.  If so, we should just disable it on all but specific known-to-pass build configs as a future PR.

3.9 and 3.10 PRs will automerge.

New changeset d0d83a94314402366e04e7ea2638f809510eb830 by Miss Islington (bot) in branch '3.10':
[3.10] bpo-45021: Fix a hang in forked children (GH-28007) (GH-28480)
https://github.com/python/cpython/commit/d0d83a94314402366e04e7ea2638f809510eb830

New changeset b06c3b364720ce8e8dfb74dfa24434e067ac89ba by Miss Islington (bot) in branch '3.9':
[3.9] bpo-45021: Fix a hang in forked children (GH-28007) (GH-28481)
https://github.com/python/cpython/commit/b06c3b364720ce8e8dfb74dfa24434e067ac89ba