This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: multiprocessing.Pool from concurrent threads failure on 3.9.0rc1
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.10, Python 3.9
Status: closed Resolution: duplicate
Dependencies: Superseder: Fix false positives in circular import detection with from-imports
View: 43517
Assigned To: Nosy List: aeros, cebtenzzre, doublex, drougge, iritkatriel, ishimoto
Priority: normal Keywords:

Created on 2020-08-17 11:37 by drougge, last changed 2022-04-11 14:59 by admin. This issue is now closed.

File name Uploaded Description Edit drougge, 2020-08-17 11:37 repro example
Messages (6)
msg375542 - (view) Author: Carl Drougge (drougge) * Date: 2020-08-17 11:37
If several threads try to start a multiprocessing.Pool at the same time when no pool has been started before this often fails with an exception like this (the exact import varies):

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/tmp/py3.9.0rc1/lib/python3.9/", line 950, in _bootstrap_inner
  File "/tmp/py3.9.0rc1/lib/python3.9/", line 888, in run
    self._target(*self._args, **self._kwargs)
  File "/tmp/py3.9.0rc1/lib/python3.9/multiprocessing/", line 118, in Pool
    from .pool import Pool
ImportError: cannot import name 'Pool' from partially initialized module 'multiprocessing.pool' (most likely due to a circular import) (/tmp/py3.9.0rc1/lib/python3.9/multiprocessing/

This happens even if Pool was imported before starting the threads and is new in 3.9. It's easy to work around by starting a pool in the main thread before starting the other threads.

I have attached a minimal example that triggers it. Tested on Debian stable and FreeBSD 11.3.
msg375544 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-08-17 13:51
I see the same in Python 3.10 on windows 10.

If I change the relative imports to absolute imports in a couple of functions in multiprocessing.context as below, the attached ( script not longer raises the exception.

    def SimpleQueue(self):
        '''Returns a queue object'''
        from multiprocessing.queues import SimpleQueue
        return SimpleQueue(ctx=self.get_context())

    def Pool(self, processes=None, initializer=None, initargs=(),
        '''Returns a process pool object'''
        from multiprocessing.pool import Pool
        return Pool(processes, initializer, initargs, maxtasksperchild,
msg375556 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-08-17 16:07
I think this is no a bug, on the basis that multiprocessing.Pool is not thread-safe.
msg379400 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2020-10-23 03:39
multiprocessing module used to work multithread environment until merged.

I guess we can make multiprocessing thread-safe again if we move local imports to global.
Does it make sense?
msg379401 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2020-10-23 03:41
Some my observation at .
msg388771 - (view) Author: (doublex) Date: 2021-03-15 19:52
Example code (fails):

import os, concurrent.futures

def parallel_callback( arg ):
    return os.getpid()

def parallel( *args ):
    def thread_callback( param ):
        with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
            future = executor.submit( parallel_callback, param )
            pid = future.result()
            print( 'pid:', pid )
            return pid
    with concurrent.futures.ThreadPoolExecutor(max_workers=len(args)) as executor:
        future = thread_callback, args )
        results = list(future)
    print( 'DONE' )

parallel( 1, 2, 3 )
Date User Action Args
2022-04-11 14:59:34adminsetgithub: 85739
2021-03-16 16:16:54pitrousetstatus: open -> closed
superseder: Fix false positives in circular import detection with from-imports
resolution: duplicate
stage: resolved
2021-03-15 19:52:58doublexsetnosy: + doublex
messages: + msg388771
2021-02-17 18:22:51pitrousetnosy: + aeros
2020-12-28 22:49:20cebtenzzresetnosy: + cebtenzzre
2020-10-23 03:41:28ishimotosetmessages: + msg379401
2020-10-23 03:39:01ishimotosetnosy: + ishimoto
messages: + msg379400
2020-08-17 16:07:35iritkatrielsetmessages: + msg375556
2020-08-17 13:51:14iritkatrielsetnosy: + iritkatriel

messages: + msg375544
versions: + Python 3.10
2020-08-17 11:37:30drouggecreate