New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
multiprocessing.Pool from concurrent threads failure on 3.9.0rc1 #85739
Comments
If several threads try to start a multiprocessing.Pool at the same time when no pool has been started before this often fails with an exception like this (the exact import varies): Exception in thread Thread-2:
Traceback (most recent call last):
File "/tmp/py3.9.0rc1/lib/python3.9/threading.py", line 950, in _bootstrap_inner
self.run()
File "/tmp/py3.9.0rc1/lib/python3.9/threading.py", line 888, in run
self._target(*self._args, **self._kwargs)
File "/tmp/py3.9.0rc1/lib/python3.9/multiprocessing/context.py", line 118, in Pool
from .pool import Pool
ImportError: cannot import name 'Pool' from partially initialized module 'multiprocessing.pool' (most likely due to a circular import) (/tmp/py3.9.0rc1/lib/python3.9/multiprocessing/pool.py) This happens even if Pool was imported before starting the threads and is new in 3.9. It's easy to work around by starting a pool in the main thread before starting the other threads. I have attached a minimal example that triggers it. Tested on Debian stable and FreeBSD 11.3. |
I see the same in Python 3.10 on windows 10. If I change the relative imports to absolute imports in a couple of functions in multiprocessing.context as below, the attached (pool_error_on_3.9.py) script not longer raises the exception. def SimpleQueue(self):
'''Returns a queue object'''
from multiprocessing.queues import SimpleQueue
return SimpleQueue(ctx=self.get_context())
def Pool(self, processes=None, initializer=None, initargs=(),
maxtasksperchild=None):
'''Returns a process pool object'''
from multiprocessing.pool import Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
context=self.get_context()) |
I think this is no a bug, on the basis that multiprocessing.Pool is not thread-safe. |
multiprocessing module used to work multithread environment until https://bugs.python.org/issue35943 merged. I guess we can make multiprocessing thread-safe again if we move local imports to global. |
Some my observation at https://discuss.python.org/t/differences-between-3-8-and-3-9-in-importing-module/5520 . |
Example code (fails): import os, concurrent.futures
def parallel_callback( arg ):
return os.getpid()
def parallel( *args ):
def thread_callback( param ):
with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
future = executor.submit( parallel_callback, param )
pid = future.result()
print( 'pid:', pid )
return pid
with concurrent.futures.ThreadPoolExecutor(max_workers=len(args)) as executor:
future = executor.map( thread_callback, args )
results = list(future)
print( 'DONE' )
parallel( 1, 2, 3 ) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: