New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Creating a multiprocess.pool.ThreadPool from a child thread blows up. #54224
Comments
Using Python 2.7 x32 on Windows XP Attempting to create a multiprocessing.pool.ThreadPool Exact text of exception Demonstration Code import unittest
from threading import Thread
from multiprocessing.pool import ThreadPool
def f(x):
return x*x
def create_and_run(cb, pool = None):
if not pool:
pool = ThreadPool(2)
r = pool.map_async(f, range(10))
cb(r.get())
class TestThreadPool(unittest.TestCase):
def setUp(self):
self.expected = [f(x) for x in range(10)]
def callback(self, data):
self.data = data
def test_creating_pool_in_mainthread(self):
"""Test multiprocessing.pool.ThreadPool from main thread"""
self.data = None
create_and_run(self.callback)
self.assertEqual(self.data, self.expected)
def test_creating_pool_in_subthread(self):
"""Test multiprocessing.pool.ThreadPool from a child thread."""
self.data = None
t = Thread(target=create_and_run, args=[self.callback])
t.start()
t.join()
self.assertEqual(self.data, self.expected)
def test_creating_pool_in_subthread_workaround(self):
"""Test running a ThreadPool created in main thread, used in child."""
self.data = None
pool = ThreadPool(2)
t = Thread(target=create_and_run, args=[self.callback, pool])
t.start()
t.join()
self.assertEqual(self.data, self.expected)
if __name__ =='__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestThreadPool)
unittest.TextTestRunner(verbosity=2).run(suite) |
I can not, for the life of me, remember why ThreadPool is there, except as a fallback. It's also not part of the documented interface as well. Additionally, in Python 3 we now have futures. |
A workaround would be to call the following in the thread you want to use ThreadPool: if not hasattr(threading.current_process(), "_children"):
threading.current_process()._children = weakref.WeakKeyDictionary() (putting this in Process could be a very simple patch) |
I think that workaround should be: if not hasattr(threading.current_thread(), "_children"):
threading.current_thread()._children = weakref.WeakKeyDictionary() |
bpo-14881 seems to be a duplicate of this. It was closed in May 2012 after 2.7, 3.2, and 3.3 were patched. The three tests in potential_issue_demo.py now pass with 2.7.5 and 3.3.2. So closing. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: