classification
Title: Creating a multiprocess.pool.ThreadPool from a child thread blows up.
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: multiprocessing.dummy craches when self._parent._children does not exist
View: 14881
Assigned To: Nosy List: Michael.Olson, acordiner, asksol, chris-, flox, jnoller, sbt, terry.reedy
Priority: normal Keywords:

Created on 2010-10-02 14:00 by Michael.Olson, last changed 2013-10-19 00:33 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
potential_issue_demo.py Michael.Olson, 2010-10-02 14:00 Demonstration code
Messages (5)
msg117875 - (view) Author: Michael Olson (Michael.Olson) Date: 2010-10-02 14:00
Using Python 2.7 x32 on Windows XP

Attempting to create a multiprocessing.pool.ThreadPool
in a child thread created using threading.Thread, an
AttributeError is thrown. A ThreadPool created in the 
main thread can be passed to the child thread and used.


Exact text of exception
-------------------
  File "D:\Dev\Python27\lib\multiprocessing\dummy\__init__.py", line 47, in star
t
    self._parent._children[self] = None
AttributeError: 'Thread' object has no attribute '_children'


Demonstration Code
-------------------
import unittest
from threading import Thread
from multiprocessing.pool import ThreadPool


def f(x):
    return x*x


def create_and_run(cb, pool = None):
    if not pool:
        pool = ThreadPool(2)
    r = pool.map_async(f, range(10))
    cb(r.get())


class TestThreadPool(unittest.TestCase):
    def setUp(self):
        self.expected = [f(x) for x in range(10)]

    def callback(self, data):
        self.data = data

    def test_creating_pool_in_mainthread(self):
        """Test multiprocessing.pool.ThreadPool from main thread"""
        self.data = None
        create_and_run(self.callback)
        self.assertEqual(self.data, self.expected)

    def test_creating_pool_in_subthread(self):
        """Test multiprocessing.pool.ThreadPool from a child thread."""
        self.data = None
        t = Thread(target=create_and_run, args=[self.callback])
        t.start()
        t.join()
        self.assertEqual(self.data, self.expected)

    def test_creating_pool_in_subthread_workaround(self):
        """Test running a ThreadPool created in main thread, used in child."""
        self.data = None
        pool = ThreadPool(2)
        t = Thread(target=create_and_run, args=[self.callback, pool])
        t.start()
        t.join()
        self.assertEqual(self.data, self.expected)


if __name__ =='__main__':
    suite = unittest.TestLoader().loadTestsFromTestCase(TestThreadPool)
    unittest.TextTestRunner(verbosity=2).run(suite)
msg117890 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2010-10-02 17:15
I can not, for the life of me, remember why ThreadPool is there, except as a fallback. It's also not part of the documented interface as well. Additionally, in Python 3 we now have futures.
msg142294 - (view) Author: Christian (chris-) Date: 2011-08-17 22:42
A workaround would be to call the following in the thread you want to use ThreadPool:

if not hasattr(threading.current_process(), "_children"):
    threading.current_process()._children = weakref.WeakKeyDictionary()

(putting this in Process could be a very simple patch)
msg146473 - (view) Author: Alister Cordiner (acordiner) Date: 2011-10-27 05:47
I think that workaround should be:

if not hasattr(threading.current_thread(), "_children"):
    threading.current_thread()._children = weakref.WeakKeyDictionary()
msg200339 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-10-19 00:33
#14881 seems to be a duplicate of this. It was closed in May 2012 after 2.7, 3.2, and 3.3 were patched. The three tests in potential_issue_demo.py now pass with 2.7.5 and 3.3.2. So closing.
History
Date User Action Args
2013-10-19 00:33:30terry.reedysetstatus: open -> closed

superseder: multiprocessing.dummy craches when self._parent._children does not exist

nosy: + terry.reedy
messages: + msg200339
resolution: duplicate
stage: resolved
2013-10-18 20:33:00sbtsetnosy: + sbt
2011-10-27 09:45:33floxsetnosy: + flox
2011-10-27 05:47:33acordinersetnosy: + acordiner
messages: + msg146473
2011-08-27 14:58:59vinay.sajipsettitle: Creating a multiproccess.pool.ThreadPool from a child thread blows up. -> Creating a multiprocess.pool.ThreadPool from a child thread blows up.
2011-08-17 22:42:10chris-setnosy: + chris-
messages: + msg142294
2010-10-02 17:15:15jnollersetmessages: + msg117890
2010-10-02 14:49:08r.david.murraysetnosy: + jnoller, asksol
type: crash -> behavior
2010-10-02 14:00:28Michael.Olsoncreate