Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating a multiprocess.pool.ThreadPool from a child thread blows up. #54224

Closed
MichaelOlson mannequin opened this issue Oct 2, 2010 · 5 comments
Closed

Creating a multiprocess.pool.ThreadPool from a child thread blows up. #54224

MichaelOlson mannequin opened this issue Oct 2, 2010 · 5 comments
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@MichaelOlson
Copy link
Mannequin

MichaelOlson mannequin commented Oct 2, 2010

BPO 10015
Nosy @terryjreedy, @florentx
Superseder
  • bpo-14881: multiprocessing.dummy craches when self._parent._children does not exist
  • Files
  • potential_issue_demo.py: Demonstration code
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2013-10-19.00:33:30.694>
    created_at = <Date 2010-10-02.14:00:28.332>
    labels = ['type-bug', 'library']
    title = 'Creating a multiprocess.pool.ThreadPool from a child thread blows up.'
    updated_at = <Date 2013-10-19.00:33:30.573>
    user = 'https://bugs.python.org/MichaelOlson'

    bugs.python.org fields:

    activity = <Date 2013-10-19.00:33:30.573>
    actor = 'terry.reedy'
    assignee = 'none'
    closed = True
    closed_date = <Date 2013-10-19.00:33:30.694>
    closer = 'terry.reedy'
    components = ['Library (Lib)']
    creation = <Date 2010-10-02.14:00:28.332>
    creator = 'Michael.Olson'
    dependencies = []
    files = ['19105']
    hgrepos = []
    issue_num = 10015
    keywords = []
    message_count = 5.0
    messages = ['117875', '117890', '142294', '146473', '200339']
    nosy_count = 8.0
    nosy_names = ['terry.reedy', 'jnoller', 'asksol', 'flox', 'Michael.Olson', 'sbt', 'chris-', 'acordiner']
    pr_nums = []
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '14881'
    type = 'behavior'
    url = 'https://bugs.python.org/issue10015'
    versions = ['Python 2.7']

    @MichaelOlson
    Copy link
    Mannequin Author

    MichaelOlson mannequin commented Oct 2, 2010

    Using Python 2.7 x32 on Windows XP

    Attempting to create a multiprocessing.pool.ThreadPool
    in a child thread created using threading.Thread, an
    AttributeError is thrown. A ThreadPool created in the
    main thread can be passed to the child thread and used.

    Exact text of exception
    -------------------
    File "D:\Dev\Python27\lib\multiprocessing\dummy\init.py", line 47, in star
    t
    self._parent._children[self] = None
    AttributeError: 'Thread' object has no attribute '_children'

    Demonstration Code
    -------------------

    import unittest
    from threading import Thread
    from multiprocessing.pool import ThreadPool
    
    
    def f(x):
        return x*x
    
    
    def create_and_run(cb, pool = None):
        if not pool:
            pool = ThreadPool(2)
        r = pool.map_async(f, range(10))
        cb(r.get())
    
    
    class TestThreadPool(unittest.TestCase):
        def setUp(self):
            self.expected = [f(x) for x in range(10)]
    
        def callback(self, data):
            self.data = data
    
        def test_creating_pool_in_mainthread(self):
            """Test multiprocessing.pool.ThreadPool from main thread"""
            self.data = None
            create_and_run(self.callback)
            self.assertEqual(self.data, self.expected)
    
        def test_creating_pool_in_subthread(self):
            """Test multiprocessing.pool.ThreadPool from a child thread."""
            self.data = None
            t = Thread(target=create_and_run, args=[self.callback])
            t.start()
            t.join()
            self.assertEqual(self.data, self.expected)
    
        def test_creating_pool_in_subthread_workaround(self):
            """Test running a ThreadPool created in main thread, used in child."""
            self.data = None
            pool = ThreadPool(2)
            t = Thread(target=create_and_run, args=[self.callback, pool])
            t.start()
            t.join()
            self.assertEqual(self.data, self.expected)
    
    
    if __name__ =='__main__':
        suite = unittest.TestLoader().loadTestsFromTestCase(TestThreadPool)
        unittest.TextTestRunner(verbosity=2).run(suite)

    @MichaelOlson MichaelOlson mannequin added type-crash A hard crash of the interpreter, possibly with a core dump stdlib Python modules in the Lib dir labels Oct 2, 2010
    @bitdancer bitdancer added type-bug An unexpected behavior, bug, or error and removed type-crash A hard crash of the interpreter, possibly with a core dump labels Oct 2, 2010
    @jnoller
    Copy link
    Mannequin

    jnoller mannequin commented Oct 2, 2010

    I can not, for the life of me, remember why ThreadPool is there, except as a fallback. It's also not part of the documented interface as well. Additionally, in Python 3 we now have futures.

    @chris
    Copy link
    Mannequin

    chris mannequin commented Aug 17, 2011

    A workaround would be to call the following in the thread you want to use ThreadPool:

    if not hasattr(threading.current_process(), "_children"):
        threading.current_process()._children = weakref.WeakKeyDictionary()

    (putting this in Process could be a very simple patch)

    @vsajip vsajip changed the title Creating a multiproccess.pool.ThreadPool from a child thread blows up. Creating a multiprocess.pool.ThreadPool from a child thread blows up. Aug 27, 2011
    @acordiner
    Copy link
    Mannequin

    acordiner mannequin commented Oct 27, 2011

    I think that workaround should be:

    if not hasattr(threading.current_thread(), "_children"):
        threading.current_thread()._children = weakref.WeakKeyDictionary()

    @terryjreedy
    Copy link
    Member

    bpo-14881 seems to be a duplicate of this. It was closed in May 2012 after 2.7, 3.2, and 3.3 were patched. The three tests in potential_issue_demo.py now pass with 2.7.5 and 3.3.2. So closing.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants