Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiprocessing hangs when Pool used within Process #49581

Closed
zzzeek mannequin opened this issue Feb 20, 2009 · 6 comments
Closed

multiprocessing hangs when Pool used within Process #49581

zzzeek mannequin opened this issue Feb 20, 2009 · 6 comments
Labels
stdlib Python modules in the Lib dir type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@zzzeek
Copy link
Mannequin

zzzeek mannequin commented Feb 20, 2009

BPO 5331

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-06-30.17:14:56.090>
created_at = <Date 2009-02-20.16:06:50.657>
labels = ['library', 'type-crash']
title = 'multiprocessing hangs when Pool used within Process'
updated_at = <Date 2009-06-30.17:14:56.088>
user = 'https://bugs.python.org/zzzeek'

bugs.python.org fields:

activity = <Date 2009-06-30.17:14:56.088>
actor = 'jnoller'
assignee = 'jnoller'
closed = True
closed_date = <Date 2009-06-30.17:14:56.090>
closer = 'jnoller'
components = ['Library (Lib)']
creation = <Date 2009-02-20.16:06:50.657>
creator = 'zzzeek'
dependencies = []
files = []
hgrepos = []
issue_num = 5331
keywords = []
message_count = 6.0
messages = ['82534', '82540', '89192', '89194', '89901', '89945']
nosy_count = 4.0
nosy_names = ['OG7', 'LambertDW', 'jnoller', 'zzzeek']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue5331'
versions = ['Python 2.6', 'Python 3.0']

@zzzeek
Copy link
Mannequin Author

zzzeek mannequin commented Feb 20, 2009

this occurs for me running on Mac OSX Leopard. The equivalent code
using "processing" in python 2.5 works fine, which is how I found this
bug - my code hung when upgraded to 2.6. Basically initiating a
multiprocessing.Pool inside of multiprocessing.Process hangs the
application. Below is code which illustrates the issue on both py2.6
and py3.0:

    from multiprocessing import Pool, Process
    import sys

    def f(x):
        return x*x

    def go():
        pool = Pool(processes=4)              
        x = pool.map(f, [1, 2, 3, 4, 5, 6, 7])
        sys.stdout.write("result: %s\n" % x)

    def go2():
        x = map(f, [1,2,3,4,5,6,7])
        sys.stdout.write("result: %s\n" % x)

    # pool alone, fine
    go()

    # process alone, fine
    p = Process(target=go2)
    p.start()
    p.join()

    # use both, hangs
    p = Process(target=go)
    p.start()
    p.join()

@zzzeek zzzeek mannequin added stdlib Python modules in the Lib dir type-crash A hard crash of the interpreter, possibly with a core dump labels Feb 20, 2009
@lambertdw
Copy link
Mannequin

lambertdw mannequin commented Feb 20, 2009

Fails also on this system.

$ p3
Python 3.0.1 (r301:69556, Feb 13 2009, 23:52:55) 
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
(...)
>>> # use both hangs
(...)
>>> p.start()
>>> p.join()
(hmm la  do diii   laaa...)^C
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/process.py",
line 119, in join
    res = self._popen.wait(timeout)
  File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",
line 118, in wait
    return self.poll(0)
  File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",
line 107, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
>>>

@OG7
Copy link
Mannequin

OG7 mannequin commented Jun 10, 2009

Apparently the pool workers die all at once, just after the pool creates
them and before the pool is used.
I added a few lines to multiprocessing/pool.py to get the stack and the
exception backtrace.

        except (EOFError, IOError):
            import traceback
            debug(traceback.format_exc())
            debug(''.join(traceback.format_stack()))
            debug('worker got EOFError or IOError -- exiting')
            break
INFO::Rule dispatcher::multiprocessing::child process calling self.run()
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3082559488
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3082104832
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3081826304
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3081822208
INFO::PoolWorker-3:1::multiprocessing::child process calling self.run()
DEBUG::PoolWorker-3:1::multiprocessing::Traceback (most recent call last):
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 57, in worker
    task = get()
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/queues.py",
line 339, in get
    return recv()
IOError: [Errno 9] Bad file descriptor

DEBUG::PoolWorker-3:1::multiprocessing:: File
"/home/who/var/co/git-svn/what-base/correlator/bin/correlator", line 30,
in <module>
what.corr.actors.main.main()
File
"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/main.py",
line 47, in main
rrp.start()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 109, in start
self._popen = Popen(self)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",
line 99, in __init__
code = process_obj._bootstrap()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 93, in run
self._target(*self._args, **self._kwargs)
File
"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/rule_dispatcher.py",
line 26, in main
initargs=(manager.agg_msgs_queue, ))
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/init.py",
line 232, in Pool
return Pool(processes, initializer, initargs)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 107, in __init__
w.start()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 109, in start
self._popen = Popen(self)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",
line 99, in __init__
code = process_obj._bootstrap()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 93, in run
self._target(*self._args, **self._kwargs)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 61, in worker
debug(''.join(traceback.format_stack()))

DEBUG::PoolWorker-3:1::multiprocessing::worker got EOFError or IOError
-- exiting
INFO::PoolWorker-3:1::multiprocessing::process shutting down
DEBUG::PoolWorker-3:1::multiprocessing::running all "atexit" finalizers
with priority >= 0

@OG7
Copy link
Mannequin

OG7 mannequin commented Jun 10, 2009

It seems the root cause is at http://bugs.python.org/issue5155 .
A workaround is to use a duplex Pipe in SimpleQueue.

@jnoller
Copy link
Mannequin

jnoller mannequin commented Jun 30, 2009

Patch attached to bpo-5313, please review

@jnoller
Copy link
Mannequin

jnoller mannequin commented Jun 30, 2009

Committed in r73708 on trunk

@jnoller jnoller mannequin closed this as completed Jun 30, 2009
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stdlib Python modules in the Lib dir type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Development

No branches or pull requests

0 participants