New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
join method of multiprocessing Pool object hangs if iterable argument of pool.map is empty #56366
Comments
When I use map method Pool object with an empty list parameter and then call close and wait methods, join() method hangs. I think this is not intended. Code to reproduce the bug is attached. PS: A similar issue (using map method with an empty list argument) is reported here[1], but it was about the chunksize parameter and it's resolved. |
I ran with 3.2, winxp with "if __name__ == '__main__':" added after the def statement (without this, process spawned 150 processes before I got logged out) and ()s added to prints. Hung on pool.join as OP said. I could only stop by closing command window as ^C was ignored. Any new test should have a timeout ;-). |
When map is called, a MapResult object is created, which adds itself to the Pool's result cache. |
Hello, This is my first patch to cpython, hope it will be accepted :) The fix that i did is to remove the ResultMap instance from the pool cache when the iterable is empty. In general here is what happen: The "map" method create a MapResult instance, which add it self automatically to the pool._cache and this ResultMap instance will be used by the task that will be created and added after in the "pool._taskqueue" to communicate the task result, but in case of an empty iterable the task will not be created and we will end up with a MapResult with no task and when we will try to join the pool, it will hang waiting for the task to set the result in the MapResult instance. For the test i created a new helper |
The test case use a helper function in test/support.py that i have proposed in issue bpo-12410. I'm dropping this comment here because i don't have the rights to edit the issue dependency. cheers; |
Here is a new patch that in the opposite of the first one, it don't try to check if the pool.join() will hang or no, after a discussion with neologix in issue bpo-12410 . |
The patch to the multiprocessing code is trivial: The difference in tests is Victor, do you agree with the simpler method, depending on faulthandler to catch a hang in the test and fail it? Or is the explicit timeout better? |
If the patch fixes the hang, there is no good reason to write code to handle a new hang. We have generic "watchdogs":
If you run directly the .py test file on a command line, you can still use CTRL+c or CTRL+z to interrupt / stop the process. You may want to improve these generic watchdogs, but write a specific watchdog for one specific test function looks useless to me. Remember that timeouts are not reliable: we have sometimes false failures because of very slow buildbots... For regrtest timeout, I tried 10, 15, 20 and 30 minutes before choosing a timeout of 60 minutes. For lower values, we have many false failures. |
New changeset 1b3d4ffcb4d1 by Richard Oudkerk in branch '3.2': New changeset 3585cb1388f2 by Richard Oudkerk in branch 'default': New changeset 7ab7836894c4 by Richard Oudkerk in branch '2.7': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: