classification
Title: multiprocessing.Pool throws exception with __main__.py
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Michael.Olson, asksol, bkabrda, davin, jnoller, mdengler, ncoghlan, python-dev, sbt, schlamar
Priority: normal Keywords: patch

Created on 2010-10-16 22:41 by Michael.Olson, last changed 2015-11-19 02:59 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue10128.patch Michael.Olson, 2010-10-20 13:09 Patch for Issue 10128
Messages (13)
msg118905 - (view) Author: Michael Olson (Michael.Olson) Date: 2010-10-16 22:41
In an application with an entry point of __main__.py, multiprocessing.Pool throws the following:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\Dev\Python27\lib\multiprocessing\forking.py", line 346, in main
    prepare(preparation_data)
  File "D:\Dev\Python27\lib\multiprocessing\forking.py", line 454, in prepare
    assert main_name not in sys.modules, main_name
AssertionError: __main__

These messages repeat as long as the application is running.

Demonstration Code, must be in file named __main__.py:
--------------------
import multiprocessing
import time

if __name__ == '__main__':
    pool = multiprocessing.Pool()
    time.sleep(2)
--------------------
msg118969 - (view) Author: Michael Olson (Michael.Olson) Date: 2010-10-17 19:36
I wrapped the offending assertion in a if main_name != '__main__'. I considered not checking the module_name against built-in modules but that seemed likely to be the sort of thing being guarded against, so I left it at an exception for __main__.

Ran a few trivial testing using Pool and Process and it seems to work fine.
msg119134 - (view) Author: Ask Solem (asksol) (Python committer) Date: 2010-10-19 12:36
Is this on Windows?  Does it work for you now?
msg119186 - (view) Author: Michael Olson (Michael.Olson) Date: 2010-10-20 03:35
Sorry about that, yes, this is on Windows XP and 7, 32 bit.

And with the if statement it seems to work fine.

v/r
-- Michael Olson
msg119204 - (view) Author: Michael Olson (Michael.Olson) Date: 2010-10-20 13:09
Ummm, I think I've been unclear on where I was making changes, I changed lib\multiprocessing\forking.py to fix the issue.

Patch attached.
msg119205 - (view) Author: Michael Olson (Michael.Olson) Date: 2010-10-20 13:13
As a note, I didn't attach a patch at first because I was fairly sure I was kludging it into submission, but at least this makes it clear as to what I did.

v/r
-- Michael Olson
msg149989 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-12-21 14:22
The fix from #10845 should be backported to Python 2.7 (bypassing the assertion isn't enough - you want to skip re-executing __main__ entirely)
msg213269 - (view) Author: Martin Dengler (mdengler) * Date: 2014-03-12 16:12
I applied the patch from #10845 to 2.7.6 and it worked well for me.
msg236443 - (view) Author: Marc Schlaich (schlamar) * Date: 2015-02-23 13:55
Please fix this. Scripts with multiprocessing bundled as wheels are broken with Python 2.7 on Windows: https://github.com/pypa/pip/issues/1891
msg236485 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-02-24 10:06
Marc's reference to pip meant I noticed something that I had missed previously - that this issue is referring specifically to the use of setuptools/pip entry points, not to the -m switch.

Entry points shouldn't be hitting this if they're importing the module containing the entry point normally - #10845 specifically related to invoking multiprocessing from a module run directly with the -m switch, which was worked around in 3.2 and 3.3, and then finally fixed properly in 3.4 by the implementation of #19946

So if there's an incompatibility between multiprocessing and entry points, it would be preferable to fix it in pip/setuptools, as that will reach many more installations than just fixing multiprocessing to better tolerate that situation.
msg254840 - (view) Author: Marc Schlaich (schlamar) * Date: 2015-11-18 08:39
Please see my latest comments to https://github.com/pypa/pip/issues/1891. 

tl;dr It is related to the -m switch as pip's wheel launcher does 
  PYTHONPATH=script.exe python -m __main__
msg254872 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-11-19 01:53
Thanks for the investigation Marc.

I'd been hesitant to backport the mitigation patch in #10845 to 2.7.x, (as it *does* represent a behavioural change), but if that code path is currently hitting an assert statement anyway, it seems reasonable to make the change for 2.7.11+
msg254875 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-11-19 02:59
New changeset 5d88c1d413b9 by Nick Coghlan in branch '2.7':
Close #10128: don't rerun __main__.py in multiprocessing
https://hg.python.org/cpython/rev/5d88c1d413b9
History
Date User Action Args
2015-11-19 02:59:57python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg254875

resolution: fixed
stage: resolved
2015-11-19 01:53:09ncoghlansetmessages: + msg254872
2015-11-18 08:39:47schlamarsetmessages: + msg254840
2015-02-24 10:06:23ncoghlansetnosy: + bkabrda
messages: + msg236485
2015-02-24 02:11:54ned.deilysetnosy: + sbt, davin
2015-02-23 13:55:54schlamarsetnosy: + schlamar
messages: + msg236443
2014-03-12 16:12:28mdenglersetmessages: + msg213269
2014-03-12 16:11:23mdenglersetnosy: + mdengler
2011-12-21 14:22:03ncoghlansetnosy: + ncoghlan
messages: + msg149989
2010-10-20 13:13:13Michael.Olsonsetmessages: + msg119205
2010-10-20 13:09:54Michael.Olsonsetstatus: closed -> open
files: + issue10128.patch
messages: + msg119204

keywords: + patch
resolution: not a bug -> (no value)
2010-10-20 07:20:48asksolsetstatus: open -> closed
resolution: not a bug
2010-10-20 03:35:16Michael.Olsonsetmessages: + msg119186
2010-10-19 12:36:51asksolsetmessages: + msg119134
2010-10-17 19:36:58Michael.Olsonsetmessages: + msg118969
2010-10-17 02:58:53r.david.murraysetnosy: + asksol
2010-10-17 02:26:07ned.deilysetnosy: + jnoller
2010-10-16 22:41:50Michael.Olsoncreate