This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Problems with multiprocessing, Python embedding and Windows
Type: behavior Stage: resolved
Components: Extension Modules, Windows Versions: Python 2.6
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, amyodov, davin, sbt
Priority: normal Keywords:

Created on 2010-06-29 21:34 by amyodov, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (4)
msg108953 - (view) Author: Alexander Myodov (amyodov) Date: 2010-06-29 21:34
Hello,

I am using Python 2.6.5 and Cython (to embed the Python interpreter) to create a simple launcher.exe launcher which then imports and uses the remaining logic from the other modules, conveniently stored in a single modules.zip file (i.e. importing from inside zip file). Such setup causes some issues when it comes to the multiprocessing.

First of all, the __main__ module is built as a binary (via Cython->mingw), but the stuff like freeze is not used. In multiprocessing/forking.py, when it comes to determination of the primary path, there is a WINEXE variable which affects the further processing. In its assignment, getattr(sys, 'frozen', False) obviously results in False, causing the WINEXE variable be False too (though in my case it is obviously wrong, the stuff like imp.is_builtin("__main__") shows that the main module is indeed builtin). This is the first problem which probably could be solved by adding an additional check for something like imp.is_builtin("__main__") to WINEXE assignment.
Further, as the WINEXE is wrongly False, it causes the "if not WINEXE:" branch to be processed, and adds the 'main_path' to the data, containing the name of the .exe file. 
On the forked side, d['main_path'] is received and parsed, and the name of the .exe file is split out of its extension, and such name is attempted to be imported, using imp.find_module/imp.load_module. The import fails, as there is indeed no such files wildly lying around in the installation; on the other hand, to get a quick fix of the problem, I thought of adding the appropriate launcher.py(.pyo/.pyc) files into the modules.zip, so that they still could be externally imported by the "launcher" name, even though it is already builtin under the __main__ name; but the second problem is that the imp.find_module/imp.load_module functions do not find the module inside the zip files, like the import statement does.
msg191514 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-06-20 14:24
We don't do non-security updates on Python 2.6 anymore.

As a workaround you might be able to do something like

    import sys, multiprocessing
    sys.frozen = True    # or multiprocessing.forking.WINEXE = True

    ...

    if __name__ == '__main__':
        multiprocessing.freeze_support()
        ...

(I am not familiar with using Cython.)
msg224353 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-30 22:17
Can this be closed as "out of date"?
msg235689 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-02-10 15:09
Closing on the basis that:
1. Richard provided a proposed workaround but did not receive a response from the OP in 1.5 years.
2. Issues encountered by the OP may have been addressed in part by changes in the way import works in the past 4.5 years.
3. This issue has been effectively "pending" (though not formally marked as such) for more than a year without further information being received from the OP.
History
Date User Action Args
2022-04-11 14:57:03adminsetgithub: 53368
2015-02-10 15:09:39davinsetstatus: pending -> closed
resolution: out of date
stage: resolved
2015-02-10 15:09:18davinsetstatus: open -> pending

nosy: + davin
messages: + msg235689

type: behavior
2014-07-30 22:17:17BreamoreBoysetnosy: + BreamoreBoy
messages: + msg224353
2013-06-20 14:24:39sbtsetmessages: + msg191514
2013-06-14 14:26:27sbtsetnosy: + sbt
2010-06-29 21:34:41amyodovcreate