classification
Title: multiprocessing: freezing apps on Windows
Type: Stage:
Components: Library (Lib) Versions: Python 2.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: jnoller Nosy List: BreamoreBoy, jnoller, sbt, sgm, spongebob
Priority: normal Keywords:

Created on 2009-07-11 06:08 by sgm, last changed 2013-05-21 22:22 by sbt.

Messages (8)
msg90405 - (view) Author: Stuart Mentzer (sgm) Date: 2009-07-11 06:08
Freezing apps with multiprocessing on Windows seems to be broken.

First, in get_command_line in multiprocessing/forking.py I find that
this code:

        if getattr(sys, 'frozen', False):
            return [sys.executable, '--multiprocessing-fork']
        else:
            prog = 'from multiprocessing.forking import main; main()'
            return [_python_exe, '-c', prog, '--multiprocessing-fork']

should be:

        elif getattr(sys, 'frozen', False) and not WINEXE:
            return [sys.executable, '--multiprocessing-fork']
        else:
            prog = 'from multiprocessing.forking import main; main()'
            return [_python_exe, '-c', prog, '--multiprocessing-fork']

in order for the _python_exe set with multiprocessing.set_executable to
be used rather than your app's exe.

Second, I can then get a working "frozen" package if I include
pythonw.exe (and use set_executable to point to it) and a subset of
Python's Lib directory that my process needs to call. If this is as
intended then it needs to be documented. This may just be a flaw in py2exe.

Third, the multiprocessing documentation page description for
set_executable has example code with the older setExecutable call.
msg90646 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-07-17 22:49
Thanks for noticing this. Unfortunately, I don't know enough about 
py2exe/windows to dig into that part. I can fix the low hanging fruit 
though
msg90690 - (view) Author: Stuart Mentzer (sgm) Date: 2009-07-18 21:13
Thanks Jesse. If you make broader changes that my patch I am happy to
test on Windows and with py2exe if that is helpful.

I will try to get the py2exe folks to look at the broader
multiprocessing issues I describe as well. If multiprocessing supports
freezing on Windows then it would be nice if its docs had a short primer
on what you'll actually need to do to get that working with freeze tools.

Stuart
msg90691 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-07-18 21:16
Sounds good - I've personally never used freeze on windows, so having some 
docs and a patch to help make it better for those that do is a definite 
plus
msg91416 - (view) Author: Stuart Mentzer (sgm) Date: 2009-08-08 05:15
Further experimentation revealed that freeze_support() works properly
and the proposed patch is not necessary or desirable for the general
freeze case. In the case of an embedded app that calls set_executable()
it seems that the "else" block calling _python_exe should be taken but
won't be, so I still think there may be a problem here.

In our situation it turned out that an imported module was grabbing the
command line arguments before freeze_support() was called and when that
was addressed the need for a separate python executable and calling
set_executable() was eliminated.

I suggest that the documentation for freeze_support() make it clear that
it must get called before an application processes the arguments or that
the application must allow a --multiprocessing-fork option with a string
value so that the initialization continues until freeze_support() is called.
msg91418 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-08-08 13:46
Thanks for following up on this Stuart - I'll update the docs accordingly
msg98154 - (view) Author: spongebob (spongebob) Date: 2010-01-22 17:38
I can't seem to freeze an app on Windows using multiprocessing.Manager()


Are apps that use Managers freezable??

Example:

from multiprocessing import freeze_support,Manager

if __name__ == '__main__':
    freeze_support()
    m=Manager()

will throw errors, when run:

Process SyncManager-1:
Traceback (most recent call last):
  File "multi_exe\build\pyi.win32\multi_exe\outPYZ1.pyz/multiprocessing.process"
, line 232, in _bootstrap
  File "multi_exe\build\pyi.win32\multi_exe\outPYZ1.pyz/multiprocessing.process"
, line 88, in run
  File "multi_exe\build\pyi.win32\multi_exe\outPYZ1.pyz/multiprocessing.managers
", line 520, in _run_server
IOError: [Errno 6] The handle is invalid
Traceback (most recent call last):
  File "<string>", line 5, in <module>
  File "multi_exe\build\pyi.win32\multi_exe\outPYZ1.pyz/multiprocessing", line 9
8, in Manager
  File "multi_exe\build\pyi.win32\multi_exe\outPYZ1.pyz/multiprocessing.managers
", line 499, in start
EOFError
msg189783 - (view) Author: Mark Lawrence (BreamoreBoy) Date: 2013-05-21 20:40
Could someone answer the question posed in msg98154 please.
History
Date User Action Args
2013-05-21 22:22:33sbtsetnosy: + sbt
2013-05-21 20:40:00BreamoreBoysetnosy: + BreamoreBoy
messages: + msg189783
2010-01-22 17:38:29spongebobsetnosy: + spongebob
messages: + msg98154
2009-08-08 13:46:26jnollersetmessages: + msg91418
2009-08-08 05:15:25sgmsetmessages: + msg91416
2009-07-18 21:16:14jnollersetmessages: + msg90691
2009-07-18 21:13:45sgmsetmessages: + msg90690
2009-07-17 22:49:35jnollersetmessages: + msg90646
2009-07-17 22:31:54ncoghlansetassignee: jnoller

nosy: + jnoller
2009-07-11 06:08:26sgmcreate