Title: multiprocessing: freezing apps on Windows
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.6
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jnoller Nosy List: davin, jnoller, sbt, sgm, spongebob
Priority: normal Keywords:

Created on 2009-07-11 06:08 by sgm, last changed 2015-03-05 18:03 by davin. This issue is now closed.

Messages (10)
msg90405 - (view) Author: Stuart Mentzer (sgm) Date: 2009-07-11 06:08
Freezing apps with multiprocessing on Windows seems to be broken.

First, in get_command_line in multiprocessing/ I find that
this code:

        if getattr(sys, 'frozen', False):
            return [sys.executable, '--multiprocessing-fork']
            prog = 'from multiprocessing.forking import main; main()'
            return [_python_exe, '-c', prog, '--multiprocessing-fork']

should be:

        elif getattr(sys, 'frozen', False) and not WINEXE:
            return [sys.executable, '--multiprocessing-fork']
            prog = 'from multiprocessing.forking import main; main()'
            return [_python_exe, '-c', prog, '--multiprocessing-fork']

in order for the _python_exe set with multiprocessing.set_executable to
be used rather than your app's exe.

Second, I can then get a working "frozen" package if I include
pythonw.exe (and use set_executable to point to it) and a subset of
Python's Lib directory that my process needs to call. If this is as
intended then it needs to be documented. This may just be a flaw in py2exe.

Third, the multiprocessing documentation page description for
set_executable has example code with the older setExecutable call.
msg90646 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-07-17 22:49
Thanks for noticing this. Unfortunately, I don't know enough about 
py2exe/windows to dig into that part. I can fix the low hanging fruit 
msg90690 - (view) Author: Stuart Mentzer (sgm) Date: 2009-07-18 21:13
Thanks Jesse. If you make broader changes that my patch I am happy to
test on Windows and with py2exe if that is helpful.

I will try to get the py2exe folks to look at the broader
multiprocessing issues I describe as well. If multiprocessing supports
freezing on Windows then it would be nice if its docs had a short primer
on what you'll actually need to do to get that working with freeze tools.

msg90691 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-07-18 21:16
Sounds good - I've personally never used freeze on windows, so having some 
docs and a patch to help make it better for those that do is a definite 
msg91416 - (view) Author: Stuart Mentzer (sgm) Date: 2009-08-08 05:15
Further experimentation revealed that freeze_support() works properly
and the proposed patch is not necessary or desirable for the general
freeze case. In the case of an embedded app that calls set_executable()
it seems that the "else" block calling _python_exe should be taken but
won't be, so I still think there may be a problem here.

In our situation it turned out that an imported module was grabbing the
command line arguments before freeze_support() was called and when that
was addressed the need for a separate python executable and calling
set_executable() was eliminated.

I suggest that the documentation for freeze_support() make it clear that
it must get called before an application processes the arguments or that
the application must allow a --multiprocessing-fork option with a string
value so that the initialization continues until freeze_support() is called.
msg91418 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-08-08 13:46
Thanks for following up on this Stuart - I'll update the docs accordingly
msg98154 - (view) Author: spongebob (spongebob) Date: 2010-01-22 17:38
I can't seem to freeze an app on Windows using multiprocessing.Manager()

Are apps that use Managers freezable??


from multiprocessing import freeze_support,Manager

if __name__ == '__main__':

will throw errors, when run:

Process SyncManager-1:
Traceback (most recent call last):
  File "multi_exe\build\pyi.win32\multi_exe\outPYZ1.pyz/multiprocessing.process"
, line 232, in _bootstrap
  File "multi_exe\build\pyi.win32\multi_exe\outPYZ1.pyz/multiprocessing.process"
, line 88, in run
  File "multi_exe\build\pyi.win32\multi_exe\outPYZ1.pyz/multiprocessing.managers
", line 520, in _run_server
IOError: [Errno 6] The handle is invalid
Traceback (most recent call last):
  File "<string>", line 5, in <module>
  File "multi_exe\build\pyi.win32\multi_exe\outPYZ1.pyz/multiprocessing", line 9
8, in Manager
  File "multi_exe\build\pyi.win32\multi_exe\outPYZ1.pyz/multiprocessing.managers
", line 499, in start
msg189783 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2013-05-21 20:40
Could someone answer the question posed in msg98154 please.
msg191520 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-06-20 14:40
I just tried freezing the program

  from multiprocessing import freeze_support,Manager

  if __name__ == '__main__':
      l = m.list([1,2,3])

using cx_Freeze with Python 2.7, and it worked fine:

  PS> cxfreeze.bat
  copying C:\Python27\lib\site-packages\cx_Freeze\bases\Console.exe -> C:\Tmp\dir\dist\foo.exe
  copying C:\Windows\system32\python27.dll -> C:\Tmp\dir\dist\python27.dll
  writing zip file C:\Tmp\dir\dist\foo.exe

  PS> dist\foo
  [1, 2, 3, 4]
  <ListProxy object, typeid 'list' at 0x206f410>
msg237281 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-03-05 18:03
The original issue now appears addressed in the docs (thanks goes to Stuart and Jesse) though it was not explicitly tracked here as a patch file.

The follow-on secondary issue from spongebob (if that is your real name) could not be reproduced by Richard.

This issue should have been marked closed->resolved some time ago.
Date User Action Args
2015-03-05 18:03:43davinsetstatus: open -> closed

type: behavior

nosy: + davin
messages: + msg237281
resolution: fixed
stage: resolved
2014-02-03 17:07:29BreamoreBoysetnosy: - BreamoreBoy
2013-06-20 14:40:59sbtsetmessages: + msg191520
2013-05-21 22:22:33sbtsetnosy: + sbt
2013-05-21 20:40:00BreamoreBoysetnosy: + BreamoreBoy
messages: + msg189783
2010-01-22 17:38:29spongebobsetnosy: + spongebob
messages: + msg98154
2009-08-08 13:46:26jnollersetmessages: + msg91418
2009-08-08 05:15:25sgmsetmessages: + msg91416
2009-07-18 21:16:14jnollersetmessages: + msg90691
2009-07-18 21:13:45sgmsetmessages: + msg90690
2009-07-17 22:49:35jnollersetmessages: + msg90646
2009-07-17 22:31:54ncoghlansetassignee: jnoller

nosy: + jnoller
2009-07-11 06:08:26sgmcreate