classification
Title: multiprocessing cannot spawn child from a Windows service
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.1, Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brian.curtin Nosy List: brian.curtin, jnoller, mher.movsisyan, orlenko, python-dev, schlamar, tim.golden
Priority: normal Keywords: patch

Created on 2009-02-06 02:00 by orlenko, last changed 2015-01-07 10:24 by schlamar. This issue is now closed.

Files
File name Uploaded Description Edit
forking-patch orlenko, 2009-02-06 02:00 Patch of the forking module
test_issue5162.py mher.movsisyan, 2011-01-03 13:10 unit test
forking_r87679.patch mher.movsisyan, 2011-01-03 13:21 patch review
Messages (9)
msg81247 - (view) Author: Volodymyr Orlenko (orlenko) Date: 2009-02-06 02:00
I think I've found a small bug with multiprocessing package on 
Windows. If you try to start a multiprocessing.Process from a Python- 
based Windows service, the child process will fail to run. When 
running the parent process as a regular Python program, everything 
works as expected. 
I've tracked the problem down to how main_path is prepared in 
multiprocessing.forking.get_preparation_data() (lines 370-377): 
def get_preparation_data(name): 
    [...skipped a few lines...] 
    if not WINEXE: 
        main_path = getattr(sys.modules['__main__'], '__file__', None) 
        if not main_path and sys.argv[0] not in ('', '-c'): 
            main_path = sys.argv[0] 
        if main_path is not None: 
            if not os.path.isabs(main_path) and \ 
                                      process.ORIGINAL_DIR is not 
None: 
                main_path = os.path.join(process.ORIGINAL_DIR, 
main_path) 
            d['main_path'] = os.path.normpath(main_path) 
    return d 
When the program is running as a Windows service, but is not packaged 
into a single executable, main_path will become the path to the 
service executable (typically, pythonservice.exe). When this data 
makes it to the child process, the prepare() function will treat 
main_path as a path to a python module, and will try to import it. 
This causes it to fail. 
My quick-and-dirty solution was to check in get_preparation_data() if 
main_path ends with '.exe', and if it does, to not pass it at all. 
This solves the problem in my case, but perhaps there's a better way 
to fix this? Here is my version of get_preparation_data(): 
def get_preparation_data(name): 
    ''' 
    Return info about parent needed by child to unpickle process 
object 
    ''' 
    from .util import _logger, _log_to_stderr 
    d = dict( 
        name=name, 
        sys_path=sys.path, 
        sys_argv=sys.argv, 
        log_to_stderr=_log_to_stderr, 
        orig_dir=process.ORIGINAL_DIR, 
        authkey=process.current_process().authkey, 
        ) 
    if _logger is not None: 
        d['log_level'] = _logger.getEffectiveLevel() 
    if not WINEXE: 
        main_path = getattr(sys.modules['__main__'], '__file__', None) 
        if not main_path and sys.argv[0] not in ('', '-c'): 
            main_path = sys.argv[0] 
        if main_path is not None: 
            if not os.path.isabs(main_path) and \ 
                                      process.ORIGINAL_DIR is not 
None: 
                main_path = os.path.join(process.ORIGINAL_DIR, 
main_path) 
            if not main_path.endswith('.exe'): 
                d['main_path'] = os.path.normpath(main_path) 
    return d
msg125166 - (view) Author: Mher Movsisyan (mher.movsisyan) Date: 2011-01-03 13:10
Attached test case demonstrates the issue.
msg125167 - (view) Author: Mher Movsisyan (mher.movsisyan) Date: 2011-01-03 13:21
Treating python services like frozen executables solves the issue. The patch is attached.
msg133523 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-04-11 14:27
This looks reasonable to me. If no one beats me to it, I'll check it in tonight.
msg133552 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-11 23:01
New changeset 1f41b1ab8924 by brian.curtin in branch '3.1':
Fix #5162. Allow child spawning from Windows services (via pywin32).
http://hg.python.org/cpython/rev/1f41b1ab8924

New changeset 184ae02e3221 by brian.curtin in branch '3.2':
Fix #5162. Allow child spawning from Windows services (via pywin32).
http://hg.python.org/cpython/rev/184ae02e3221

New changeset 3c2bdea18b5c by brian.curtin in branch 'default':
Fix #5162. Allow child spawning from Windows services (via pywin32).
http://hg.python.org/cpython/rev/3c2bdea18b5c

New changeset 6507a5ba5c27 by brian.curtin in branch '2.7':
Fix #5162. Allow child spawning from Windows services (via pywin32).
http://hg.python.org/cpython/rev/6507a5ba5c27
msg133553 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-04-11 23:02
Thanks for the patch!
msg133554 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-11 23:06
New changeset a280672d3d8d by brian.curtin in branch '2.7':
Add NEWS item for #5162.
http://hg.python.org/cpython/rev/a280672d3d8d
msg133555 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-11 23:20
New changeset c26474c6504a by brian.curtin in branch '3.1':
Add NEWS item for #5162.
http://hg.python.org/cpython/rev/c26474c6504a

New changeset 68ef2bf1aa99 by brian.curtin in branch '3.2':
Add NEWS item for #5162.
http://hg.python.org/cpython/rev/68ef2bf1aa99

New changeset 2c4043070f05 by brian.curtin in branch 'default':
Add NEWS item for #5162.
http://hg.python.org/cpython/rev/2c4043070f05
msg233572 - (view) Author: Marc Schlaich (schlamar) * Date: 2015-01-07 10:24
This issue is not fully fixed, there are some occasions where you can still run into it. One example is if you want to spawn a new multiprocessing.Process as sub process in a multiprocessing.Process:


pythonservice.exe
  - multiprocessing.Process
    - multiprocessing.Process (does not start!)

In this case you get:

  WINSERVICE: False
  WINEXE: False
  _python_exe: C:\Python27\python.exe
  prep data: {'authkey': '...', 'sys_path': [...], 'name': 'test', 'orig_dir': '...', 'sys_argv': ['C:\\Python27\\lib\\site-packages\\win32\\PythonService.exe'], 'main_path': 'C:\\Python27\\lib\\site-packages\\win32\\PythonService.exe', 'log_to_stderr': False}
History
Date User Action Args
2015-01-07 10:24:35schlamarsetnosy: + schlamar
messages: + msg233572
2011-04-18 23:28:37brian.curtinsetstatus: open -> closed
2011-04-11 23:20:08python-devsetmessages: + msg133555
2011-04-11 23:06:57python-devsetmessages: + msg133554
2011-04-11 23:02:38brian.curtinsetassignee: jnoller -> brian.curtin
resolution: fixed
messages: + msg133553
stage: patch review -> resolved
2011-04-11 23:01:37python-devsetnosy: + python-dev
messages: + msg133552
2011-04-11 14:27:37brian.curtinsetmessages: + msg133523
2011-04-11 14:21:39pitrousetstage: test needed -> patch review
2011-04-11 14:21:35pitrousetnosy: + tim.golden, brian.curtin

components: + Windows
versions: + Python 3.3
2011-01-03 13:21:31mher.movsisyansetfiles: + forking_r87679.patch
nosy: jnoller, orlenko, mher.movsisyan
messages: + msg125167
2011-01-03 13:10:28mher.movsisyansetfiles: + test_issue5162.py
nosy: + mher.movsisyan
messages: + msg125166

2010-07-10 09:41:43BreamoreBoysetversions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
2009-03-29 15:44:33jnollersetpriority: normal
2009-03-06 21:44:06ajaksu2setkeywords: + patch
assignee: jnoller
stage: test needed
2009-02-06 02:00:31orlenkocreate