classification
Title: Multiprocessing infinite loop
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: jnoller Nosy List: Kain94, Rodrigue.Alcazar, amaury.forgeotdarc, asksol, davin, dsdale24, jnoller, pthiem, r.david.murray
Priority: normal Keywords: patch

Created on 2010-03-09 09:01 by Kain94, last changed 2015-02-10 14:49 by davin. This issue is now closed.

Files
File name Uploaded Description Edit
multiprocessing.rst.diff Rodrigue.Alcazar, 2011-04-10 14:35 Patch to the multiprocessing documentation file review
Messages (10)
msg100707 - (view) Author: Benjamin VENELLE (Kain94) Date: 2010-03-09 09:01
Hi,

The following code results in an infinite loop -->

# ----
import multiprocessing

def f(m):
	print(m)

p = multiprocessing.Process(target=f, args=('pouet',))
p.start()
# ----

I've firstly think about an issue in my code when Python loads this module so I've added a "if __name__ == '__main__':" and it works well. But, with the following code Python enters in the same infinite loop -->

proc.py:

# ----
import multiprocessing

def f(m):
	print(m)

class P:
	def __init__(self, msg):
		self.msg = msg
	
	def start(self):
		p = multiprocessing.Process(target=f, args=(self.msg,))
		p.start()
# ----

my_script.py:
# ----
from proc import P

p = P("pouet")
p.start()
# ----

It's like Python loads all parent's process memory space before starting process which issues in an infinite call to Process.start() ...
msg100724 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-03-09 14:38
Are you running this on windows?
msg100731 - (view) Author: Benjamin VENELLE (Kain94) Date: 2010-03-09 15:01
Sorry I've not made clear my working platform.

Yes, I'm running Python 3.1.1 32 bit on a Windows 7 x64.
msg100733 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-03-09 15:27
The restriction that imposes the "__name__= '__main__'" idiom also applies when multiprocessing is not used in the main module.

Actually the main module is always reloaded in the subprocess.  The docs should be more explicit about it.
msg133468 - (view) Author: Rodrigue Alcazar (Rodrigue.Alcazar) Date: 2011-04-10 14:35
I have tried to clearly state that the main module is imported by a newly created process. I have also added a comment explaining that an infinite loop like the one Benjamin describes could be created.
msg134187 - (view) Author: Darren Dale (dsdale24) Date: 2011-04-20 22:45
I think I have a similar situation:

C:\Py\Scripts\foo
---
if __name__ == '__main__':
    import bar
    bar.main()

C:\Py\Lib\site-packages\bar.py
---
from multiprocessing import Pool

def task(arg):
    return arg

def main():
    pool = Pool()
    res = pool.apply_async(task, (3.14,))

    print res.get()

if __name__ == '__main__':
    main()


I can run "python bar.py". "python C:\Py\Scripts\foo" yields an infinite stream of errors:

 File "<string>", line 1 in <module>
 File "C:\Python27\lib\multiprocessing\forking.py", line 346, in main
     prepare(preparation_data)
 File "C:\Python27\lib\multiprocessing\forking.py", line 455, in prepare
     file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named foo

This same scheme works fine on linux. Have I just overlooked something simple?
msg134190 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-04-20 23:15
Your code doesn't appear to have anything to do with the reported bug, which is about an infinite loop.  But FYI to my understanding your script can't work on windows, since foo can't be imported.  On linux, foo doesn't need to be imported.
msg181088 - (view) Author: Philip Thiem (pthiem) Date: 2013-02-01 16:49
As an alternative, see http://bugs.python.org/issue10845

It contains a patch for the 3.X series which fixes the infinity loop.

Applying it to the 2.X series will fix the issue and make a change the documentation unnecessary.
msg181089 - (view) Author: Philip Thiem (pthiem) Date: 2013-02-01 16:53
Actually sorry, now that I reread the details a second time, I'm not sure that is this the same deal.  I'll just file a separate bug.
msg234991 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-01-29 21:21
On Windows 7 (64-bit), it was not possible to reproduce any infinite looping behavior with the supplied example code.

Specifically, with the two examples from Benjamin, the observed behavior when running them was the same under 2.7.8 and default (3.5):  a RuntimeError results with a message suggesting that the code perhaps needs to leverage the "if __name__ == '__main__'" idiom to avoid having both parent and all subsequent children processes starting up a new process because they're all unintentionally running the same lines of code intended only for the parent to run.  Adding that idiomatic test to each of the two examples permits them to run to a happy conclusion.  That is, in the case of the first example we make that one-line code change to read:
# ---
import multiprocessing

def f(m):
    print(m)

if __name__ == '__main__':
    p = multiprocessing.Process(target=f, args=('pouet',))
    p.start()
# ---

This would be a recommended practice on unix-y systems as well as Windows.

Aside: It was not possible to reproduce the issue injected by Darren either -- perhaps the example code provided was not quite what he intended.

The infinite looping behavior described in the original issue description might well have been reproducible in much earlier releases.  In the current default (3.5) branch (or in 2.7.8), it is no longer possible to reproduce.  I'm tempted to mark this as "out of date" but instead will opt for "works for me" and close the issue.
History
Date User Action Args
2015-02-10 14:49:53davinsetstage: needs patch -> resolved
2015-01-29 21:21:04davinsetstatus: open -> closed

nosy: + davin
messages: + msg234991

resolution: works for me
2013-02-01 16:53:29pthiemsetmessages: + msg181089
2013-02-01 16:49:56pthiemsetnosy: + pthiem
messages: + msg181088
2011-04-20 23:15:32r.david.murraysetmessages: + msg134190
2011-04-20 22:45:05dsdale24setnosy: + dsdale24
messages: + msg134187
2011-04-10 14:35:01Rodrigue.Alcazarsetfiles: + multiprocessing.rst.diff

nosy: + Rodrigue.Alcazar
messages: + msg133468

keywords: + patch
2010-10-05 15:11:50asksolsetnosy: + asksol
2010-03-09 16:10:23r.david.murraysetversions: + Python 2.6, Python 2.7, Python 3.2
nosy: amaury.forgeotdarc, jnoller, r.david.murray, Kain94
priority: normal
components: + Documentation, - Library (Lib)
type: crash -> behavior
stage: needs patch
2010-03-09 15:27:56amaury.forgeotdarcsetassignee: jnoller

messages: + msg100733
nosy: + amaury.forgeotdarc, jnoller
2010-03-09 15:01:27Kain94setmessages: + msg100731
2010-03-09 14:38:43r.david.murraysetnosy: + r.david.murray
messages: + msg100724
2010-03-09 09:01:52Kain94create