classification
Title: Multiprocessing system crash
Type: crash Stage:
Components: Windows Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Rock.Achu, pitrou
Priority: normal Keywords:

Created on 2011-12-30 04:39 by Rock.Achu, last changed 2011-12-30 18:10 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
issue.zip Rock.Achu, 2011-12-30 04:39 File that causes the problem
error.txt Rock.Achu, 2011-12-30 04:42 Huge Traceback
Messages (14)
msg150355 - (view) Author: Rock Achu (Rock.Achu) Date: 2011-12-30 04:39
running this script repeatedly causes corruption in the console window. 

Just before posting this report, it led to a complete windows system freeze.

running python 2.7 x86 on windows x64.

The programs seems to run a couple times (when it is supposed to run once), with the output becoming more and more corrupted. 
One time it printed lots of triple quotes (""") and import os messages, even though none of those are in the script.

To reproduce run the spawner.test.bat file inside the zip.

Remember not to run with stuff you wont want to lose open.
msg150356 - (view) Author: Rock Achu (Rock.Achu) Date: 2011-12-30 04:42
I hit Ctrl-C right after opening the bat file and got this mess:
msg150357 - (view) Author: Rock Achu (Rock.Achu) Date: 2011-12-30 04:50
On a hunch opened task manager. There were over 9000 python processes open. Shocking. 

That was probably the cause of the system freeze.

However I only had 20 threads open. Why so many processes?
msg150358 - (view) Author: Rock Achu (Rock.Achu) Date: 2011-12-30 04:53
Or did I only have 20 threads?
if the program was running in a loop, then it would have spawned 20 threads each, leading to the huge amount of processes, and probably the other issues.


So why does it run so many times?
msg150359 - (view) Author: Rock Achu (Rock.Achu) Date: 2011-12-30 05:26
Ok, using other code, I narrowed it down to this code and before:
for i in xrange(10):
    spawner.newproc(run=True)

repeating infinitely


which only should set a variable (self.busy) to false
msg150360 - (view) Author: Rock Achu (Rock.Achu) Date: 2011-12-30 05:35
By inserting print statements and feverishly killing the process, I narrowed it down to the statement:
self.thread.start()
that causes the issue.
which runs _M_Process.func() in a multiprocessing.Process
msg150361 - (view) Author: Rock Achu (Rock.Achu) Date: 2011-12-30 05:36
More clues:

the function _M_Process.func isn't being called.


????
msg150362 - (view) Author: Rock Achu (Rock.Achu) Date: 2011-12-30 05:37
Changing the function name and removing the argument (self) passed to func doesn't change  a thing.
msg150363 - (view) Author: Rock Achu (Rock.Achu) Date: 2011-12-30 05:39
Alright. The issue stays here. Passing nothing to multiprocessing.Process still reproduces the issue.


someone else know how to deal with this?
msg150364 - (view) Author: Rock Achu (Rock.Achu) Date: 2011-12-30 05:58
Alright. Just running:

import multiprocessing
thread = multiprocessing.Process()
thread.start()

raw_input()


-------------
Causes the crash. This time there is no output, so it is essentailly invisible. 
Caught me by surprise.
msg150370 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-30 10:55
Well, have you read http://docs.python.org/library/multiprocessing.html#windows ?
(especially "Safe importing of main module")
msg150384 - (view) Author: Rock Achu (Rock.Achu) Date: 2011-12-30 18:03
Erm.. No. I was unaware I had to do so.
Next time I'll be more careful.

Anyways, the docs say that python should throw a RuntimeError, but it didn't?
And would it be doable to add a warning at the top of the multiprocessing tutorial/page that section?
msg150386 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-30 18:08
> Anyways, the docs say that python should throw a RuntimeError, but it
> didn't?

I think running it directly would raise a RuntimeError, but importing it wouldn't.

> And would it be doable to add a warning at the top of the
> multiprocessing tutorial/page that section?

In the introduction you have the following note, together with a link to the relevant section:

“Functionality within this package requires that the __main__ module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here.”
msg150387 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-30 18:10
Ok, closing this issue.
History
Date User Action Args
2011-12-30 18:10:01pitrousetstatus: open -> closed
resolution: not a bug
messages: + msg150387
2011-12-30 18:08:06pitrousetmessages: + msg150386
2011-12-30 18:03:32Rock.Achusetmessages: + msg150384
2011-12-30 10:55:55pitrousetnosy: + pitrou
messages: + msg150370
2011-12-30 05:58:42Rock.Achusetmessages: + msg150364
2011-12-30 05:39:32Rock.Achusetmessages: + msg150363
2011-12-30 05:37:33Rock.Achusetmessages: + msg150362
2011-12-30 05:36:04Rock.Achusetmessages: + msg150361
2011-12-30 05:35:12Rock.Achusetmessages: + msg150360
2011-12-30 05:26:38Rock.Achusetmessages: + msg150359
2011-12-30 04:53:44Rock.Achusetmessages: + msg150358
2011-12-30 04:50:16Rock.Achusetmessages: + msg150357
2011-12-30 04:42:56Rock.Achusetfiles: + error.txt

messages: + msg150356
2011-12-30 04:39:42Rock.Achucreate