Author jnoller
Recipients jnoller
Date 2009-08-05.20:23:50
SpamBayes Score 4.03014e-10
Marked as misclassified No
Message-id <1249503832.66.0.0114205396453.issue6653@psf.upfronthosting.co.za>
In-reply-to
Content
I have example code to show this. It creates a system-wide memory leak 
on Linux/Unix (present until the next reboot), unless the last statement 
in the target of mp.Process ensures a manual clean up of the globals.

The problem is line 353 in multiprocessing/forking.py. The function 
exit() is defined as os._exit on Linux and ExitProcess on Windows, none 
of which allows normal clean up.

>>> help(os._exit)
Help on built-in function _exit in module nt:

_exit(...)
   _exit(status)

   Exit to the system with specified status, without normal exit 
processing.


The problem is fixed if line 353 in forking.py is changed from

exit(exitcode)

to

sys.exit(exitcode)




Test run without bugfix:

G:\DEVELO~1\SHARED~2>python test.py
open handle to 569f439b24e24fc8a547b81932616066
[[ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]]
open handle to 0582d4b161c546f582c1c96e7bd0c39d
open handle to 569f439b24e24fc8a547b81932616066
modified array
closed handle to 569f439b24e24fc8a547b81932616066
[[ 1.  1.  1.  0.]
 [ 1.  1.  1.  0.]]
closed handle to 569f439b24e24fc8a547b81932616066


You can see here that opening and closing of handles are unmatched. This 
is on Windows, where the kernel ensures the clean-up, so it may not 
matter. But on Unix this would have created a permament (system wide) 
memory leak! What is happening here is globals not being cleaned up due 
to the use of os._exit instead of sys.exit.


Test run with bugfix:

G:\DEVELO~1\SHARED~2>python test.py
open handle to 930778d27b414253bc329f2b70adaa05
[[ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]]
open handle to 3f6cebf8c5de413685bb770d02ae9666
open handle to 930778d27b414253bc329f2b70adaa05
modified array
closed handle to 930778d27b414253bc329f2b70adaa05
closed handle to 3f6cebf8c5de413685bb770d02ae9666
[[ 1.  1.  1.  0.]
 [ 1.  1.  1.  0.]]
closed handle to 930778d27b414253bc329f2b70adaa05



Now all allocations and deallocations are matched.


Regards,
Sturla Molden
History
Date User Action Args
2009-08-05 20:23:52jnollersetrecipients: + jnoller
2009-08-05 20:23:52jnollersetmessageid: <1249503832.66.0.0114205396453.issue6653@psf.upfronthosting.co.za>
2009-08-05 20:23:50jnollerlinkissue6653 messages
2009-08-05 20:23:50jnollercreate