classification
Title: ForkAwareThreadLock leak after fork
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Marc.Brünink, neologix, python-dev, sbt
Priority: normal Keywords: patch

Created on 2013-03-27 03:22 by Marc.Brünink, last changed 2013-04-17 23:00 by sbt. This issue is now closed.

Files
File name Uploaded Description Edit
test_fork_02.py Marc.Brünink, 2013-03-27 03:22 test case
afterfork.diff neologix, 2013-03-27 07:53
forkawarethreadlock.patch sbt, 2013-03-27 10:58
Messages (6)
msg185315 - (view) Author: Marc Brünink (Marc.Brünink) Date: 2013-03-27 03:22
Importing multiprocessing.managers and creating new processes recursively is not a good idea. You will run out of memory very fast.  On my
machine with 8GB it only takes about 20 forks until I start swapping.
msg185318 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-03-27 07:53
That's due to a leak in forker-registered handlers:
The _afterfork_registry is never cleared, so spawning processes recursively keeps feeling it, which ends up consuming a huge amount of memory and slowing process creation greatly.

Could you try the attached patch?
msg185322 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-03-27 10:58
It seems to be a problem with ForkAwareThreadLock.  Could you try the attached patch?
msg185323 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-03-27 11:02
_afterfork_registry is not supposed to be cleared.  But the problem with ForkAwareThreadLocal meant that the size of the registry at generation n is 2**n!
msg185550 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-03-30 08:51
Richard, your patch is indeed better, and fixes the problem.
With a test it should be ready to go in.
msg187205 - (view) Author: Roundup Robot (python-dev) Date: 2013-04-17 20:29
New changeset 66731a1b1aa4 by Richard Oudkerk in branch '2.7':
Issue #17555: Fix ForkAwareThreadLock so that size of after fork
http://hg.python.org/cpython/rev/66731a1b1aa4

New changeset 21314f27a40c by Richard Oudkerk in branch '3.3':
Issue #17555: Fix ForkAwareThreadLock so that size of after fork
http://hg.python.org/cpython/rev/21314f27a40c

New changeset 2c350a75ff43 by Richard Oudkerk in branch 'default':
Issue #17555: Fix ForkAwareThreadLock so that size of after fork
http://hg.python.org/cpython/rev/2c350a75ff43
History
Date User Action Args
2013-04-17 23:00:44sbtsetstatus: open -> closed
resolution: fixed
stage: resolved
2013-04-17 20:29:14python-devsetnosy: + python-dev
messages: + msg187205
2013-04-17 17:14:33sbtsettitle: Creating new processes after importing multiprocessing.managers consumes more and more memory -> ForkAwareThreadLock leak after fork
versions: + Python 3.3, Python 3.4
2013-03-30 08:51:25neologixsetmessages: + msg185550
2013-03-27 11:02:14sbtsetmessages: + msg185323
2013-03-27 10:58:33sbtsetfiles: + forkawarethreadlock.patch

messages: + msg185322
2013-03-27 07:53:37neologixsetnosy: + sbt
2013-03-27 07:53:28neologixsetfiles: + afterfork.diff

nosy: + neologix
messages: + msg185318

keywords: + patch
2013-03-27 03:22:55Marc.Brüninkcreate