Created on 2013-03-27 03:22 by Marc.Brünink, last changed 2013-04-17 23:00 by sbt. This issue is now closed.
| Files | ||||
|---|---|---|---|---|
| File name | Uploaded | Description | Edit | |
| test_fork_02.py | Marc.Brünink, 2013-03-27 03:22 | test case | ||
| afterfork.diff | neologix, 2013-03-27 07:53 | |||
| forkawarethreadlock.patch | sbt, 2013-03-27 10:58 | |||
| Messages (6) | |||
|---|---|---|---|
| msg185315 - (view) | Author: Marc Brünink (Marc.Brünink) | Date: 2013-03-27 03:22 | |
Importing multiprocessing.managers and creating new processes recursively is not a good idea. You will run out of memory very fast. On my machine with 8GB it only takes about 20 forks until I start swapping. |
|||
| msg185318 - (view) | Author: Charles-François Natali (neologix) *  |
Date: 2013-03-27 07:53 | |
That's due to a leak in forker-registered handlers: The _afterfork_registry is never cleared, so spawning processes recursively keeps feeling it, which ends up consuming a huge amount of memory and slowing process creation greatly. Could you try the attached patch? |
|||
| msg185322 - (view) | Author: Richard Oudkerk (sbt) * |
Date: 2013-03-27 10:58 | |
It seems to be a problem with ForkAwareThreadLock. Could you try the attached patch? |
|||
| msg185323 - (view) | Author: Richard Oudkerk (sbt) * |
Date: 2013-03-27 11:02 | |
_afterfork_registry is not supposed to be cleared. But the problem with ForkAwareThreadLocal meant that the size of the registry at generation n is 2**n! |
|||
| msg185550 - (view) | Author: Charles-François Natali (neologix) * |
Date: 2013-03-30 08:51 | |
Richard, your patch is indeed better, and fixes the problem. With a test it should be ready to go in. |
|||
| msg187205 - (view) | Author: Roundup Robot (python-dev) | Date: 2013-04-17 20:29 | |
New changeset 66731a1b1aa4 by Richard Oudkerk in branch '2.7': Issue #17555: Fix ForkAwareThreadLock so that size of after fork http://hg.python.org/cpython/rev/66731a1b1aa4 New changeset 21314f27a40c by Richard Oudkerk in branch '3.3': Issue #17555: Fix ForkAwareThreadLock so that size of after fork http://hg.python.org/cpython/rev/21314f27a40c New changeset 2c350a75ff43 by Richard Oudkerk in branch 'default': Issue #17555: Fix ForkAwareThreadLock so that size of after fork http://hg.python.org/cpython/rev/2c350a75ff43 |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2013-04-17 23:00:44 | sbt | set | status: open -> closed resolution: fixed stage: resolved |
| 2013-04-17 20:29:14 | python-dev | set | nosy:
+ python-dev messages: + msg187205 |
| 2013-04-17 17:14:33 | sbt | set | title: Creating new processes after importing multiprocessing.managers consumes more and more memory -> ForkAwareThreadLock leak after fork versions: + Python 3.3, Python 3.4 |
| 2013-03-30 08:51:25 | neologix | set | messages: + msg185550 |
| 2013-03-27 11:02:14 | sbt | set | messages: + msg185323 |
| 2013-03-27 10:58:33 | sbt | set | files:
+ forkawarethreadlock.patch messages: + msg185322 |
| 2013-03-27 07:53:37 | neologix | set | nosy:
+ sbt |
| 2013-03-27 07:53:28 | neologix | set | files:
+ afterfork.diff nosy: + neologix messages: + msg185318 keywords: + patch |
| 2013-03-27 03:22:55 | Marc.Brünink | create | |