classification
Title: multiprocessing 'NoneType' object is not callable
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.3, Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: Arfrever, belopolsky, benjamin.peterson, chris.jerdonek, georg.brandl, hynek, larry, mcdonc, mitar, pjenvey, python-dev, sbt, tseaver
Priority: release blocker Keywords: patch

Created on 2012-09-07 18:56 by mcdonc, last changed 2013-02-02 16:18 by benjamin.peterson. This issue is now closed.

Files
File name Uploaded Description Edit
shutdown_typeerror-tip.patch mcdonc, 2012-09-07 19:15 review
shutdown_typeerror-27.patch mcdonc, 2012-09-07 19:23 review
Messages (18)
msg170003 - (view) Author: Chris McDonough (mcdonc) Date: 2012-09-07 18:56
The symptom is an exact duplicate of the symptom reported in the following (closed) issue:

http://bugs.python.org/issue9775

The issue is also related to the following other issues:

http://bugs.python.org/issue4106
http://bugs.python.org/issue9205
http://bugs.python.org/issue9207

To reproduce the symptom driving the patches that will be attached to this issue:

  git clone git://github.com/pypa/pip.git
  cd pip
  /any/python setup.py dev
  /any/python setup.py test

You can either wait for the entire test suite to finish or you can press ctrl-C at any time (the tests take a long time).  In either case, a traceback like the following will be printed to the console.

  Error in atexit._run_exitfuncs:
  Traceback (most recent call last):
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
      info('process shutting down')
  TypeError: 'NoneType' object is not callable
  Error in sys.exitfunc:
  Traceback (most recent call last):
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
      func(*targs, **kargs)
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
      info('process shutting down')
  TypeError: 'NoneType' object is not callable

From what I understand in other issues, multiprocessing.util._exit_function shouldn't actually be called *after*  the containing module's globals are destroyed (it should be called before), but this does indeed happen.

Patches will be attached that anticipate the symptom and prevent a shutdown error.  One will be attached for Python 2.7 branch, the other for the Python tip.   Each causes functions that are called at shutdown time to keep a reference around to other functions and globals used within the function, and each does some checks for the insane state and prevents an error from happening in this insane state.
msg170008 - (view) Author: Chris McDonough (mcdonc) Date: 2012-09-07 19:15
Patch for tip.
msg170009 - (view) Author: Chris McDonough (mcdonc) Date: 2012-09-07 19:23
2.7 branch patch.
msg170022 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-09-08 04:24
The patch makes sense.  I'll take another look over the weekend, but it seems to be ready to be applied.
msg170023 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-08 05:21
+    # NB: we hold on to references to functions in the arglist due to the

This is a nit, but I think adding "NB:", "Note:", etc. to the beginning of a comment is redundant because by being a comment it is already implicit that it should be noted.
msg170119 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-09 17:28
New changeset 27d410dd5431 by Alexander Belopolsky in branch '3.2':
Issue #15881: Fixed atexit hook in multiprocessing.
http://hg.python.org/cpython/rev/27d410dd5431

New changeset 08c680918ff8 by Alexander Belopolsky in branch 'default':
Issue #15881: Fixed atexit hook in multiprocessing.
http://hg.python.org/cpython/rev/08c680918ff8
msg170120 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-09 17:31
New changeset db67b848ddc3 by Alexander Belopolsky in branch '3.2':
Issue #15881: Fixed 3.2 backport.
http://hg.python.org/cpython/rev/db67b848ddc3
msg170121 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-09-09 17:55
Applied to 3.2 and 3.3.  Thanks for the patch!

Leaving it open pending 2.7 commit.
msg170122 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-09 18:12
New changeset b05547e8ff92 by Alexander Belopolsky in branch '3.2':
Issue #15881: Added NEWS entry and proper credit.
http://hg.python.org/cpython/rev/b05547e8ff92
msg170187 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-09-10 13:07
I see the same error on Windows (when pressing ^C), but on Linux I get

Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 28, in _run_exitfuncs
    import traceback
  File "/usr/lib/python2.7/traceback.py", line 3, in <module>
    import linecache
  File "/usr/lib/python2.7/linecache.py", line 9, in <module>
    import os
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'

This also suggests that module teardown has begun before/while sys.exitfunc is running.
msg170215 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-09-10 18:41
I suspect the problem is caused by nose's isolate plugin.

With this enabled, a copy of sys.modules is saved before each test and then restored after the test.  This causes garbage collection of newly imported modules.  The destructor for the module type causes all globals to be replaced by None.

This will break the atexit function registered by multiprocessing since it depends on globals.

PS. A simple work-around (which does not require people to upgrade to a bugfixed version of Python) is to put

    try:
        import multiprocessing
    except ImportError:
        pass

near the beginning of setup.py.  After this change I don't get the error when running "python setup.py test".
msg170220 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-09-10 19:23
Actually, I am not so sure it is the isolate plugin.  But I do think that sys.modules is being manipulated somewhere before shutdown.
msg170230 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-09-10 20:22
Actually it is test.with_project_on_sys_path() in setuptools/commands/test.py that does the save/restore of sys.modules.  See

    http://www.eby-sarna.com/pipermail/peak/2010-May/003357.html
msg170446 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-13 16:28
New changeset 2b79b4848f44 by Richard Oudkerk in branch 'default':
Issue #15881: Clarify comment in exit function
http://hg.python.org/cpython/rev/2b79b4848f44
msg180305 - (view) Author: Tres Seaver (tseaver) * Date: 2013-01-20 17:25
I can reproduce the bug against the 2.7 tip.

Reviewing the 2.7 patch:

- The 2.7 tip has 'Misc/ACKS' instead of 'Doc/ACKS.txt'.

- In 'Misc/ACKS', the line adding 'Chris McDonough' should add it in alpha order.

- The remainder of the patch looks correct, and applies cleanly to the tip of the 2.7 branch.

- After applying the patch to the 2.7 tip, I can no longer reproduce the bug.

- After applying the patch, the full test suite ('./python -m test.testall -j3') passes all tests.
msg180438 - (view) Author: Philip Jenvey (pjenvey) * (Python committer) Date: 2013-01-22 23:13
Targeting this for 2.7.4. If Alexander doesn't get to it, ping me and I'll do it
msg181024 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-01-31 14:35
Philip, if you could backport, that'd be great.
msg181173 - (view) Author: Roundup Robot (python-dev) Date: 2013-02-02 16:17
New changeset 0a58fa8e9bac by Benjamin Peterson in branch '2.7':
Issue #15881: Fixed atexit hook in multiprocessing.
http://hg.python.org/cpython/rev/0a58fa8e9bac
History
Date User Action Args
2013-02-02 16:18:57benjamin.petersonsetstatus: open -> closed
2013-02-02 16:18:46benjamin.petersonsetversions: - Python 2.7
2013-02-02 16:17:08python-devsetmessages: + msg181173
2013-01-31 14:35:57benjamin.petersonsetmessages: + msg181024
2013-01-22 23:13:17pjenveysetpriority: normal -> release blocker
nosy: + pjenvey, benjamin.peterson, georg.brandl, larry
messages: + msg180438

2013-01-20 22:03:04hyneksetnosy: + hynek
2013-01-20 17:25:18tseaversetnosy: + tseaver
messages: + msg180305
2012-09-13 16:28:17python-devsetmessages: + msg170446
2012-09-12 08:48:37mitarsetnosy: + mitar
2012-09-10 22:41:46Arfreversetnosy: + Arfrever
2012-09-10 20:22:45sbtsetmessages: + msg170230
2012-09-10 19:23:33sbtsetmessages: + msg170220
2012-09-10 18:41:46sbtsetmessages: + msg170215
2012-09-10 13:07:59sbtsetmessages: + msg170187
2012-09-09 18:12:01python-devsetmessages: + msg170122
2012-09-09 17:55:39belopolskysetresolution: fixed
messages: + msg170121
stage: commit review -> resolved
2012-09-09 17:31:20python-devsetmessages: + msg170120
2012-09-09 17:28:06python-devsetnosy: + python-dev
messages: + msg170119
2012-09-08 05:21:30chris.jerdoneksetnosy: + chris.jerdonek
messages: + msg170023
2012-09-08 04:24:28belopolskysetnosy: + belopolsky
messages: + msg170022

assignee: belopolsky
stage: patch review -> commit review
2012-09-07 19:40:11mark.dickinsonsetstage: patch review
versions: - Python 2.6, Python 3.1
2012-09-07 19:23:27mcdoncsetfiles: + shutdown_typeerror-27.patch

messages: + msg170009
2012-09-07 19:15:56mcdoncsetfiles: + shutdown_typeerror-tip.patch
keywords: + patch
messages: + msg170008
2012-09-07 19:13:35sbtsetnosy: + sbt
2012-09-07 18:56:39mcdonccreate