The symptom is an exact duplicate of the symptom reported in the following (closed) issue:

http://bugs.python.org/issue9775

The issue is also related to the following other issues:

http://bugs.python.org/issue4106
http://bugs.python.org/issue9205
http://bugs.python.org/issue9207

To reproduce the symptom driving the patches that will be attached to this issue:

  git clone git://github.com/pypa/pip.git
  cd pip
  /any/python setup.py dev
  /any/python setup.py test

You can either wait for the entire test suite to finish or you can press ctrl-C at any time (the tests take a long time).  In either case, a traceback like the following will be printed to the console.

  Error in atexit._run_exitfuncs:
  Traceback (most recent call last):
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
      info('process shutting down')
  TypeError: 'NoneType' object is not callable
  Error in sys.exitfunc:
  Traceback (most recent call last):
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
      func(*targs, **kargs)
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
      info('process shutting down')
  TypeError: 'NoneType' object is not callable

From what I understand in other issues, multiprocessing.util._exit_function shouldn't actually be called *after*  the containing module's globals are destroyed (it should be called before), but this does indeed happen.

Patches will be attached that anticipate the symptom and prevent a shutdown error.  One will be attached for Python 2.7 branch, the other for the Python tip.   Each causes functions that are called at shutdown time to keep a reference around to other functions and globals used within the function, and each does some checks for the insane state and prevents an error from happening in this insane state.

Patch for tip.

2.7 branch patch.

The patch makes sense.  I'll take another look over the weekend, but it seems to be ready to be applied.

+    # NB: we hold on to references to functions in the arglist due to the

This is a nit, but I think adding "NB:", "Note:", etc. to the beginning of a comment is redundant because by being a comment it is already implicit that it should be noted.

New changeset 27d410dd5431 by Alexander Belopolsky in branch '3.2':
Issue #15881: Fixed atexit hook in multiprocessing.
http://hg.python.org/cpython/rev/27d410dd5431

New changeset 08c680918ff8 by Alexander Belopolsky in branch 'default':
Issue #15881: Fixed atexit hook in multiprocessing.
http://hg.python.org/cpython/rev/08c680918ff8

New changeset db67b848ddc3 by Alexander Belopolsky in branch '3.2':
Issue #15881: Fixed 3.2 backport.
http://hg.python.org/cpython/rev/db67b848ddc3

Applied to 3.2 and 3.3.  Thanks for the patch!

Leaving it open pending 2.7 commit.

New changeset b05547e8ff92 by Alexander Belopolsky in branch '3.2':
Issue #15881: Added NEWS entry and proper credit.
http://hg.python.org/cpython/rev/b05547e8ff92

I see the same error on Windows (when pressing ^C), but on Linux I get

Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 28, in _run_exitfuncs
    import traceback
  File "/usr/lib/python2.7/traceback.py", line 3, in <module>
    import linecache
  File "/usr/lib/python2.7/linecache.py", line 9, in <module>
    import os
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'

This also suggests that module teardown has begun before/while sys.exitfunc is running.

I suspect the problem is caused by nose's isolate plugin.

With this enabled, a copy of sys.modules is saved before each test and then restored after the test.  This causes garbage collection of newly imported modules.  The destructor for the module type causes all globals to be replaced by None.

This will break the atexit function registered by multiprocessing since it depends on globals.

PS. A simple work-around (which does not require people to upgrade to a bugfixed version of Python) is to put

    try:
        import multiprocessing
    except ImportError:
        pass

near the beginning of setup.py.  After this change I don't get the error when running "python setup.py test".

Actually, I am not so sure it is the isolate plugin.  But I do think that sys.modules is being manipulated somewhere before shutdown.

Actually it is test.with_project_on_sys_path() in setuptools/commands/test.py that does the save/restore of sys.modules.  See

    http://www.eby-sarna.com/pipermail/peak/2010-May/003357.html

New changeset 2b79b4848f44 by Richard Oudkerk in branch 'default':
Issue #15881: Clarify comment in exit function
http://hg.python.org/cpython/rev/2b79b4848f44

I can reproduce the bug against the 2.7 tip.

Reviewing the 2.7 patch:

- The 2.7 tip has 'Misc/ACKS' instead of 'Doc/ACKS.txt'.

- In 'Misc/ACKS', the line adding 'Chris McDonough' should add it in alpha order.

- The remainder of the patch looks correct, and applies cleanly to the tip of the 2.7 branch.

- After applying the patch to the 2.7 tip, I can no longer reproduce the bug.

- After applying the patch, the full test suite ('./python -m test.testall -j3') passes all tests.

Targeting this for 2.7.4. If Alexander doesn't get to it, ping me and I'll do it

Philip, if you could backport, that'd be great.

New changeset 0a58fa8e9bac by Benjamin Peterson in branch '2.7':
Issue #15881: Fixed atexit hook in multiprocessing.
http://hg.python.org/cpython/rev/0a58fa8e9bac