classification
Title: Crash when importing builtin module during interpreter shutdown
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, barry, belopolsky, brett.cannon, christian.heimes, hodgestar, loewis, r.david.murray, vstinner
Priority: critical Keywords: patch

Created on 2008-10-30 00:57 by amaury.forgeotdarc, last changed 2010-12-15 01:36 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
interactive_cleanup.patch vstinner, 2008-10-31 01:24
pymodule_create2.patch vstinner, 2008-10-31 01:44 Remove Py_IsInitialized() from PyModule_Create2()
clarify___del__.diff brett.cannon, 2008-11-19 22:27 Clarify that imports are not guaranteed to work in __del__
a.py loewis, 2008-11-20 15:48
check-import-machinery-only.diff hodgestar, 2010-11-20 11:34 Relax check in PyModule_Create2 to only check that import machinery is still available.
check-import-machinery-only-with-test.diff r.david.murray, 2010-12-14 17:50
Messages (21)
msg75344 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-10-30 00:57
The following interactive session crashes the interpreter on shutdown:

>>> class C:
...   def __del__(self):
...      import warnings
...
>>> C()
<__main__.C object at 0x00C29D58>
>>> ^Z
Fatal Python error: Interpreter not initialized (version mismatch?)


This problem actually happens with the file object, which can call PyErr_WarnEx when it 
is deallocated and closed, see example in issue4226.
This "Fatal" message is new in 3.0. If I ignore it all seems correct, and the module is 
correctly imported.
msg75347 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-10-30 01:03
I can reproduce the same output with CTRL+D on Linux.

$ python3.0
Python 3.0rc1+ (py3k:67029M, Oct 26 2008, 23:48:21)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class C:
...     def __del__(self):
...         print("del")
...         import warnings
...
[33876 refs]
>>> C()
<__main__.C object at 0x7f4740a98d80>
[33893 refs]
>>> exit()
[33900 refs]
del
Fatal Python error: Interpreter not initialized (version mismatch?)
Aborted
msg75402 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-10-31 01:10
The problem occurs in PyImport_Cleanup() on this line:
  PyDict_SetItemString(dict, "_", Py_None);

So the problem is that "_" (special variable in the Python 
interpreter) is removed too late, or that a destructor loads a module 
when Python is exiting.

Call trace:

Py_Finalize() -> PyImport_Cleanup() -> 
builtin___import__("warnings") -> builtin___import__("_warnings") -> 
_PyWarnings_Init() -> PyModule_Create2()

Full backtrace:
(gdb) where
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7d68875 in raise () from /lib/tls/i686/cmov/libc.so.6
#2  0xb7d6a201 in abort () from /lib/tls/i686/cmov/libc.so.6
#3  0x080ba54b in Py_FatalError (msg=0x8163ad0 "Interpreter not 
initialized (version mismatch?)") at Python/pythonrun.c:1922
#4  0x081248e2 in PyModule_Create2 (module=0x8174e20, 
module_api_version=1013) at Objects/moduleobject.c:71
#5  0x080882d7 in _PyWarnings_Init () at Python/_warnings.c:879
#6  0x080abcb0 in init_builtin (name=0xbf8627a3 "_warnings") at 
Python/import.c:1867
#7  0x080aba9f in load_module (name=0xbf8627a3 "_warnings", fp=0x0, 
buf=0xbf8627a3 "_warnings", type=6, loader=0x0) at 
Python/import.c:1795
#8  0x080ad54f in import_submodule (mod=0x81695a4, 
subname=0xbf863833 "_warnings", fullname=0xbf863833 "_warnings") at 
Python/import.c:2567
#9  0x080acd21 in load_next (mod=0x81695a4, altmod=0x81695a4, 
p_name=0xbf863814, buf=0xbf863833 "_warnings", p_buflen=0xbf86381c)
    at Python/import.c:2372
#10 0x080ac26e in import_module_level (name=0x0, globals=0xb7b2802c, 
locals=0xb7b2802c, fromlist=0xb7b208fc, level=0) at 
Python/import.c:2089
#11 0x080ac5b2 in PyImport_ImportModuleLevel 
(name=0xb7b2c650 "_warnings", globals=0xb7b2802c, locals=0xb7b2802c, 
fromlist=0xb7b208fc,
    level=0) at Python/import.c:2140
#12 0x08088b0d in builtin___import__ (self=0xb7be8eac, 
args=0xb7b2098c, kwds=0x0) at Python/bltinmodule.c:173
#13 0x081241c6 in PyCFunction_Call (func=0xb7be8f6c, arg=0xb7b2098c, 
kw=0x0) at Objects/methodobject.c:84
#14 0x080ec5b8 in PyObject_Call (func=0xb7be8f6c, arg=0xb7b2098c, 
kw=0x0) at Objects/abstract.c:2184
#15 0x08094bae in PyEval_CallObjectWithKeywords (func=0xb7be8f6c, 
arg=0xb7b2098c, kw=0x0) at Python/ceval.c:3278
#16 0x080916dd in PyEval_EvalFrameEx (f=0x826661c, throwflag=0) at 
Python/ceval.c:1985
#17 0x08093d64 in PyEval_EvalCodeEx (co=0xb7b25608, 
globals=0xb7b2802c, locals=0xb7b2802c, args=0x0, argcount=0, kws=0x0, 
kwcount=0,
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at 
Python/ceval.c:2835
#18 0x0808d2b4 in PyEval_EvalCode (co=0xb7b25608, globals=0xb7b2802c, 
locals=0xb7b2802c) at Python/ceval.c:519
#19 0x080a9be8 in PyImport_ExecCodeModuleEx 
(name=0xbf866dd3 "warnings", co=0xb7b25608,
    pathname=0xbf864ca3 "/home/haypo/prog/py3k/Lib/warnings.pyc") at 
Python/import.c:729
#20 0x080aa41e in load_source_module (name=0xbf866dd3 "warnings", 
pathname=0xbf864ca3 "/home/haypo/prog/py3k/Lib/warnings.pyc", 
fp=0x82650f8)
    at Python/import.c:1020
#21 0x080aba14 in load_module (name=0xbf866dd3 "warnings", 
fp=0x82650f8, buf=0xbf865d43 "/home/haypo/prog/py3k/Lib/warnings.py", 
type=1,
    loader=0x0) at Python/import.c:1773
#22 0x080ad54f in import_submodule (mod=0x81695a4, 
subname=0xbf866dd3 "warnings", fullname=0xbf866dd3 "warnings") at 
Python/import.c:2567
#23 0x080acd21 in load_next (mod=0x81695a4, altmod=0x81695a4, 
p_name=0xbf866db4, buf=0xbf866dd3 "warnings", p_buflen=0xbf866dbc)
    at Python/import.c:2372
#24 0x080ac26e in import_module_level (name=0x0, globals=0xb7c0402c, 
locals=0x81695a4, fromlist=0x81695a4, level=0) at Python/import.c:2089
#25 0x080ac5b2 in PyImport_ImportModuleLevel 
(name=0xb7b1f850 "warnings", globals=0xb7c0402c, locals=0x81695a4, 
fromlist=0x81695a4, level=0)
    at Python/import.c:2140
#26 0x08088b0d in builtin___import__ (self=0xb7be8eac, 
args=0xb7b87e6c, kwds=0x0) at Python/bltinmodule.c:173
#27 0x081241c6 in PyCFunction_Call (func=0xb7be8f6c, arg=0xb7b87e6c, 
kw=0x0) at Objects/methodobject.c:84
#28 0x080ec5b8 in PyObject_Call (func=0xb7be8f6c, arg=0xb7b87e6c, 
kw=0x0) at Objects/abstract.c:2184
#29 0x08094bae in PyEval_CallObjectWithKeywords (func=0xb7be8f6c, 
arg=0xb7b87e6c, kw=0x0) at Python/ceval.c:3278
#30 0x080916dd in PyEval_EvalFrameEx (f=0x8264fa4, throwflag=0) at 
Python/ceval.c:1985
#31 0x08093d64 in PyEval_EvalCodeEx (co=0xb7b62c80, 
globals=0xb7c0402c, locals=0x0, args=0xb7b1f4d8, argcount=1, kws=0x0, 
kwcount=0,
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at 
Python/ceval.c:2835
---Type <return> to continue, or q <return> to quit---
#32 0x08111b1e in function_call (func=0xb7b2626c, arg=0xb7b1f4cc, 
kw=0x0) at Objects/funcobject.c:628
#33 0x080ec5b8 in PyObject_Call (func=0xb7b2626c, arg=0xb7b1f4cc, 
kw=0x0) at Objects/abstract.c:2184
#34 0x080fe956 in method_call (func=0xb7b2626c, arg=0xb7b1f4cc, 
kw=0x0) at Objects/classobject.c:323
#35 0x080ec5b8 in PyObject_Call (func=0xb7b1f6cc, arg=0xb7bde02c, 
kw=0x0) at Objects/abstract.c:2184
#36 0x08094bae in PyEval_CallObjectWithKeywords (func=0xb7b1f6cc, 
arg=0xb7bde02c, kw=0x0) at Python/ceval.c:3278
#37 0x0806bf2d in slot_tp_del (self=0xb7b2906c) at 
Objects/typeobject.c:5230
#38 0x080613eb in subtype_dealloc (self=0xb7b2906c) at 
Objects/typeobject.c:922
#39 0x0811eb64 in insertdict (mp=0xb7be0acc, key=0xb7c03ac0, 
hash=-724865314, value=0x81695a4) at Objects/dictobject.c:461
#40 0x0811f20d in PyDict_SetItem (op=0xb7be0acc, key=0xb7c03ac0, 
value=0x81695a4) at Objects/dictobject.c:730
#41 0x081218a4 in PyDict_SetItemString (v=0xb7be0acc, 
key=0x8153c82 "_", item=0x81695a4) at Objects/dictobject.c:2092
#42 0x080a902b in PyImport_Cleanup () at Python/import.c:419
#43 0x080b7518 in Py_Finalize () at Python/pythonrun.c:408
#44 0x080c80ad in Py_Main (argc=1, argv=0x81af008) at 
Modules/main.c:639
#45 0x0805a2ab in main (argc=1, argv=0xbf869624) 
at ./Modules/python.c:70
msg75403 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-10-31 01:24
Here is a patch fixing this issue: set builtins._=None at exit.

The new introduced function, PyRun_InteractiveCleanup(), might also be 
called at the end of other functions like PyRun_InteractiveOne() or 
PyRun_InteractiveOneFlags(). If not, PyRun_InteractiveCleanup() should 
be exported.
msg75404 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-10-31 01:44
Another solution: just remove the initialization test from 
PyModule_Create2(). I don't know if it's a safety test or something 
like that.
msg75405 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-10-31 01:46
Python2 is not affected because the bug comes from PyModule_Create2() 
which was introduced in Python3.
msg75409 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-10-31 08:37
Setting builtins._=None is not enough.

My example is very simple, but the crash I had initially was during the
deletion of sys.last_tb: the C() object may be stored anywhere.

I agree that the test in PyModule_Create2 is too strong: the import
machinery is still functional at this point. It could be replaced by
something that becomes true on _PyImport_Init(), and false on
_PyImport_Fini().
msg75482 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-11-04 00:10
Can you add some tests for this problem?
msg75542 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-11-06 03:14
This seems like a corner case to me.  We should fix it before the final
release, but I don't think it's worth holding up 3.0rc2 for it.  Deferring.
msg75576 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-11-06 18:21
PyModule_Create2 might be new, but the warning certainly is not - for
2.x, it lives in Py_InitModule4. Indeed, I can reproduce the problem
with 2.7a0, replacing the import of warnings with an import of imp.
msg75589 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-11-07 01:48
Strange, I can't reproduce the problem with any Python version. Even
py3k doesn't crash with either "import imp" and "import warnings".

$ cat ../issue4236.py
class Crasher(object):
    def __del__(self):
        print("__del__ called")
        import imp

crasher = Crasher()

$ python2.4 issue4236.py
__del__ called
Exception exceptions.ImportError: 'No module named imp' in <bound method
Crasher.__del__ of <__main__.Crasher object at 0x7f0e6c4e44d0>> ignored

$ python2.5 issue4236.py
__del__ called
Exception exceptions.ImportError: 'No module named imp' in <bound method
Crasher.__del__ of <__main__.Crasher object at 0x7fbbde5ef1d0>> ignored

$ python2.6 issue4236.py
__del__ called
Exception ImportError: 'No module named imp' in <bound method
Crasher.__del__ of <__main__.Crasher object at 0x7f3d810dbf90>> ignored

$ py3k/python issue4236.py
__del__ called
Exception ImportError: 'No module named imp' in <bound method
Crasher.__del__ of <__main__.Crasher object at 0x9ab3e0>> ignored
msg75600 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-07 09:12
Christian,
The initial example only works in interactive mode.
To make your script crash, you have to set
    __builtins__._ = Crasher()

There are several other places where Crasher() will crash on shutdown:
     sys.argv.insert(0, Crasher())
     sys.ps1 = Crasher()
     sys.last_type = Crasher()
     sys.path_hooks.append(Crasher())
     sys.path_importer_cache[''] = Crasher()
But
     sys.meta_path.append(Crasher())
does not crash.
msg76075 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-11-19 22:08
So if you look at Python/pythonrun.c, there is a comment from Tim Peters
(from r34776) where he explicitly points out that this is possibility
but that it has never been reported before. Oops. =)
msg76076 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-11-19 22:27
I don't think there is anything to fix here beyond the docs for __del__.
You should never expect anything to be working in __del__, and that
includes the import machinery. It should be bare-bones, not trying to
pull in new code!

I have attached a patch which tries to clarify this fact in the language
docs for __del__.
msg76110 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-11-20 15:48
Here is a test case (a.py) which produces a FatalError back to Python
2.4 at least)
msg76113 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-11-20 16:02
After consultation with MvL and Crys_ on irc, we've agreed that this
should be fixed someday but it's a pathological case that shouldn't hold
up the release.  I'm lowering to critical because I don't think it
should even hold up the final release.
msg110208 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-07-13 15:49
The bug is still present in 2.7, and 3.2.
msg121615 - (view) Author: Simon Cross (hodgestar) Date: 2010-11-20 11:34
I'm attaching a patch to relax the check in PyModule_Create2 as suggested by the Amaury (http://bugs.python.org/issue4236#msg75409).

The patch uses "PyThreadState_Get()->interp->modules == NULL" to determine whether the import machinery has been cleaned up yet.

The test suite still appears to pass.
msg123962 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-12-14 17:50
Brett applied his doc patch in r69092.  Attached is a patch that combines Simon's patch with Martin's test program turned into a unit test.  I confirm that the test suite passes with the patch applied (and fails with just the test applied).

From the text of the original error message, I wonder if there is some way to trigger this error at interpreter startup, and if so whether or not that is tested by the test suite.

Otherwise, is there any reason not to apply this patch?
msg123978 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-12-14 21:21
Nothing jumps to my mind.
msg123998 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-12-15 01:36
Committed to py3k in r87251, 3.1 in r87252, and 2.7 in r87255.
History
Date User Action Args
2010-12-15 01:36:41r.david.murraysetstatus: open -> closed
nosy: loewis, barry, brett.cannon, amaury.forgeotdarc, belopolsky, vstinner, christian.heimes, hodgestar, r.david.murray
messages: + msg123998

resolution: fixed
stage: patch review -> resolved
2010-12-14 21:21:07brett.cannonsetmessages: + msg123978
2010-12-14 17:50:33r.david.murraysetfiles: + check-import-machinery-only-with-test.diff

nosy: + r.david.murray
messages: + msg123962

stage: needs patch -> patch review
2010-11-20 11:34:29hodgestarsetfiles: + check-import-machinery-only.diff
nosy: + hodgestar
messages: + msg121615

2010-08-08 23:08:34terry.reedysetversions: - Python 2.6, Python 3.0
2010-07-13 15:49:44belopolskysetnosy: + belopolsky
messages: + msg110208
2009-05-29 22:24:45r.david.murraysetnosy: loewis, barry, brett.cannon, amaury.forgeotdarc, vstinner, christian.heimes
versions: + Python 2.6, Python 3.1, Python 2.7, Python 3.2
type: crash
components: + Interpreter Core
stage: needs patch
2008-11-24 18:13:44brett.cannonsetassignee: brett.cannon ->
2008-11-20 16:02:55barrysetpriority: release blocker -> critical
assignee: brett.cannon
messages: + msg76113
2008-11-20 15:48:13loewissetfiles: + a.py
messages: + msg76110
2008-11-19 22:27:23brett.cannonsetfiles: + clarify___del__.diff
messages: + msg76076
2008-11-19 22:08:17brett.cannonsetnosy: + brett.cannon
messages: + msg76075
2008-11-07 13:31:21barrysetpriority: deferred blocker -> release blocker
2008-11-07 09:12:15amaury.forgeotdarcsetmessages: + msg75600
2008-11-07 01:48:24christian.heimessetmessages: + msg75589
2008-11-06 18:21:04loewissetmessages: + msg75576
2008-11-06 03:14:12barrysetpriority: release blocker -> deferred blocker
messages: + msg75542
2008-11-04 00:10:15barrysetnosy: + barry
messages: + msg75482
2008-10-31 08:37:41amaury.forgeotdarcsetnosy: + loewis
messages: + msg75409
2008-10-31 01:46:41vstinnersetmessages: + msg75405
2008-10-31 01:44:40vstinnersetfiles: + pymodule_create2.patch
messages: + msg75404
2008-10-31 01:24:26vstinnersetfiles: + interactive_cleanup.patch
keywords: + patch
messages: + msg75403
2008-10-31 01:10:25vstinnersetnosy: + vstinner
messages: + msg75402
2008-10-30 01:03:51christian.heimessetnosy: + christian.heimes
messages: + msg75347
2008-10-30 00:59:13benjamin.petersonlinkissue4226 superseder
2008-10-30 00:57:45amaury.forgeotdarccreate