classification
Title: Mysterious atexit fail
Type: Stage:
Components: Interpreter Core, Library (Lib) Versions: Python 2.7, Python 2.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, sbt, techtonik
Priority: normal Keywords:

Created on 2012-12-18 20:35 by techtonik, last changed 2012-12-20 20:00 by sbt.

Files
File name Uploaded Description Edit
wow.py techtonik, 2012-12-18 20:36
wy.py techtonik, 2012-12-18 20:36
Messages (14)
msg177707 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-18 20:35
Weird bug. Attached are two files: wow.py and wy.py
When wow.py is executed - it fails with single ImportError.
But when wy.py is executed (which is a simple "import wow" statement) it fails with one ImportError and TypeError exceptions.
msg177708 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-18 20:36
...one ImportError and *two* TypeError exceptions.
msg177709 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-18 20:39
>py wy.py
Traceback (most recent call last):
  File "wy.py", line 1, in <module>
    import wow
  File "E:\scons\wow.py", line 12, in <module>
    import fail
ImportError: No module named fail
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Python27\lib\atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "E:\scons\wow.py", line 8, in _clean
    cleanlist = [ c for c in _Cleanup if c ]
TypeError: 'NoneType' object is not iterable
Error in sys.exitfunc:
Traceback (most recent call last):
  File "C:\Python27\lib\atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "E:\scons\wow.py", line 8, in _clean
    cleanlist = [ c for c in _Cleanup if c ]
TypeError: 'NoneType' object is not iterable
msg177710 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-12-18 20:48
It's because _Cleanup is None (as all global variables) when atexit registered functions executed. Don't use globals in such code.
msg177716 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-18 21:19
But why _Cleanup is not None when wow.py is executed standalone?
msg177724 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-12-19 00:10
When you run wy.py the wow module gets partially imported, and then garbage collected because it fails to import successfully.  The destructor for the module replaces values in the module's __dict__ with None.  So when the cleanup function runs you get the unexpected error.

When you run wow.py directly, wow (i.e. the main module) will not be garbage collected, so _Cleanup is never replaced by None.
msg177768 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-19 17:56
This is not repeatable in Python 3. Is it possible to fix it for Python 2 as well?

Is it possible to postpone registration until the import is finished successfully? Or at least give atexit handler a chance to run before global variable stack is purged? Or destroy atexit handler if its namespace is purged?
msg177773 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-12-19 19:38
Infortunately, the behaviour is the same in 2.7 and 3.2, and only changes with 3.3, which means that this is probably related to the new implementation of the import system.
This won't be backported.

[It would be interesting to understand why there is a difference, btw... is a reference to the module held somewhere?]

A possible workaround for such module-level calls atexit.register() is to ensure that used globals are captured in the function namespace, a bit like some __del__ methods:

def _clean(_Cleanup=_Cleanup):
    ....
msg177777 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-12-19 20:23
OK, found the difference between 3.2 and 3.3:
The ImportError exception is still alive when atexit handlers are called, with its __traceback__, and all local variables in Python frames.
And since 3.3 the import system is built with Python functions...

More exactly, I could find the incomplete 'wow' module in sys.last_value.__traceback__.tb_next.tb_frame.f_back.f_back.f_back.f_locals['module']

But all this is not really related to the current issue.
Things should be better in the future, when modules are cleared with true garbage collection.
msg177785 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-12-19 22:13
> Things should be better in the future, when modules are cleared with true 
> garbage collection.

When is this future of which you speak?

I am not sure whether it would affect performance, but a weakrefable subclass of dict could be used for module dicts.  Then the module destructor could just save the module's dict in a WeakValueDictionary keyed by the id (assuming we are not yet shutting down).  At shutdown the saved module dicts could be purged by replacing all values with None.

Or maybe something similar is possible without using a dict subclass.
msg177790 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-12-19 22:59
See issue812369 for the shutdown procedure and modules cleanup.
msg177791 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-12-19 23:06
> See issue812369 for the shutdown procedure and modules cleanup.

I am aware of that issue, but the original patch is 9 years old.  Which is why I ask if/when it will actually happen.
msg177802 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-12-20 08:17
I don't know, you should probably ask there. One blocker is to make builtin and extension modules participate in GC, search "pep3121" to see the many intermediate issues.
msg177858 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-12-20 20:00
Perhaps the simplest thing would be to stop doing anything special when a module is garbage collected: the garbage collector can take care of any orphaned ref-cycles involving the module dict.  Then at shutdown the remaining modules in sys.modules could have their dicts "purged" in the old way.

This would be orthogonal to issue812369.  In fact Armin's original post says that this is a change worth investigating, though his patch does not do it.
History
Date User Action Args
2012-12-20 20:00:25sbtsetmessages: + msg177858
2012-12-20 08:17:28amaury.forgeotdarcsetmessages: + msg177802
2012-12-19 23:06:03sbtsetmessages: + msg177791
2012-12-19 22:59:08amaury.forgeotdarcsetmessages: + msg177790
2012-12-19 22:13:37sbtsetmessages: + msg177785
versions: + Python 2.6
2012-12-19 21:16:59serhiy.storchakasetnosy: - serhiy.storchaka

versions: - Python 2.6
2012-12-19 20:23:22amaury.forgeotdarcsetmessages: + msg177777
2012-12-19 19:38:23amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg177773
2012-12-19 17:56:42techtoniksetmessages: + msg177768
2012-12-19 00:10:39sbtsetnosy: + sbt
messages: + msg177724
2012-12-18 21:19:25techtoniksetmessages: + msg177716
2012-12-18 20:48:26serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg177710
2012-12-18 20:39:47techtoniksetmessages: + msg177709
2012-12-18 20:36:55techtoniksetmessages: + msg177708
2012-12-18 20:36:09techtoniksetfiles: + wy.py
2012-12-18 20:36:01techtoniksetfiles: + wow.py
2012-12-18 20:35:40techtonikcreate