After few seconds (30 sec to 5 minutes), my program (Fusil) crashs at:

PyEval_EvalFrameEx (f=0x85b4324, throwflag=0) at Python/ceval.c:2459
    Py_CLEAR(tstate->exc_traceback);

It crashs because tstate->exc_traceback points to a zombi object:
{_ob_next = 0xdbdbdbdb, _ob_prev = 0xdbdbdbdb, ob_refcnt = -606348326, 
ob_type = 0xdbdbdbdb}

(refcnt is 0xdbdbdbdb-1)

I'm using py3k rev 65882 compiled with CFLAGS "-O0 -ggdb" 
and --with-pydebug. Sorry, I don't have more informations yet and I 
can't explain how to reproduce the bug :-/

The crash is on ProcError exception raising (line 21):

    20      except IOError as err:
    21          raise ProcError("Unable to open %r: %s" % (filename, 

---

(gdb) where
#0  0x0805bcc7 in PyObject_Hash (v=0x8691034) at Objects/object.c:796
#1  0x08062125 in set_contains_key (so=0x856ec6c, key=0x8691034) at 
Objects/setobject.c:682
#2  0x0806659f in PySet_Contains (anyset=0x856ec6c, key=0x8691034) at 
Objects/setobject.c:2263
#3  0x080def3a in print_exception_recursive (f=0xb7bea654, 
value=0x8656814, seen=0x856ec6c) at Python/pythonrun.c:1432
#4  0x080df0ea in PyErr_Display (exception=0x84542ac, value=0x8656814, 
tb=0x865bc34) at Python/pythonrun.c:1470
#5  0x080e68f6 in sys_excepthook (self=0xb7e013b4, args=0x865b334) at 
Python/sysmodule.c:119
#6  0x08161d29 in PyCFunction_Call (func=0xb7e016c4, arg=0x865b334, 
kw=0x0) at Objects/methodobject.c:81
#7  0x08118cc5 in PyObject_Call (func=0xb7e016c4, arg=0x865b334, 
kw=0x0) at Objects/abstract.c:2181
#8  0x080b2ea8 in PyEval_CallObjectWithKeywords (func=0xb7e016c4, 
arg=0x865b334, kw=0x0) at Python/ceval.c:3283
#9  0x080de463 in PyErr_PrintEx (set_sys_last_vars=1) at 
Python/pythonrun.c:1258
#10 0x080de06f in PyErr_Print () at Python/pythonrun.c:1150
#11 0x080e0007 in Py_FatalError (msg=0xbf9a1498 "Python/ceval.c:2459 
object at 0x865b6b4 has negative ref count -606348326")
    at Python/pythonrun.c:1863
#12 0x0805a821 in _Py_NegativeRefcount 
(fname=0x8197098 "Python/ceval.c", lineno=2459, op=0x865b6b4) at 
Objects/object.c:194
#13 0x080b004f in PyEval_EvalFrameEx (f=0x85b4324, throwflag=0) at 
Python/ceval.c:2459
(...)

(gdb) frame 13
#13 0x080b004f in PyEval_EvalFrameEx (f=0x85c9794, throwflag=0) at 
Python/ceval.c:2459
2459                                            
Py_CLEAR(tstate->exc_traceback);

(gdb) pystack
/home/haypo/ptrace/ptrace/linux_proc.py (21): openProc
/home/haypo/ptrace/ptrace/linux_proc.py (24): readProc
/home/haypo/ptrace/ptrace/linux_proc.py (31): readProcessProc
/home/haypo/ptrace/ptrace/linux_proc.py (74): readProcessStat
(...)

---

linux_proc.py code near line 21:
    (...)
     7  class ProcError(Exception):
     8      pass
    (...)
    16  def openProc(path):
    17      try:
    18          filename = procFilename(path)
    19          return open(filename)
    20      except IOError as err:
    21          raise ProcError("Unable to open %r: %s" % (filename, 
err))
    (...)
    29  def readProcessProc(pid, key):
    30      try:
    31          return readProc(path_join(str(pid), str(key)))
    32      except ProcError as error:
    33          raise ProcError("Process %s doesn't exist: %s" % (
    34              pid, error))
    (...)
    73  def readProcessStat(pid):
    74      stat = readProcessProc(pid, 'stat')
    75      return ProcessState(stat)
    76

I also noticed a crash in PyErr_SetObject(): in block <line 68..86>, 
tstate->exc_value value may changes and so it's possible that 
tstate->exc_value becomes NULL. I added a test to avoid this crash:

Index: Python/errors.c
===================================================================
--- Python/errors.c     (révision 65899)
+++ Python/errors.c     (copie de travail)
@@ -88,7 +88,7 @@
                   This is O(chain length) but context chains are
                   usually very short. Sensitive readers may try
                   to inline the call to PyException_GetContext. */
-               if (tstate->exc_value != value) {
+               if (tstate->exc_value != value && tstate->exc_value != 
NULL) {
                        PyObject *o = tstate->exc_value, *context;
                        while ((context = PyException_GetContext(o))) 
{
                                Py_DECREF(context);

> I also noticed a crash in PyErr_SetObject(): in block <line 68..86>, 
> tstate->exc_value value may changes and so it's possible that 
> tstate->exc_value becomes NULL. 

How can tstate->exc_value become NULL at that point? Could you investigate?

> How can tstate->exc_value become NULL at that point? [error.c:86]
> Could you investigate?

PyEval_CallObject(exception, args) may calls PyErr_SetObject(). Since 
the same thread state is the same, tstate->exc_value also changes 
during this call.

Selon STINNER Victor <report@bugs.python.org>:
>
> > How can tstate->exc_value become NULL at that point? [error.c:86]
> > Could you investigate?
>
> PyEval_CallObject(exception, args) may calls PyErr_SetObject(). Since
> the same thread state is the same, tstate->exc_value also changes
> during this call.

Hmm, indeed.
However, I don't see why your Python code triggers that. Does instantiating the
ProcError object somehow fail, and for what reason? Or did you witness it in
other circumstances?

As for the original problem ("tstate->exc_traceback points to a zombi object")
it would be nice to have a small snippet of Python code to reproduce it. Is it
possible for you to do that?

I retract what I said above. PyErr_SetObject() and friends only change
tstate->curexc_* variables, not tstate->exc_*.

The former (tstate->curexc_*) contain the current, pending (uncaught)
exception state of the thread. The latter (tstate->exc_*) contain the
exception currently handled by an "except" statement in Python code. 

So, theoretically, there is no way calling PyEval_CallObject() can
change tstate->exc_value:
- if it raises an exception or calls code that raises an exception
without catching it, the exception ends up in tstate->curexc_*, not in
tstate->exc_*
- if it calls code that raises an exception /and catches it/, exception
stacking means the exception should be discarded after the end of the
"except" block and the last tstate->exc_* values restored.

Of course, there may be bugs in the implementation of the above. The
best way to find out would be to have a small snippet of Python code
reproducing the problem. :-)

About the PyEval_CallObject() call in errors.c, here is an example:
   Call exception<0x81dcee0>(args<0x8751dc4>)
with exception=
   object  : <class 'AttributeError'>
   type    : type
   refcount: 6
   address : 0x81dcee0
and args=
   object  : ("type object 'ProcError' has no 
attribute '__subclasscheck__'",)
   type    : tuple
   refcount: 1
   address : 0x8751dc4

This exception may comes from PyObject_IsSubclass() which calls 
PyObject_GetAttr(cls, "__subclasscheck__").

[...]
>
> This exception may comes from PyObject_IsSubclass() which calls
> PyObject_GetAttr(cls, "__subclasscheck__").

If you look at the code for PyObject_IsSubclass(), there is a pair of
PyErr_Fetch / PyErr_Restore calls around PyObject_GetAttr(cls,
"__subclasscheck__"). So this exception should vanish immediately rather than
stay around.

I'm unable to write a short Python script to reproduce the bug. So 
download the full program:
   http://neudorf.hachoir.org/tmp/fusil3000.tar.gz

Run in with:
   $ gdb python3.0
   (gdb) run fusil-python --fast

The problem comes when PyErr_SetObject triggers garbage collection which
runs python code (finalizers...).

I could reproduce the crash several times, and each time garbage
collection was triggered by the normalization of the exception (the call
to PyEval_CallObject(exception, args)).

An obvious fix is to save exc_value near the "/* Implicit exception
chaining */" comment.

However, I fear that there are other places where exc_value is reset,
and "exception chaining" may break in subtle ways.

> The problem comes when PyErr_SetObject triggers garbage collection which
> runs python code (finalizers...).

Mmmh, normally this shouldn't change the value of tstate->exc_value once
that Python code returns. That's what exception stacking is for.

Having a snippet deterministically reproducing the problem would really
help in any case.

> An obvious fix is to save exc_value near the "/* Implicit exception
> chaining */" comment.

Well, it may be a fix for the crash but I'm not sure it makes the
semantics correct. If tstate->exc_value was really changed, there is a
reason and I'm not sure it should be ignored.

> Having a snippet deterministically reproducing the problem 
> would really help in any case.
Here you are. Don't ask how I found this.

The attached script, when run, prints

(KeyError(), ValueError())
(KeyError(), None)

The current exception context (tstate->exc_value) is lost when a
generator is deleted, if it was paused inside a "try" block.

This is the cause of the crash: the gc runs inside PyErr_SetObject and
collects such a generator.

It looks that the problem is that PyErr_SetObject() is not re-entrant.
The crash occurs when PyErr_SetObject() is called (indirectly) by 
PyErr_SetObject(): tstate->exc_value value is changed (set to NULL).

As noticed by amaury.forgeotdarc, the problem is linked to be garbage 
collector: the crash occurs when the garbage collector calls a 
destructor which will reuse PyErr_SetObject().

Great job amaury! So in ceval, here is the block responsible to clear 
the execution informations:

Index: Python/ceval.c
===================================================================
--- Python/ceval.c      (révision 65915)
+++ Python/ceval.c      (copie de travail)
@@ -2453,11 +2453,6 @@

                        if (b->b_type == EXCEPT_HANDLER) {
                                UNWIND_EXCEPT_HANDLER(b);
-                               if (why == WHY_EXCEPTION) {
-                                       Py_CLEAR(tstate->exc_type);
-                                       Py_CLEAR(tstate->exc_value);
-                                       
Py_CLEAR(tstate->exc_traceback);
-                               }
                                continue;
                        }
                        UNWIND_BLOCK(b);

Without these 5 lines, the bug disappears and it looks (i tried few 
tests) that CPython is still ok.

Unfortunately, removing those lines causes a RuntimeError about
exceeding the recursion limit for about 1/3 of the tests.

Can you try with the following patch,
"throwflag" is specific to a generator being deleted.

Index: Python/ceval.c
===================================================================
--- Python/ceval.c      (revision 65919)
+++ Python/ceval.c      (working copy)
@@ -2453,7 +2453,7 @@

                        if (b->b_type == EXCEPT_HANDLER) {
                                UNWIND_EXCEPT_HANDLER(b);
-                               if (why == WHY_EXCEPTION) {
+                               if (why == WHY_EXCEPTION && !throwflag) {
                                        Py_CLEAR(tstate->exc_type);
                                        Py_CLEAR(tstate->exc_value);
                                        Py_CLEAR(tstate->exc_traceback);

And if it doesn't work, try the following instead:

if (why == WHY_EXCEPTION) {
	PyObject *tmp1, *tmp2, *tmp3;
	tmp1 = tstate->exc_type;
	tmp2 = tstate->exc_value;
	tmp3 = tstate->exc_traceback;
	tstate->exc_type = NULL;
	tstate->exc_value = NULL;
	tstate->exc_traceback = NULL;
	Py_XDECREF(tmp1);
	Py_XDECREF(tmp2);
	Py_XDECREF(tmp3);
}

This last proposal does not correct the behaviour of lostcontext.py.

Here's Amaury's patch and a unittest based on lostcontext.py.

Short patch review: __context__ tests are mostly found in test_raise.py
(class TestContext)

test_exceptions.py seems to only deal with the exception object.

Patch was applied in r65921.

Ok. I'll move the test after the betas.

The bug is not closed :-/ With py3k trunk, I still get a crash. So I 
added a flag to detect inner calls. Here is an example of inner call 
backtrace:

(gdb) where
(...)
#2  0xb7df4201 in abort () from /lib/tls/i686/cmov/libc.so.6
#3  0x080a23a9 in PyErr_SetObject (exception=0x818b760, 
value=0xb75a3694) at Python/errors.c:61
#4  0x08094048 in do_raise (exc=0xb75a3694, cause=0x0) at 
Python/ceval.c:2928
#5  0x0808fa70 in PyEval_EvalFrameEx (f=0x83557d4, throwflag=0) at 
Python/ceval.c:1510
(...)
#23 0x08094bd8 in PyEval_CallObjectWithKeywords (func=0xb7a6c96c, 
arg=0xb7c6802c, kw=0x0) at Python/ceval.c:3283
#24 0x0806befd in slot_tp_del (self=0xb759d44c) at 
Objects/typeobject.c:5268
(...)
#26 0x0811f05d in PyDict_Clear (op=0xb7595e84) at 
Objects/dictobject.c:846
#27 0x08121112 in dict_tp_clear (op=0xb7595e84) at 
Objects/dictobject.c:1841
#28 0x080c81e9 in delete_garbage (collectable=0xbfcc75c4, 
old=0x8179434) at Modules/gcmodule.c:684
(...)
#31 0x080c9000 in _PyObject_GC_Malloc (basicsize=28) at 
Modules/gcmodule.c:1333
#32 0x08060aee in PyType_GenericAlloc (type=0x818b0a0, nitems=0) at 
Objects/typeobject.c:667
(...)
#37 0x080a24cc in PyErr_SetObject (exception=0x818b0a0, 
value=0xb75acea0) at Python/errors.c:87
(...)
#44 0x080912f7 in PyEval_EvalFrameEx (f=0x83660e4, throwflag=0) at 
Python/ceval.c:1940
(...)

First Python stack:

(gdb) pystack
/home/haypo/prog/py3k/Lib/io.py (1074): _flush_unlocked
/home/haypo/prog/py3k/Lib/io.py (1070): flush
/home/haypo/prog/py3k/Lib/io.py (1454): flush
/home/haypo/prog/py3k/Lib/io.py (1459): close
/home/haypo/prog/py3k/Lib/io.py (390): __del__
/home/haypo/fusil3000/fusil/process/cpu_probe.py (30): live
/home/haypo/fusil3000/fusil/mas/univers.py (15): executeAgent
(...)

Second Python stack:

(gdb) pystack
/home/haypo/fusil3000/fusil/process/cpu_probe.py (30): live
/home/haypo/fusil3000/fusil/mas/univers.py (15): executeAgent
/home/haypo/fusil3000/fusil/mas/univers.py (24): execute
/home/haypo/fusil3000/fusil/application.py (196): executeProject
(...)

So the real fix is to make PyErr_SetObject() re-entrant.

On a Windows box, I manage to make the following script reliably hang on
a non-debug build of beta3. This can be a good base for further diagnosing.


def f():
    class Bug:
        def __del__(self):
            1/0

    import gc
    trash = [Bug()]
    trash.append(trash)
    try:
        gc.collect()
        gc.set_threshold(1, 1, 1)
        del trash
        len()
    except TypeError:
        raise

if __name__ == "__main__":
    f()

Here is a new snippet with strange exception handling:
--------------------- 8< -------------------------
from gc import collect
import _weakref

class FuzzingUserClass:
    pass

obj = _weakref.ref(FuzzingUserClass)

# Exception not raised??
obj.__init__(
    0,
    0,
    0,
)

# Exception catched here??
collect()
--------------------- 8< -------------------------

Result:
    Exception TypeError: '__init__ expected at most 2 arguments, 
    got 3' in 'garbage collection' ignored
    Fatal Python error: unexpected exception during garbage collection
    Abandon (core dumped)

The exception is raised in Objects/weakrefobject.c:
  weakref___init__() 
  => parse_weakref_init_args() 
  => PyArg_UnpackTuple() *here*

Haypo, this is a separate bug I think. Please open a new ticket :)

@pitrou: Ok, done (issue #3634). We were right, it's a different bug.

Antoine, your script hangs at the end due to the io.py deadlock (see #3618)

- at the end of the script, flush_io() is called
- this enters code in io.py
- here, garbage collection occurs (thanks to low thresholds)
- the Bug() instance is collected
- the exception is handled by a PyErr_WriteUnraisable
- which tries to print

I found another way to loose the current exception context; see
lostcontext2.py

Completely removing the block starting with
       if (why == WHY_EXCEPTION && !throwflag) {
corrects the script;

but according to Benjamin: """removing those lines causes a RuntimeError
about exceeding the recursion limit for about 1/3 of the tests."""

So a better fix must be found. At least we have the unit test :-).

It's now known that PyErr_SetObject() have to be re-entrant because of 
the garbage collector interaction. As I wrote in my comments, tstate 
may be changed during PyEval_CallObject() call. The problem is to 
known which values have to be protected during PyErr_SetObject() 
re-entrant call... I tried to save/restore tstate->exc_value if it's 
value changes, but I'm not sure that it's enough.

Ok, it was not enough: exc_{type,value,traceback} have to be 
saved/restored. So here is a new patch. I doesn't change Python 
behaviour for previous pitrou snippet (msg71658): it doesn't crash 
here and display two errors.

I think that this issue mix differents bugs:
 (1) lostcontext.py fixed by « if (why == WHY_EXCEPTION 
&& !throwflag) »
 (2) pitrou snippet
 (3) PyErr_SetObject() is no re-entrant: not fixed yet (please review 
my new patch attached to this comment ;-))

With my last patch (pyerr_setobject_reentrant-v2.patch) my program 
works fine! I known that PyErr_SetObject() makes re-entrant calls 
every ~30 seconds, so the code is tested ;-)

Victor, your patch addresses the symptom of the problem, not the cause.

The cause is that in some cases, the exception context chain is lost. Of
course what could be a minor annoyance becomes dramatic when this
precisely happens in code that makes use of this context.

IMO, PyErr_SetObject should not restore the previous exc_* members; it's
not its job to correct others' mistakes.
I join a patch that only protects it from crashing in this case: it just
saves exc_value in a local variable.

The bug shown by lostcontext2.py is still unresolved, but at least it
won't crash the interpreter.

Agreed with Amaury, it's not PyErr_SetObject's job to try to
save/restore the tstate->exc_* variables. We'll probably have to live
with the small context-losing glitches in 3.0.

For 3.1, a radical solution would be to drop the "exception
normalization" misfeature (which is a source of complications and
potential bugs) and always instantiate exception objects as soon as they
are raised rather than lazily. We need to make sure the performance loss
is reasonable though.

Hmm, answering to myself, I don't think dropping exception normalization
would solve the problem, it would just let it occur in a different place...

Benjamin,

I am testing the patch proposed earlier by Victor in
http://bugs.python.org/msg71579 (who might be the correct one, after all),
and I don't see the problems you mentioned. 
I run the test suite without problem, on winXP & Linux Debian x86.

Can you please try again?

I can't reproduce it either. I suspect that I removed the wrong lines
initially and caused other problems. :) Sorry for the false alarm.

Some progress:
The lines suppressed by the patch at http://bugs.python.org/msg71579
either do nothing (because e.g exc_type is already NULL or None),
or happen to be in a case similar to the script "lostcontext2.py"
(Most of the time, the flush() function in io.py).

So again, I think these lines should be suppressed.
The test suite still pass, and all the "-R::" I ran did return identical
result.
Can this patch go in?

> Can this patch go in?

I'm ok for it.

Committed as r66056.