The code attached is giving the error below

D:\bin\Python251\Lib\threading.py:697: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
return _active[_get_ident()]
Exception exceptions.SystemError: 'error return without exception set' in <generator object at 0x00B07E68> ignored
Ok, I have no clues why there could be an error here...

Bug bpo-1733973 is related.

What is happening in this bug, is that tokenize.tokenize is using generate_tokens to generate the tokens. Since tokeneater() fails while the iteration is occuring, the exception is raised and it unwinds the tokenize() call. This garbage collects the generate_tokens generator -- which throws a GeneratorExit into the generator.

This time, sysmodule.c's trace_trampoline is the one assuming it can overwrite PyErr_*.

Maybe the correct way of fixing the bug is saving+clearing/restoring PyErr_* in ceval.c:call_trace, rather than in each trace/profile handler.

The upside is that they all seem to assume that they can overwrite PyErr_* and this will fix them all.
The downside is that the PyErr_* that's been set in the frame will not be accessible to the trace function. This can be fixed by storing it in some global.

Basically, the fix is to call PyErr_Fetch before the trace/profile func, and PyErr_Restore after it. The problem is that the trace/profile func is a function ptr in the PyThreadState that's used by more than Python's ceval.c. It seems to be liberally used by pyexpat and others. This means that this function pointer should always point at the wrapper function rather than the given function, which means storing more information in the thread state, and then extracting it from the thread state again.

This got more complicated and intricate than I intended, and the whole thing has a wrong feel to it - as it seems to me that all calls to the trace/profile func should be in one centralized place, and that place should be wrapped.

I hope someone else will implement the fix or at least comment on this. If anyone is not clear on what the fix is, please post questions (and if this bug system has mailing notifications) and I'll further explain.

Note that there is already a call_trace_protected in ceval.c which saves/restores the exception,
it just doesn't seem to be called in this instance.

This has chnaged with python-2.5.2.
runing python openpkg under linux centos-5.1,
the test case case4.py don't give any error message anymore, but
still block like with 2.5.1:
# strace -p 937
Process 937 attached - interrupt to quit
futex(0x9b9cc78, FUTEX_WAIT, 0, NULL

Hope this help

The "funny" errors around trace & exceptions have been corrected with
bpo-1265.

The remaining problem is a livelock: when the interpreter exits,
threading.Thread.__delete is called, and this calls
"_active_limbo_lock.acquire".

But the sys.settrace is still enabled... and threading.currentThread()
can also call "_active_limbo_lock.acquire" when the main thread cannot
be found...

A solution could be to clear sys.settrace when closing the interpreter.

The deadlock was resolved by r62136.
I think I was lucky this time. I was close to disable sys.settrace
before interpreter shutdown. But is would disallow debugging inside the
finalization procedures.