We have a crash in our product when tracing is enabled by
sys.settrace() and threading.settrace(). If a Python generator is
created in a C thread, calling the generator later in another thread
may crash if Python tracing is enabled.

 - the C thread calls PyGILState_Ensure() which creates a temporary
Python thread state
 - a generator is created, the generator has a reference to a Python
frame which keeps a reference to the temporary Python thread state
 - the C thread calls PyGILState_Releases() which destroys the
temporary Python thread state
 - when the generator is called later in another thread, call_trace()
reads the Python thread state from the generator frame, which is the
destroyed frame => it does crash on a pointer dereference if the
memory was reallocated (by malloc()) and the data were erased

To reproduce the crash, unpack the attached
generator_frame_bug.tar.gz, compile the C module using:

   python setup.py build

and then run:

   PYTHONPATH=$(ls -d build/lib*/) python test.py

(or just "python test.py if you installed the _test module).
You may need to use Valgrind to see the error, or call memset(tstate,
0xFF, sizeof(*tstate)) before free(tstate) in tstate_delete_common().

Calling the generator should update its reference to the Python state
thread in its frame. The generator may also clears frame->f_tstate (to
detect bugs earlier), as it does for frame->f_back (to avoid a
reference cycle). Attached patch implements this fix for Python 3.3.

Here's the patch ;-)

The proposed fix sounds reasonable to me. Would it be possible to work something into test_capi to actually test it?

It may not even have to specifically test the crash - any operation that accessed the tstate on the frame and could be shown to be accessing the wrong thread state when called from another thread could demonstrate the problem.

For what it's worth, I think I've seen this bug in 2.6 and 2.5, using generators created in python threads, while profiling not tracing.

I'm creating generators in one python thread and storing them in a variable.  In a different python thread I overwrite the variable, which garbage collects the generator.  Sometimes it causes an interpreter crash.  Other times, it seems to trigger a return event in the profiler, but it's associated with an incorrect thread.  The thread in question is often (always?) in the profiler code, so it looks like the profiler is profiling itself.

Rather than ensuring that f_tstate always points to the current frame,
just remove it altogether.

Patch attached

I think I've tripped over a variation on this theme using pyqt and 2.7:

When working in a QThread, the PyGILState_Ensure call when transitioning control from Qt to python will frequently allocate a new thread state - because every time control returns to the Qt event loop, gilstate_counter is likely to become zero.

If a generator is kept around longer than this, it becomes quite likely to have an older thread state in f_tstate. This happens all the time.

The access that blows up on me is PyEval_GetRestricted, since PyFrame_IsRestricted checks f_tstate. Annoyingly this debris is still called on the possibly-restricted operations even when restricted execution is nowhere in sight.

Hence, any use of open() in a long-lived generator in a QThread is probably going to crash.

Patch against 2.7?