Author njs
Recipients asvetlov, lemburg, njs, tim.peters, twouters, yselivanov
Date 2020-05-27.03:56:31
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1590551791.79.0.515609150701.issue40789@roundup.psfhosted.org>
In-reply-to
Content
Consider the following short program. demo() is a trivial async function that creates a QObject instance, connects a Python signal, and then exits. When we call `send(None)` on this object, we expect to get a StopIteration exception.

-----

from PySide2 import QtCore

class MyQObject(QtCore.QObject):
    sig = QtCore.Signal()

async def demo():
    myqobject = MyQObject()
    myqobject.sig.connect(lambda: None)
    return 1

coro = demo()
try:
    coro.send(None)
except StopIteration as exc:
    print(f"OK: got {exc!r}")
except SystemError as exc:
    print(f"WTF: got {exc!r}")

-----

Actual output (tested on 3.8.2, but I think the code is present on all versions):

-----
StopIteration: 1
WTF: got SystemError("<method 'send' of 'coroutine' objects> returned NULL without setting an error")
-----

So there are two weird things here: the StopIteration exception is being printed on the console for some reason, and then the actual `send` method is raising SystemError instead of StopIteration.

Here's what I think is happening:

In genobject.c:gen_send_ex, when the coroutine finishes, we call _PyGen_SetStopIterationValue to raise the StopIteration exception:

https://github.com/python/cpython/blob/404b23b85b17c84e022779f31fc89cb0ed0d37e8/Objects/genobject.c#L241

Then, after that, gen_send_ex clears the frame object and drops references to it:

https://github.com/python/cpython/blob/404b23b85b17c84e022779f31fc89cb0ed0d37e8/Objects/genobject.c#L266-L273

At this point, the reference count for `myqobject` drops to zero, so its destructor is invoked. And this destructor ends up clearing the current exception again. Here's a stack trace:

-----
#0  0x0000000000677eb7 in _PyErr_Fetch (p_traceback=0x7ffd9fda77d0, 
    p_value=0x7ffd9fda77d8, p_type=0x7ffd9fda77e0, tstate=0x2511280)
    at ../Python/errors.c:399
#1  _PyErr_PrintEx (tstate=0x2511280, set_sys_last_vars=1) at ../Python/pythonrun.c:670
#2  0x00007f1afb455967 in PySide::GlobalReceiverV2::qt_metacall(QMetaObject::Call, int, void**) ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/PySide2/libpyside2.abi3.so.5.14
#3  0x00007f1afaf2f657 in void doActivate<false>(QObject*, int, void**) ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/PySide2/Qt/lib/libQt5Core.so.5
#4  0x00007f1afaf2a37f in QObject::destroyed(QObject*) ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/PySide2/Qt/lib/libQt5Core.so.5
#5  0x00007f1afaf2d742 in QObject::~QObject() ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/PySide2/Qt/lib/libQt5Core.so.5
#6  0x00007f1afb852681 in QObjectWrapper::~QObjectWrapper() ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/PySide2/QtCore.abi3.so
#7  0x00007f1afbf785bb in SbkDeallocWrapperCommon ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/shiboken2/libshiboken2.abi3.so.5.14
#8  0x00000000005a4fbc in subtype_dealloc (self=<optimized out>)
    at ../Objects/typeobject.c:1289
#9  0x00000000005e8c08 in _Py_Dealloc (op=<optimized out>) at ../Objects/object.c:2215
#10 _Py_DECREF (filename=0x881795 "../Objects/frameobject.c", lineno=430, 
    op=<optimized out>) at ../Include/object.h:478
#11 frame_dealloc (f=Frame 0x7f1afc572dd0, for file qget-min.py, line 12, in demo ())
    at ../Objects/frameobject.c:430
#12 0x00000000004fdf30 in _Py_Dealloc (
    op=Frame 0x7f1afc572dd0, for file qget-min.py, line 12, in demo ())
    at ../Objects/object.c:2215
#13 _Py_DECREF (filename=<synthetic pointer>, lineno=279, 
    op=Frame 0x7f1afc572dd0, for file qget-min.py, line 12, in demo ())
    at ../Include/object.h:478
#14 gen_send_ex (gen=0x7f1afbd08440, arg=<optimized out>, exc=<optimized out>, 
    closing=<optimized out>) at ../Objects/genobject.c:279

------

We can read the source for PySide::GlobalReceiverV2::qt_metacall here: https://sources.debian.org/src/pyside2/5.13.2-3/sources/pyside2/libpyside/globalreceiverv2.cpp/?hl=310#L310

And we see that it (potentially) runs some arbitrary Python code, and then handles any exceptions by doing:

if (PyErr_Occurred()) {
    PyErr_Print();
}

This is intended to catch exceptions caused by the code it just executed, but in this case, gen_send_ex ends up invoking it with an exception already active, so PySide2 gets confused and clears the StopIteration.

-----------------------------------

OK so... what to do. I'm actually not 100% certain whether this is a CPython bug or a PySide2 bug.

In PySide2, it could be worked around by saving the exception state before executing that code, and then restoring it afterwards.

In gen_send_ex, it could be worked around by dropping the reference to the frame before setting the StopIteration exception.

In CPython in general, it could be worked around by not invoking deallocators with a live exception... I'm actually pretty surprised that this is even possible! It seems like having a live exception when you start executing arbitrary Python code would be bad. So maybe that's the real bug? Adding both "asyncio" and "memory management" interest groups to the nosy.
History
Date User Action Args
2020-05-27 03:56:31njssetrecipients: + njs, lemburg, tim.peters, twouters, asvetlov, yselivanov
2020-05-27 03:56:31njssetmessageid: <1590551791.79.0.515609150701.issue40789@roundup.psfhosted.org>
2020-05-27 03:56:31njslinkissue40789 messages
2020-05-27 03:56:31njscreate