Author vstinner
Recipients Mark.Shannon, scoder, vstinner
Date 2021-10-13.10:04:56
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1634119496.34.0.761308006875.issue40421@roundup.psfhosted.org>
In-reply-to
Content
The coverage project has a ctrace C extension which access PyFrameObject.f_lasti which is gone in Python 3.11. It uses MyFrame_lasti() helper to handle Python 3.10 lasti change:
---
// The f_lasti field changed meaning in 3.10.0a7. It had been bytes, but
// now is instructions, so we need to adjust it to use it as a byte index.
#if PY_VERSION_HEX >= 0x030A00A7
#define MyFrame_lasti(f)    (f->f_lasti * 2)
#else
#define MyFrame_lasti(f)    f->f_lasti
#endif // 3.10.0a7
---

f_lasti is used for two things in coverage/ctracer/tracer.c:

(1) get the last opcode:
----
            /* Need to distinguish between RETURN_VALUE and YIELD_VALUE. Read
             * the current bytecode to see what it is.  In unusual circumstances
             * (Cython code), co_code can be the empty string, so range-check
             * f_lasti before reading the byte.
             */
            int bytecode = RETURN_VALUE;
            PyObject * pCode = MyFrame_GetCode(frame)->co_code;
            int lasti = MyFrame_lasti(frame);

            if (lasti < PyBytes_GET_SIZE(pCode)) {
                bytecode = PyBytes_AS_STRING(pCode)[lasti];
            }
            if (bytecode != YIELD_VALUE) {
                int first = MyFrame_GetCode(frame)->co_firstlineno;
                if (CTracer_record_pair(self, self->pcur_entry->last_line, -first) < 0) {
                    goto error;
                }
            }
----

(2) get the line number, with a special case for generator which is not started yet (lasti < 0)
---
    /* A call event is really a "start frame" event, and can happen for
     * re-entering a generator also.  f_lasti is -1 for a true call, and a
     * real byte offset for a generator re-entry.
     */
    if (frame->f_lasti < 0) {
        self->pcur_entry->last_line = -MyFrame_GetCode(frame)->co_firstlineno;
    }
    else {
        self->pcur_entry->last_line = PyFrame_GetLineNumber(frame);
    }
---

Since Python 3.10.0a3, PyFrame_GetLineNumber() handles the case of negative f_lasti, thanks to the commit 877df851c3ecdb55306840e247596e7b7805a60a related to the PEP 626 implementation:
---
int
PyCode_Addr2Line(PyCodeObject *co, int addrq)
{
    if (addrq < 0) {
        return co->co_firstlineno;
    }
    ...
}
---


=> coverage would need an abstraction to get the last opcode: use case (1).


I recall that an old version of asyncio also had to get latest opcode, in pure Python, to workaround the CPython bpo-21209 bug:

+# Check for CPython issue #21209
+def has_yield_from_bug():
+    class MyGen:
+        def __init__(self):
+            self.send_args = None
+        def __iter__(self):
+            return self
+        def __next__(self):
+            return 42
+        def send(self, *what):
+            self.send_args = what
+            return None
+    def yield_from_gen(gen):
+        yield from gen
+    value = (1, 2, 3)
+    gen = MyGen()
+    coro = yield_from_gen(gen)
+    next(coro)
+    coro.send(value)
+    return gen.send_args != (value,)
+_YIELD_FROM_BUG = has_yield_from_bug()
+del has_yield_from_bug

(...)

+    if _YIELD_FROM_BUG:
+        # For for CPython issue #21209: using "yield from" and a custom
+        # generator, generator.send(tuple) unpacks the tuple instead of passing
+        # the tuple unchanged. Check if the caller is a generator using "yield
+        # from" to decide if the parameter should be unpacked or not.
+        def send(self, *value):
+            frame = sys._getframe()
+            caller = frame.f_back
+            assert caller.f_lasti >= 0
+            if caller.f_code.co_code[caller.f_lasti] != _YIELD_FROM:
+                value = value[0]
+            return self.gen.send(value)
+    else:
+        def send(self, value):
+            return self.gen.send(value)

Hopefully, this code could be be removed from asyncio, since the bug was fixed (and asyncio is now only maintained in the Python stdlib, it's not longer a third party module).
History
Date User Action Args
2021-10-13 10:04:56vstinnersetrecipients: + vstinner, scoder, Mark.Shannon
2021-10-13 10:04:56vstinnersetmessageid: <1634119496.34.0.761308006875.issue40421@roundup.psfhosted.org>
2021-10-13 10:04:56vstinnerlinkissue40421 messages
2021-10-13 10:04:56vstinnercreate