This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: code.replace(co_code=new_code) no longer catch exceptions on Python 3.11
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.11
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, dom1310df, gregory.p.smith, gvanrossum, vstinner
Priority: normal Keywords:

Created on 2022-04-01 09:31 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (10)
msg416479 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-04-01 09:31
Since bpo-40222 "Zero cost exception handling", code object created by from bytecode with code.replace(co_code=new_code) no longer catch exceptions on Python 3.11, unless an exception table is set explicitly.

Example:
---
def f():
    try:
        print("raise")
        raise ValueError
    except ValueError:
        print("except")
    else:
        print("else")
    print("exit func")

def g(): pass

if 1:
    code = f.__code__
    g.__code__ = g.__code__.replace(
        co_code=code.co_code,
        co_consts=code.co_consts,
        co_names=code.co_names,
        co_flags=code.co_flags,
        co_stacksize=code.co_stacksize)
else:
    g.__code__ = f.__code__  # this code path works on Python 3.11

g()
---

Output with Python 3.10 (ok):
---
raise
except
exit func
---

Output with Python 3.11 (oops):
---
raise
Traceback (most recent call last):
  ...
ValueError
---

Would it make sense to automatically compute co_exceptiontable on code.replace(co_code=new_code)? If it's computed automatically, I don't know if it makes still sense to call code.replace(co_code=new_code, co_exceptiontable=new_table).

It seems like currently, the only implementation to build an exception table lives in Python/compile.c which compiles AST to bytecode. It cannot be reused for code.replace() which takes bytecode as input, not AST.

--

If code.replace() is not updated to recompute co_exceptiontable, at least, it would be nice to document the bpo-40222 changes in What's New in Python 3.11 and in the CodeType documentation:

* https://docs.python.org/dev/library/types.html#types.CodeType
* https://docs.python.org/dev/whatsnew/3.11.html
msg416499 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-04-01 16:24
How would you compute the exception table from the bytecode? There are no clues in the bytecode about where the try and except blocks are.
msg416501 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-04-01 17:47
> How would you compute the exception table from the bytecode? There are no clues in the bytecode about where the try and except blocks are.

Disassemble the bytecode to rebuild basic blocks and detect which ones are except blocks. I don't know how the exception table works :-) It's just a guess.

Or do you think that this job should be done by the caller?
msg416503 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-04-01 18:05
python-dev thread:
https://mail.python.org/archives/list/python-dev@python.org/thread/KWSPCLXDHBAP2U4LBSMLQEOC7LREDMPB/

Mark wrote:

"You can pass the exception table the same way you pass all the other arguments. The exception table depends on the code, but that is nothing new. The bytecode library already recomputes the consts, names, etc."

Constants and names are easy to build, it's just an array and the bytecode refers to their index.

Building the exception table is more complicated. It's nice that the format is documented in https://github.com/python/cpython/blob/main/Objects/exception_handling_notes.txt but it would be more convenient to put it in the regular Python documentation (docs.python.org), no? I discovered that file by mistake with filename completion in my editor while looking for Objects/exceptions.c :-)
msg416518 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-04-01 20:38
Guido (msg416498)
> Surely the bigger issue is that the contents of new_code itself must be totally different? Also there are other tables that need to be adjusted if you really do change co_code, e.g. the debugging tables.

Do you consider that .replace() must reject changing co_code if other tables are not updated?

Debugging tables are not strictly required just to *execute* code, no?

If you consider that the caller *must* update co_exceptiontable, replace() must raise an exception in this case, to prevent creating a code object which would behave in a strange way (broken exception handling).

If someone really wants testing an empty exception table just for fun, it would still be possible to pass co_exceptiontable=b''.

My concern is more about people upgrading to Python 3.11 and who "suddenly" don't get their exceptions handled anymore. I would prefer catching such bug at the replace() call, rather than having to execute the code (and only notice the bug in production? oops).
msg416637 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-04-03 18:28
[Victor]
> Do you consider that .replace() must reject changing co_code if other tables are not updated?

I simply don't believe it can always do that correctly, so I believe it should not do it.

> Debugging tables are not strictly required just to *execute* code, no?

No, but if they are wrong crashes might happen when they are consulted. At the very least they would confuse users.

> If you consider that the caller *must* update co_exceptiontable, replace() must raise an exception in this case, to prevent creating a code object which would behave in a strange way (broken exception handling).

No. There are a zillion use cases. If you are using .replace() you are taking responsibility for the result.

> If someone really wants testing an empty exception table just for fun, it would still be possible to pass co_exceptiontable=b''.

> My concern is more about people upgrading to Python 3.11 and who "suddenly" don't get their exceptions handled anymore. I would prefer catching such bug at the replace() call, rather than having to execute the code (and only notice the bug in production? oops).

Where would these people get the value that they're using to replace co_code? Surely if they are generating bytecode it will already be broken. Pretty much all instructions are different in 3.11.
msg416825 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-04-05 22:35
This idea just cannot work. Take these two functions:

def f():
    foo()
    try:
        bar()
    except:
        pass

def g():
    try:
        foo()
        bar()
    except:
        pass

Using dis to look at their disassembly, the only hint that in f(), the call to foo() is outside the try block and in g() it is inside it is the presence of some NOP opcodes. The actual demarcation of where the try blocks start and end is exclusively determined by the exception table.

It just doesn't make sense to try to validate that correct parameters are being passed in when you are modifying co_code and friends.
msg416827 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-04-05 22:59
>>> def f():
...     foo()
...     try:
...         bar()
...     except:
...         pass
... 
>>> def g():
...     try:
...         foo()
...         bar()
...     except:
...         pass
... 

>>> dis.dis(f)
  1           0 RESUME                   0

  2           2 LOAD_GLOBAL              1 (NULL + foo)
             14 PRECALL                  0
             18 CALL                     0
             28 POP_TOP

  3          30 NOP

  4          32 LOAD_GLOBAL              3 (NULL + bar)
             44 PRECALL                  0
             48 CALL                     0
             58 POP_TOP
             60 LOAD_CONST               0 (None)
             62 RETURN_VALUE
        >>   64 PUSH_EXC_INFO

  5          66 POP_TOP

  6          68 POP_EXCEPT
             70 LOAD_CONST               0 (None)
             72 RETURN_VALUE
        >>   74 COPY                     3
             76 POP_EXCEPT
             78 RERAISE                  1
ExceptionTable:
  32 to 58 -> 64 [0]
  64 to 66 -> 74 [1] lasti

>>> dis.dis(g)
  1           0 RESUME                   0

  2           2 NOP

  3           4 LOAD_GLOBAL              1 (NULL + foo)
             16 PRECALL                  0
             20 CALL                     0
             30 POP_TOP

  4          32 LOAD_GLOBAL              3 (NULL + bar)
             44 PRECALL                  0
             48 CALL                     0
             58 POP_TOP
             60 LOAD_CONST               0 (None)
             62 RETURN_VALUE
        >>   64 PUSH_EXC_INFO

  5          66 POP_TOP

  6          68 POP_EXCEPT
             70 LOAD_CONST               0 (None)
             72 RETURN_VALUE
        >>   74 COPY                     3
             76 POP_EXCEPT
             78 RERAISE                  1
ExceptionTable:
  4 to 58 -> 64 [0]
  64 to 66 -> 74 [1] lasti


Oh, I didn't follow recent bytecode changes. Ok, now I see that it is not longer possible to build the exception table just from the bytecode. The purpose of the exception table is to handle exceptions: the opcodes related to exception handles are simply gone in Python 3.11.

I was thinking about looking for things like PUSH_EXC_INFO or POP_EXCEPT, but as Guido shows, it doesn't work: the start of the "try" block cannot be detected in the bytecode anymore in Python 3.11.


> If code.replace() is not updated to recompute co_exceptiontable, at least, it would be nice to document the bpo-40222 changes in What's New in Python 3.11 and in the CodeType documentation

You closed the issue. I understand that you don't want to document CodeType.replace() changes neither. Users of this API should follow Python development and notice that their code no longer works with Python 3.11.
msg416829 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-04-05 23:03
If you think the changes to .replace() should be documented just open a new bpo. You made this issue about your various proposals to change .replace().
msg416832 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-04-05 23:22
Ok sure, I created bpo-47236 "Document types.CodeType.replace() changes about co_exceptiontable".
History
Date User Action Args
2022-04-11 14:59:57adminsetgithub: 91341
2022-04-05 23:22:21vstinnersetmessages: + msg416832
2022-04-05 23:03:54gvanrossumsetmessages: + msg416829
2022-04-05 22:59:54vstinnersetmessages: + msg416827
2022-04-05 22:35:28gvanrossumsetstatus: open -> closed
resolution: wont fix
messages: + msg416825

stage: resolved
2022-04-05 06:51:42gregory.p.smithsetnosy: + gregory.p.smith
2022-04-03 18:28:42gvanrossumsetmessages: + msg416637
2022-04-03 12:17:20dom1310dfsetnosy: + dom1310df
2022-04-01 20:38:46vstinnersetmessages: + msg416518
2022-04-01 18:05:04vstinnersetmessages: + msg416503
2022-04-01 17:47:12vstinnersetmessages: + msg416501
2022-04-01 16:24:21gvanrossumsetnosy: + gvanrossum
messages: + msg416499
2022-04-01 16:05:01eric.snowsetnosy: + Mark.Shannon
2022-04-01 09:31:33vstinnercreate