classification
Title: peephole optimizer bug relating to JUMP_IF_NOT_EXC_MATCH
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: FFY00, Mark.Shannon, corona10, methane, miurahr
Priority: normal Keywords: patch

Created on 2020-10-17 02:33 by miurahr, last changed 2020-10-22 15:42 by corona10. This issue is now closed.

Files
File name Uploaded Description Edit
test_main.py miurahr, 2020-10-17 02:33 test case to reproduce
py_stacktrace.txt miurahr, 2020-10-17 02:33 pytest result which become segmentation fault
gdb_backtrace.txt miurahr, 2020-10-17 02:34 gdb backtrace
fix42057.patch methane, 2020-10-19 15:17
Pull Requests
URL Status Linked Edit
PR 22802 merged methane, 2020-10-20 02:45
PR 22878 merged methane, 2020-10-22 02:10
PR 22893 merged Mark.Shannon, 2020-10-22 14:32
Messages (11)
msg378796 - (view) Author: Hiroshi Miura (miurahr) * Date: 2020-10-17 02:33
I've observed that pytest becomes segmentation fault on python 3.9.0 with attached case.

I've tested the case with several python versions;
  python 3.9.0a2 - good
  python 3.9.0a3, 3.9.0-final - bad
  python 3.10.0a1 - good

- OS: Mint Linux 20, Linux kernel 5.8.14
- Attachments:
    * test_main.py  - pytest test case to reproduce
    * py_stacktrace.txt - pytest result which become segmentation fault
    * gdb_backtrace.txt - gdb backtrace

So I've bisected and  a result is as follows;

9af0e47b1705457bb6b327c197f2ec5737a1d8f6 is the first bad commit
commit 9af0e47b1705457bb6b327c197f2ec5737a1d8f6
Author: Mark Shannon <mark@hotpy.org>
Date:   Tue Jan 14 10:12:45 2020 +0000

    bpo-39156: Break up COMPARE_OP into four logically distinct opcodes. (GH-17754)

    Break up COMPARE_OP into four logically distinct opcodes:
    * COMPARE_OP for rich comparisons
    * IS_OP for 'is' and 'is not' tests
    * CONTAINS_OP for 'in' and 'is not' tests
    * JUMP_IF_NOT_EXC_MATCH for checking exceptions in 'try-except' statements.

 Doc/library/dis.rst                                |   21 +
 Include/opcode.h                                   |    8 +-
 Lib/importlib/_bootstrap_external.py               |    3 +-
 Lib/opcode.py                                      |    7 +-
 Lib/test/test_dis.py                               |  141 +-
 Lib/test/test_peepholer.py                         |   12 +-
 Lib/test/test_positional_only_arg.py               |    6 +-
 .../2019-12-30-10-53-59.bpo-39156.veT-CB.rst       |    9 +
 PC/launcher.c                                      |    3 +-
 Python/ceval.c                                     |  137 +-
 Python/compile.c                                   |   71 +-
 Python/importlib.h                                 | 2922 +++++++------
 Python/importlib_external.h                        | 4560 ++++++++++----------
 Python/importlib_zipimport.h                       | 1831 ++++----
 Python/opcode_targets.h                            |    6 +-
 Python/peephole.c                                  |    6 +-
 Tools/scripts/generate_opcode_h.py                 |    5 -
 17 files changed, 4901 insertions(+), 4847 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-12-30-10-53-59.bpo-39156.veT-CB.rst
msg378798 - (view) Author: Hiroshi Miura (miurahr) * Date: 2020-10-17 03:18
Here is a result of running pytest on python 3.9.0 with gdb.

==================================================== test session starts ====================================================
platform linux -- Python 3.9.0, pytest-4.6.9, py-1.8.1, pluggy-0.13.0
rootdir: /home/miurahr/Projects/cpython
collected 2 items

test_main.py .
Program received signal SIGSEGV, Segmentation fault.
PyException_GetContext (self=self@entry=<traceback at remote 0x7ffff62b47c0>) at ../Objects/exceptions.c:351
warning: Source file is more recent than executable.
351	    Py_XINCREF(context);
msg378799 - (view) Author: Hiroshi Miura (miurahr) * Date: 2020-10-17 03:43
FYI: A following commit fixes the issue in 3.10 development branch.

6e8128f02e ("bpo-41323: Perform 'peephole' optimizations directly on the CFG. (GH-21517)", 2020-07-30)
msg378802 - (view) Author: Hiroshi Miura (miurahr) * Date: 2020-10-17 04:55
A test code does not always reproduce the issue. Please try it in several times.

It seems to be happened when multiple threads try execute a same function which produces an exception, and both callers try to catch the exception at the same time.
msg378948 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-10-19 13:02
The test example has no reference to pytest in it.

How are you running it?
Can you produce a segmentation fault when run without pytest?
msg378952 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-10-19 15:05
I confirmed the issue. The simple version of the reproducer is:

```
def callee():
    raise Exception

def caller():
    try:
        callee()
    except Exception or Exception:
        pass

caller()
```

I can see assertion failure consisntently, when I use python --with-pydebug.

It seems peephole bug. Without peephole:

```
  7     >>   12 DUP_TOP
             14 LOAD_GLOBAL              1 (Exception)
             16 JUMP_IF_TRUE_OR_POP     20
             18 LOAD_GLOBAL              1 (Exception)
        >>   20 JUMP_IF_NOT_EXC_MATCH    32
             22 POP_TOP
             24 POP_TOP
             26 POP_TOP

  8          28 POP_EXCEPT
             30 JUMP_FORWARD             2 (to 34)
        >>   32 RERAISE
        >>   34 LOAD_CONST               0 (None)
             36 RETURN_VALUE
```

And with peephole:

```
  7     >>   12 DUP_TOP
             14 LOAD_GLOBAL              1 (Exception)
             16 POP_JUMP_IF_TRUE        22
             18 LOAD_GLOBAL              1 (Exception)
             20 JUMP_IF_NOT_EXC_MATCH    32
        >>   22 POP_TOP
             24 POP_TOP
             26 POP_TOP

  8          28 POP_EXCEPT
             30 JUMP_FORWARD             2 (to 34)
        >>   32 RERAISE
        >>   34 LOAD_CONST               0 (None)
             36 RETURN_VALUE
```

JUMP_IF_TRUE_OR_POP is converted into POP_JUMP_IF_TRUE. Exception is popped although JUMP_IF_NOT_EXC_MATCH needs it.
msg378953 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-10-19 15:17
I confirmed fix42057.patch fix the assertion failure. But I don't know where and how to write test yet.
msg379268 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-10-22 02:06
New changeset 07a44d9572c7746568a7fe2fbcd42127fd6d4019 by Inada Naoki in branch '3.9':
bpo-42057: Fix peephole optimizer (GH-22802)
https://github.com/python/cpython/commit/07a44d9572c7746568a7fe2fbcd42127fd6d4019
msg379270 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-10-22 02:28
New changeset 8f6787d93db1b6022db44b1e1d22460c2b74f60b by Inada Naoki in branch '3.9':
bpo-42057: Add a test case (GH-22878)
https://github.com/python/cpython/commit/8f6787d93db1b6022db44b1e1d22460c2b74f60b
msg379271 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-10-22 02:38
Thank you for reporting with reproducer.
msg379304 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-10-22 15:42
New changeset b52432cb8cd7f5f1fc71ea3b7c0ea11573d504f0 by Mark Shannon in branch 'master':
bpo-42057: Add regression test to master. (GH-22893)
https://github.com/python/cpython/commit/b52432cb8cd7f5f1fc71ea3b7c0ea11573d504f0
History
Date User Action Args
2020-10-22 15:42:36corona10setnosy: + corona10
messages: + msg379304
2020-10-22 14:32:26Mark.Shannonsetpull_requests: + pull_request21825
2020-10-22 02:38:54methanesetstatus: open -> closed
resolution: fixed
messages: + msg379271

stage: patch review -> resolved
2020-10-22 02:28:43methanesetmessages: + msg379270
2020-10-22 02:10:33methanesetpull_requests: + pull_request21820
2020-10-22 02:06:11methanesetmessages: + msg379268
2020-10-20 02:45:40methanesetstage: patch review
pull_requests: + pull_request21759
2020-10-20 02:24:36methanesettitle: pytest case which catch exceptions become segfault -> peephole optimizer bug relating to JUMP_IF_NOT_EXC_MATCH
2020-10-19 15:17:22methanesetfiles: + fix42057.patch
keywords: + patch
messages: + msg378953
2020-10-19 15:05:32methanesetnosy: + methane
messages: + msg378952
2020-10-19 13:02:40Mark.Shannonsetmessages: + msg378948
2020-10-18 00:30:19FFY00setnosy: + FFY00
2020-10-17 04:55:21miurahrsetmessages: + msg378802
2020-10-17 03:43:17miurahrsetnosy: + Mark.Shannon
messages: + msg378799
2020-10-17 03:18:10miurahrsetmessages: + msg378798
2020-10-17 02:34:10miurahrsetfiles: + gdb_backtrace.txt
2020-10-17 02:33:53miurahrsetfiles: + py_stacktrace.txt
2020-10-17 02:33:29miurahrcreate