classification
Title: crash on windows invoking flake8
Type: crash Stage: resolved
Components: Windows Versions: Python 3.11, Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: Anthony Sottile, ammar2, christian.heimes, erlendaasland, miss-islington, pablogsal, paul.moore, shreyanavigyan, stestagg, steve.dower, tim.golden, twouters, vstinner, zach.ware
Priority: Keywords: 3.10regression, patch

Created on 2021-05-20 03:24 by Anthony Sottile, last changed 2021-08-06 12:20 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
ref.py vstinner, 2021-05-21 00:08
debug_subtype_dealloc.patch vstinner, 2021-05-21 00:08
Pull Requests
URL Status Linked Edit
PR 26274 merged vstinner, 2021-05-20 23:46
PR 26290 merged miss-islington, 2021-05-21 17:20
PR 27165 merged twouters, 2021-07-15 14:11
PR 27174 merged miss-islington, 2021-07-15 22:41
PR 27175 merged miss-islington, 2021-07-15 22:41
PR 27176 merged miss-islington, 2021-07-15 22:42
Messages (45)
msg393999 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2021-05-20 03:24
I installed python using the installers from python.org -- I originally reproduced this using github actions using pyflakes's testsuite

>ver

Microsoft Windows [Version 10.0.19041.985]

>venv\Scripts\python --version
Python 3.10.0b1

C:\Users\asott\AppData\Local\Temp\y\pyflakes>venv\Scripts\python --version --version
Python 3.10.0b1 (tags/v3.10.0b1:ba42175, May  3 2021, 20:22:30) [MSC v.1928 64 bit (AMD64)]

to reproduce:

```
git clone https://github.com/pycqa/pyflakes
cd pyflakes
git checkout b02ba019e16f7c156ec63c2ea05c627a0fe86c48
```

# install tox somehow
```
C:\python310\python -m venv venv
venv\Scripts\pip install tox
```

here are the versions I have at that point:

```
>venv\Scripts\pip freeze --all
appdirs==1.4.4
colorama==0.4.4
distlib==0.3.1
filelock==3.0.12
packaging==20.9
pip==21.1.1
pluggy==0.13.1
py==1.10.0
pyparsing==2.4.7
setuptools==56.0.0
six==1.16.0
toml==0.10.2
tox==3.23.1
virtualenv==20.4.6
```

then run this a few times:

`venv\Scripts\tox -e py310`

even with `setenv = PYTHONFAULTHANDLER=1` I couldn't get a trace, though maybe that's a linux thing?

it occasionally crashes like this:

```
py310 run-test: commands[2] | flake8 pyflakes setup.py
ERROR: InvocationError for command 'C:\Users\asott\AppData\Local\Temp\y\pyflakes\.tox\py310\Scripts\flake8.EXE' pyflakes setup.py (exited with code 3221225477)
```

from some googling:

> in hex is 0xc0000005 which is the Windows code for an access violation


I don't do much development on windows so I'm passing the torch to someone who knows more :)
msg394000 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2021-05-20 03:28
I can reproduce it outside of tox using:

venv\Scripts\pip install flake8==3.6.0
venv\Scripts\pip install -e . --no-deps  # ignore the conflict, but fix pyflakes

C:\Users\asott\AppData\Local\Temp\y\pyflakes>venv\Scripts\flake8.exe setup.py

C:\Users\asott\AppData\Local\Temp\y\pyflakes>echo %ERRORLEVEL%
-1073741819
msg394002 - (view) Author: Shreyan Avigyan (shreyanavigyan) * Date: 2021-05-20 06:41
This looks like a pyflakes error to me. And you've also mentioned you can reproduce it outside tox with pyflakes.
msg394003 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-05-20 06:55
Anthony, could you please check if any of your dependencies has a native C extension? On Windows they have a .pyd extension.
msg394004 - (view) Author: Shreyan Avigyan (shreyanavigyan) * Date: 2021-05-20 07:19
It's reproducible. I reproduced it on my Windows 10 with Python 3.10.0b1
msg394026 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2021-05-20 14:28
everything in this virtualenv is pure python so I don't think it's a faulty third party extension module
msg394028 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-05-20 14:43
Pablo, Steve, please take a look.
msg394035 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-05-20 16:55
Is b02ba019e16f7c156ec63c2ea05c627a0fe86c48 the correct commit? I cannot checkout this:

❯ git clone https://github.com/pycqa/pyflakes

Cloning into 'pyflakes'...
remote: Enumerating objects: 2651, done.
remote: Counting objects: 100% (53/53), done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 2651 (delta 27), reused 28 (delta 15), pack-reused 2598
Receiving objects: 100% (2651/2651), 905.17 KiB | 10.40 MiB/s, done.
Resolving deltas: 100% (1774/1774), done.

/tmp
❯ cd pyflakes

pyflakes on  master  pyenv 3.9.1
❯ git checkout b02ba019e16f7c156ec63c2ea05c627a0fe86c48

fatal: reference is not a tree: b02ba019e16f7c156ec63c2ea05c627a0fe86c48
msg394037 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2021-05-20 16:57
ah sorry, the branch got squash-merged

this is the equivalent revision after the merge: f3b1b44bf3d2d5927004fa1c2fcf1ab2def816b9
msg394039 - (view) Author: Ammar Askar (ammar2) * (Python committer) Date: 2021-05-20 17:12
Can recreate on the latest 3.10 checkout, taking a look.
msg394040 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-20 17:21
Where does flake8.exe come from? Is it created by pip? Or by Python? Is it part of the flake8 project?
msg394041 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2021-05-20 17:22
vstinner I showed the directions above, but here they are again:


venv\Scripts\pip install flake8==3.6.0
venv\Scripts\pip install -e . --no-deps  # ignore the conflict, but fix pyflakes
msg394042 - (view) Author: Shreyan Avigyan (shreyanavigyan) * Date: 2021-05-20 17:23
This is a flake8 error not tox error. Though if it was a complete flake8 error then it would fail in all Python. But it only fails in 3.10 (don't know about 3.11 because haven't tested it yet.)
msg394048 - (view) Author: Ammar Askar (ammar2) * (Python committer) Date: 2021-05-20 18:35
The segfault seems to be occuring on typeobject.c:1456 during interpreter finalization:

    if (type->tp_flags & Py_TPFLAGS_HEAPTYPE && !(base->tp_flags & Py_TPFLAGS_HEAPTYPE))

where `type` seems to have already been deallocated.

Here's the stack trace:

 	python310.dll!subtype_dealloc(_object * self) Line 1456	C
 	[Inline Frame] python310.dll!_Py_Dealloc(_object *) Line 2288	C
 	[Inline Frame] python310.dll!_Py_DECREF(_object *) Line 500	C
 	[Inline Frame] python310.dll!_Py_XDECREF(_object * op) Line 567	C
>	python310.dll!dict_dealloc(PyDictObject * mp) Line 2067	C
 	[Inline Frame] python310.dll!_Py_Dealloc(_object *) Line 2288	C
 	[Inline Frame] python310.dll!_Py_DECREF(_object *) Line 500	C
 	python310.dll!ast_clear(AST_object * self) Line 784	C
 	python310.dll!delete_garbage(_ts * tstate, _gc_runtime_state * gcstate, PyGC_Head * collectable, PyGC_Head * old) Line 1018	C
 	python310.dll!gc_collect_main(_ts * tstate, int generation, __int64 * n_collected, __int64 * n_uncollectable, int nofail) Line 1304	C
 	[Inline Frame] python310.dll!_PyGC_CollectNoFail(_ts *) Line 2123	C
 	python310.dll!interpreter_clear(_is * interp, _ts * tstate) Line 326	C
 	[Inline Frame] python310.dll!_PyInterpreterState_Clear(_ts *) Line 358	C
 	python310.dll!finalize_interp_clear(_ts * tstate) Line 1639	C
 	python310.dll!Py_FinalizeEx() Line 1813	C
 	[Inline Frame] python310.dll!Py_RunMain() Line 668	C
 	[Inline Frame] python310.dll!pymain_main(_PyArgv *) Line 696	C
 	python310.dll!Py_Main(int argc, wchar_t * * argv) Line 708	C
msg394050 - (view) Author: Steve Stagg (stestagg) Date: 2021-05-20 19:19
I've found that it's reproducible if you run flake8 on a file that just contains:
`a.b`
whereas:
`a`
and
`a(b)`
don't seem to trigger the error.

The object being cleaned up when the segfault appears, seems to be a subclass of ast.AST
msg394051 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-05-20 19:19
Given how hairy finding these GC errors is, I would recommend to bisect this so we can trim down the search to some commit, although my bet is some of the heap type conversions.
msg394053 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-05-20 19:20
Also, why this doesn't happen on Linux?
msg394063 - (view) Author: Ammar Askar (ammar2) * (Python committer) Date: 2021-05-20 20:48
The bisect was bizarre to say the least...

It points to commit 11159d2c9d6616497ef4cc62953a5c3cc8454afb

bpo-43080: pprint for dataclass instances (GH-24389)
* Added pprint support for dataclass instances which don't have a custom __repr__.

which doesn't touch any C code by itself and isn't the heap type conversions.

If I check out to latest 3.10 and revert that commit, this issue seems to go away. Can someone else confirm just to make sure the bisect was done correctly?
msg394065 - (view) Author: Steve Stagg (stestagg) Date: 2021-05-20 21:08
I got the same result from a bisect:

11159d2c9d6616497ef4cc62953a5c3cc8454afb
msg394066 - (view) Author: Steve Stagg (stestagg) Date: 2021-05-20 21:15
So, for me, the single line from 11159d that causes the problem is this:

pprint.py:
import dataclasses as _dataclasses

Obviously importing dataclasses here is having some side-effect
msg394067 - (view) Author: Ammar Askar (ammar2) * (Python committer) Date: 2021-05-20 21:27
Just to thicken the mystery a bit further, if you comment out

    import inspect

in dataclasses.py, the error also goes away.
msg394068 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-05-20 21:31
My bet is because this is caused by the AST module using heap typed here. There is probably a missing strong reference somewhere.
msg394069 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-05-20 21:32
I'm trying to reproduce on Linux but I am unable to make it fail even with the address sanitizer activated :(
msg394070 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-05-20 21:34
FWIW, I'm also unable to reproduce on macOS (so far).
msg394073 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-20 21:51
This bug is really hard to reproduce on Windows. It depends on flake8 is run. It's a random crash in the last GC collection at Python exit.

Crash related to AST in interpreter_clear() remains me bpo-41796.

Extract of the code:

static void
interpreter_clear(PyInterpreterState *interp, PyThreadState *tstate)
{
    ...
    _PyAST_Fini(interp);
    ...
    /* Last garbage collection on this interpreter */
    _PyGC_CollectNoFail(tstate);
    _PyGC_Fini(interp);
    ...
}

The crash occurs in _PyGC_CollectNoFail().

_PyAST_Fini() clears references to AST types in the interpreter AST state. AST type instances hold a reference to the their heap type:

static int
ast_traverse(AST_object *self, visitproc visit, void *arg)
{
    Py_VISIT(Py_TYPE(self));
    Py_VISIT(self->dict);
    return 0;
}

But the crash happens when self->dict is deallocated.
msg394075 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-05-20 22:45
> I've found that it's reproducible if you run flake8 on a file that just contains: `a.b`

ISTR there were some changes made to assigning attributes on AST classes recently? I forget who did them, but I remember discussing it during the sprints last year. Perhaps it's related to that? (Though more likely a subclass, as Steve suggests.)
msg394076 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2021-05-20 22:55
would it maybe be helpful to bisect a history where the dataclasses / inspect import change is introduced earlier?  this would perhaps help pinpoint the other commit which is causing this?
msg394080 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-20 23:45
I have a reproducer (.py script) on Windows, but the crash rate is between 1/3 and 1/2. Also, in VS, if I run "import bug" (bug.py) in the REPL, I fail to reproduce the crash. Using "exec(open('../bug.py').read())" is more likely to trigger the crash, but in VS, I have hard time to trigger the bug. I only reproduced it once.

The bug seems to require very precise conditions. Maybe it depends on the randomized hash function.
msg394082 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-21 00:08
Apply attached debug_subtype_dealloc.patch to reproduce the issue on Linux with attached ref.py script:

$ ./python ref.py 
exit
subtype_dealloc(_ABC): call basedealloc() with Py_REFCNT(type)=1
subtype_dealloc(_Precedence): call basedealloc() with Py_REFCNT(type)=1
subtype_dealloc(property): call basedealloc() with Py_REFCNT(type)=1
subtype_dealloc(FlagBoundary): call basedealloc() with Py_REFCNT(type)=1
LAST GC
Cycle.__del__
Cycle.__del__
subtype_dealloc(keyword): call basedealloc() with Py_REFCNT(type)=1
python: Objects/typeobject.c:1462: subtype_dealloc: Assertion `!_PyMem_IsPtrFreed(type->tp_name)' failed.
Abandon (core dumped)
msg394085 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-05-21 00:14
Ok, I got a crash under the address sanitizer using ref.py:

 ./python lel.py
exit
Cycle.__del__
Cycle.__del__
=================================================================
==77503==ERROR: AddressSanitizer: heap-use-after-free on address 0x61900005a638 at pc 0x55a491f59376 bp 0x7fff8b27cd10 sp 0x7fff8b27cd00
READ of size 8 at 0x61900005a638 thread T0
    #0 0x55a491f59375 in subtype_dealloc Objects/typeobject.c:1456
    #1 0x55a491ebb5e4 in _Py_DECREF Include/object.h:500
    #2 0x55a491ebb5e4 in _Py_XDECREF Include/object.h:567
    #3 0x55a491ebb5e4 in list_dealloc Objects/listobject.c:342
    #4 0x55a491eebe44 in _Py_DECREF Include/object.h:500
    #5 0x55a491eebe44 in _Py_XDECREF Include/object.h:567
    #6 0x55a491eebe44 in dict_dealloc Objects/dictobject.c:2068
    #7 0x55a492305eec in _Py_DECREF Include/object.h:500
    #8 0x55a492305eec in ast_dealloc Python/Python-ast.c:764
    #9 0x55a491f59065 in subtype_dealloc Objects/typeobject.c:1450
    #10 0x55a491eebe44 in _Py_DECREF Include/object.h:500
    #11 0x55a491eebe44 in _Py_XDECREF Include/object.h:567
    #12 0x55a491eebe44 in dict_dealloc Objects/dictobject.c:2068
    #13 0x55a492305eec in _Py_DECREF Include/object.h:500
    #14 0x55a492305eec in ast_dealloc Python/Python-ast.c:764
    #15 0x55a491f59065 in subtype_dealloc Objects/typeobject.c:1450
    #16 0x55a491ebb5e4 in _Py_DECREF Include/object.h:500
    #17 0x55a491ebb5e4 in _Py_XDECREF Include/object.h:567
    #18 0x55a491ebb5e4 in list_dealloc Objects/listobject.c:342
    #19 0x55a491eebe44 in _Py_DECREF Include/object.h:500
    #20 0x55a491eebe44 in _Py_XDECREF Include/object.h:567
    #21 0x55a491eebe44 in dict_dealloc Objects/dictobject.c:2068
    #22 0x55a492305e1f in _Py_DECREF Include/object.h:500
    #23 0x55a492305e1f in ast_clear Python/Python-ast.c:782
    #24 0x55a49216367b in delete_garbage Modules/gcmodule.c:1017
    #25 0x55a49216367b in gc_collect_main Modules/gcmodule.c:1300
    #26 0x55a492165fe5 in _PyGC_CollectNoFail Modules/gcmodule.c:2123
    #27 0x55a492105745 in interpreter_clear Python/pystate.c:326
    #28 0x55a4920f5565 in finalize_interp_clear Python/pylifecycle.c:1634
    #29 0x55a4920fa882 in Py_FinalizeEx Python/pylifecycle.c:1812
    #30 0x55a491e72870 in Py_RunMain Modules/main.c:668
    #31 0x55a491e72870 in pymain_main Modules/main.c:696
    #32 0x55a491e72870 in Py_BytesMain Modules/main.c:720
    #33 0x7f772d82eb24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)
    #34 0x55a491e6ec2d in _start (/home/pablogsal/github/python/master/python+0x174c2d)
0x61900005a638 is located 184 bytes inside of 944-byte region [0x61900005a580,0x61900005a930)
freed by thread T0 here:
    #0 0x7f772dbfaf19 in __interceptor_free /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:127
    #1 0x55a491f5466e in type_dealloc Objects/typeobject.c:4041
    #2 0x55a491f59065 in subtype_dealloc Objects/typeobject.c:1450
    #3 0x55a491ebb5e4 in _Py_DECREF Include/object.h:500
    #4 0x55a491ebb5e4 in _Py_XDECREF Include/object.h:567
    #5 0x55a491ebb5e4 in list_dealloc Objects/listobject.c:342
    #6 0x55a491eebe44 in _Py_DECREF Include/object.h:500
    #7 0x55a491eebe44 in _Py_XDECREF Include/object.h:567
    #8 0x55a491eebe44 in dict_dealloc Objects/dictobject.c:2068
    #9 0x55a492305eec in _Py_DECREF Include/object.h:500
    #10 0x55a492305eec in ast_dealloc Python/Python-ast.c:764
    #11 0x55a491f59065 in subtype_dealloc Objects/typeobject.c:1450
    #12 0x55a491eebe44 in _Py_DECREF Include/object.h:500
    #13 0x55a491eebe44 in _Py_XDECREF Include/object.h:567
    #14 0x55a491eebe44 in dict_dealloc Objects/dictobject.c:2068
    #15 0x55a492305eec in _Py_DECREF Include/object.h:500
    #16 0x55a492305eec in ast_dealloc Python/Python-ast.c:764
    #17 0x55a491f59065 in subtype_dealloc Objects/typeobject.c:1450
    #18 0x55a491ebb5e4 in _Py_DECREF Include/object.h:500
    #19 0x55a491ebb5e4 in _Py_XDECREF Include/object.h:567
    #20 0x55a491ebb5e4 in list_dealloc Objects/listobject.c:342
    #21 0x55a491eebe44 in _Py_DECREF Include/object.h:500
    #22 0x55a491eebe44 in _Py_XDECREF Include/object.h:567
    #23 0x55a491eebe44 in dict_dealloc Objects/dictobject.c:2068
    #24 0x55a492305e1f in _Py_DECREF Include/object.h:500
    #25 0x55a492305e1f in ast_clear Python/Python-ast.c:782
    #26 0x55a49216367b in delete_garbage Modules/gcmodule.c:1017
    #27 0x55a49216367b in gc_collect_main Modules/gcmodule.c:1300
    #28 0x55a492165fe5 in _PyGC_CollectNoFail Modules/gcmodule.c:2123
    #29 0x55a492105745 in interpreter_clear Python/pystate.c:326
    #30 0x55a4920f5565 in finalize_interp_clear Python/pylifecycle.c:1634
    #31 0x55a4920fa882 in Py_FinalizeEx Python/pylifecycle.c:1812
    #32 0x55a491e72870 in Py_RunMain Modules/main.c:668
    #33 0x55a491e72870 in pymain_main Modules/main.c:696
    #34 0x55a491e72870 in Py_BytesMain Modules/main.c:720
    #35 0x7f772d82eb24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)

previously allocated by thread T0 here:
    #0 0x7f772dbfb279 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x55a4921667b5 in _PyObject_GC_Alloc Modules/gcmodule.c:2250
    #2 0x55a4921667b5 in _PyObject_GC_Malloc Modules/gcmodule.c:2277
    #3 0x55a491f56986 in PyType_GenericAlloc Objects/typeobject.c:1160
    #4 0x55a491f866ea in type_new_alloc Objects/typeobject.c:2732
    #5 0x55a491f866ea in type_new_init Objects/typeobject.c:3144
    #6 0x55a491f866ea in type_new_impl Objects/typeobject.c:3167
    #7 0x55a491f866ea in type_new Objects/typeobject.c:3312
    #8 0x55a491f5b377 in type_call Objects/typeobject.c:1127
    #9 0x55a491e92ad8 in _PyObject_MakeTpCall Objects/call.c:215
    #10 0x55a491e93f33 in _PyObject_VectorcallTstate Include/cpython/abstract.h:114
    #11 0x55a491e93f33 in _PyObject_CallFunctionVa Objects/call.c:485
    #12 0x55a491e973af in PyObject_CallFunction Objects/call.c:507
    #13 0x55a49230623b in make_type Python/Python-ast.c:935
    #14 0x55a49231d15f in init_types Python/Python-ast.c:1735
    #15 0x55a49231edaf in get_ast_state Python/Python-ast.c:19
    #16 0x55a49231edaf in astmodule_exec Python/Python-ast.c:10795
    #17 0x55a491f1c866 in PyModule_ExecDef Objects/moduleobject.c:407
    #18 0x55a4920bddf2 in _imp_exec_builtin (/home/pablogsal/github/python/master/python+0x3c3df2)
    #19 0x55a492303267 in cfunction_vectorcall_O Objects/methodobject.c:512
    #20 0x55a491e94d69 in PyVectorcall_Call Objects/call.c:255
    #21 0x55a491e58b83 in do_call_core Python/ceval.c:5937
    #22 0x55a491e58b83 in _PyEval_EvalFrameDefault Python/ceval.c:4278
    #23 0x55a492050e77 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
    #24 0x55a492050e77 in _PyEval_Vector Python/ceval.c:5069
    #25 0x55a491e617da in _PyObject_VectorcallTstate Include/cpython/abstract.h:114
    #26 0x55a491e617da in PyObject_Vectorcall Include/cpython/abstract.h:123
    #27 0x55a491e617da in call_function Python/ceval.c:5885
    #28 0x55a491e617da in _PyEval_EvalFrameDefault Python/ceval.c:4214
    #29 0x55a492050e77 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
    #30 0x55a492050e77 in _PyEval_Vector Python/ceval.c:5069
    #31 0x55a491e692fd in _PyObject_VectorcallTstate Include/cpython/abstract.h:114
    #32 0x55a491e692fd in PyObject_Vectorcall Include/cpython/abstract.h:123
    #33 0x55a491e692fd in call_function Python/ceval.c:5885
    #34 0x55a491e692fd in _PyEval_EvalFrameDefault Python/ceval.c:4182
    #35 0x55a492050e77 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
    #36 0x55a492050e77 in _PyEval_Vector Python/ceval.c:5069
    #37 0x55a491e617da in _PyObject_VectorcallTstate Include/cpython/abstract.h:114
    #38 0x55a491e617da in PyObject_Vectorcall Include/cpython/abstract.h:123
    #39 0x55a491e617da in call_function Python/ceval.c:5885
    #40 0x55a491e617da in _PyEval_EvalFrameDefault Python/ceval.c:4214
    #41 0x55a492050e77 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
    #42 0x55a492050e77 in _PyEval_Vector Python/ceval.c:5069
    #43 0x55a491e617da in _PyObject_VectorcallTstate Include/cpython/abstract.h:114
    #44 0x55a491e617da in PyObject_Vectorcall Include/cpython/abstract.h:123
    #45 0x55a491e617da in call_function Python/ceval.c:5885
    #46 0x55a491e617da in _PyEval_EvalFrameDefault Python/ceval.c:4214
    #47 0x55a492050e77 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
    #48 0x55a492050e77 in _PyEval_Vector Python/ceval.c:5069
    #49 0x55a491e93a05 in _PyObject_VectorcallTstate Include/cpython/abstract.h:114
    #50 0x55a491e93a05 in object_vacall Objects/call.c:734
    #51 0x55a491e99424 in _PyObject_CallMethodIdObjArgs Objects/call.c:825
    #52 0x55a4920c27f7 in import_find_and_load Python/import.c:1499
    #53 0x55a4920c27f7 in PyImport_ImportModuleLevelObject Python/import.c:1600
    #54 0x55a491e68ac5 in import_name Python/ceval.c:6010
    #55 0x55a491e68ac5 in _PyEval_EvalFrameDefault Python/ceval.c:3701
    #56 0x55a49205077f in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
    #57 0x55a49205077f in _PyEval_Vector Python/ceval.c:5069
    #58 0x55a49205077f in PyEval_EvalCode Python/ceval.c:1135
SUMMARY: AddressSanitizer: heap-use-after-free Objects/typeobject.c:1456 in subtype_dealloc
Shadow bytes around the buggy address:
  0x0c3280003470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3280003480: 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa fa
  0x0c3280003490: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c32800034a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c32800034b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c32800034c0: fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd
  0x0c32800034d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c32800034e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c32800034f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3280003500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3280003510: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==77503==ABORTING
msg394086 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-21 00:17
Valgrind detects the bug (unmodified Python):

$ PYTHONMALLOC=debug valgrind ./python ref.py 
(...)
==607098== Invalid read of size 8
==607098==    at 0x493FBE: subtype_dealloc (typeobject.c:1456)
==607098==    by 0x47C914: _Py_Dealloc (object.c:2288)
==607098==    by 0x45F6BF: _Py_DECREF (object.h:500)
==607098==    by 0x45F70D: _Py_XDECREF (object.h:567)
==607098==    by 0x4654F5: dict_dealloc (dictobject.c:2068)
==607098==    by 0x47C914: _Py_Dealloc (object.c:2288)
==607098==    by 0x668EBE: _Py_DECREF (object.h:500)
==607098==    by 0x66E8BE: ast_dealloc (Python-ast.c:764)
==607098==    by 0x493FB9: subtype_dealloc (typeobject.c:1450)
(...)
msg394100 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-05-21 09:30
> Crash related to AST in interpreter_clear() remains me bpo-41796.

Well remembered, Victor!

Bisecting using Pablo's reproducer:
fd957c124c44441d9c5eaf61f7af8cf266bafcb1 is the first bad commit
commit fd957c124c44441d9c5eaf61f7af8cf266bafcb1
Author: Victor Stinner <vstinner@python.org>
Date:   Tue Nov 3 18:07:15 2020 +0100

    bpo-41796: Call _PyAST_Fini() earlier to fix a leak (GH-23131)
msg394131 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-21 17:25
Anthony Sottile: I'm surprised that AST nodes survive until the last GC collection. It seems like somehow a reference cycle prevent to delete these nodes, and this reference cycle is kept alive somehow until the last GC collection at Python exit.

It would be interesting to follow references. You may start from PyInterpreterState which keeps objects alive until the last GC collection:

* os.register_at_fork() callbacks
* codecs.register() callback
* Python audit hooks
* sys.modules objects
* sys.__dict__

Callbacks keep global variables of a module alive through its __globals__ attribute (namespace of the module where it's defined).
msg394161 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2021-05-21 21:44
that version of flake8 uses multiprocessing (even for 1 file) -- would the ast objects be involved in that way? (pyflakes also makes reference cyles to handle "parent" relationships)
msg394163 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-21 22:50
New changeset 50b0d148a68072292832eb69bdf1815b8059355f by Miss Islington (bot) in branch '3.10':
bpo-44184: Fix subtype_dealloc() for freed type (GH-26274) (GH-26290)
https://github.com/python/cpython/commit/50b0d148a68072292832eb69bdf1815b8059355f
msg394164 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-21 22:56
The issue is fixed by:

commit 615069eb08494d089bf24e43547fbc482ed699b8
Author: Victor Stinner <vstinner@python.org>
Date:   Fri May 21 19:19:54 2021 +0200

    bpo-44184: Fix subtype_dealloc() for freed type (GH-26274)
    
    Fix a crash at Python exit when a deallocator function removes the
    last strong reference to a heap type.
    
    Don't read type memory after calling basedealloc() since
    basedealloc() can deallocate the type and free its memory.
    
    _PyMem_IsPtrFreed() argument is now constant.
msg394165 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-21 22:57
Anthony Sottile: "that version of flake8 uses multiprocessing (even for 1 file) -- would the ast objects be involved in that way? (pyflakes also makes reference cyles to handle "parent" relationships)"

I expect flake8 to use the ast somewhere to analyze source code. But I don't have the bandwidth to investigate flake8 creates a reference cycle.

The Python regression is fixed, I closed the issue. Thanks for the bug report Anthony!
msg394172 - (view) Author: Ammar Askar (ammar2) * (Python committer) Date: 2021-05-22 00:30
Indeed, it's quite a tricky issue so I'm glad it was caught in the beta.

Thank you for the report Anthony. Thanks for tracing the root cause and the fix Victor and thank you to everyone who helped debug.
msg397553 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2021-07-15 14:07
Reopening this issue, as there is another branch (for non-GC heaptypes) earlier in subtype_dealloc that I believe suffers from the same problem. Actually triggering the error in a test has been difficult because as far as I can tell it relies on garbage collection at the right time, but reading the code it seems clear it's problematic. I'll prepare a PR to fix it there.

I'm also reopening this issue because I believe it should've been backported to 3.9, and possibly 3.8 (if it's considered a security problem to get python to read and write freed memory). I found this issue in 3.9 while debugging a pybind11 crash. I'll backport after the other PR is in (or rejected).
msg397584 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2021-07-15 22:41
New changeset 074e7659f208051b6b973f7fdb654dd22b93aaa2 by T. Wouters in branch 'main':
bpo-44184: Apply GH-26274 to the non-GC-type branch of subtype_dealloc (GH-27165)
https://github.com/python/cpython/commit/074e7659f208051b6b973f7fdb654dd22b93aaa2
msg397585 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2021-07-15 23:18
New changeset 6aa59c68dc7910c0675ad23c1f9d88edfb81dfcb by Miss Islington (bot) in branch '3.10':
bpo-44184: Apply GH-26274 to the non-GC-type branch of subtype_dealloc (GH-27165) (GH-27174)
https://github.com/python/cpython/commit/6aa59c68dc7910c0675ad23c1f9d88edfb81dfcb
msg397586 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2021-07-15 23:29
Fix extended to the other branch (and backported to 3.10), and both parts backported to 3.9. I don't think it counts as a security issue, so not backporting to 3.8.
msg397587 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2021-07-15 23:35
New changeset 0b4704973dbef712d05bdd62349bb4244f545430 by Miss Islington (bot) in branch '3.9':
bpo-44184: Apply GH-26274 to the non-GC-type branch of subtype_dealloc (GH-27165) (GH-27175)
https://github.com/python/cpython/commit/0b4704973dbef712d05bdd62349bb4244f545430
msg397588 - (view) Author: miss-islington (miss-islington) Date: 2021-07-15 23:36
New changeset 298ee657ab8170adf75a186c0414b7ca3baf1991 by Miss Islington (bot) in branch '3.9':
bpo-44184: Fix subtype_dealloc() for freed type (GH-26274)
https://github.com/python/cpython/commit/298ee657ab8170adf75a186c0414b7ca3baf1991
msg399078 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-08-06 12:20
> Fix extended to the other branch (and backported to 3.10), and both parts backported to 3.9. I don't think it counts as a security issue, so not backporting to 3.8.

Thanks Thomas, I missed this code path!
History
Date User Action Args
2021-08-06 12:20:51vstinnersetmessages: + msg399078
2021-07-15 23:36:59miss-islingtonsetmessages: + msg397588
2021-07-15 23:35:33twouterssetmessages: + msg397587
2021-07-15 23:29:54twouterssetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-07-15 23:29:42twouterssetmessages: + msg397586
2021-07-15 23:18:41twouterssetmessages: + msg397585
2021-07-15 22:42:19miss-islingtonsetpull_requests: + pull_request25713
2021-07-15 22:41:13miss-islingtonsetpull_requests: + pull_request25712
2021-07-15 22:41:07miss-islingtonsetpull_requests: + pull_request25711
2021-07-15 22:41:05twouterssetmessages: + msg397584
2021-07-15 14:11:19twouterssetstage: resolved -> patch review
pull_requests: + pull_request25701
2021-07-15 14:07:59twouterssetstatus: closed -> open

nosy: + twouters
messages: + msg397553

resolution: fixed -> (no value)
2021-05-22 00:30:11ammar2setmessages: + msg394172
2021-05-21 22:57:50vstinnersetpriority: release blocker ->

messages: + msg394165
2021-05-21 22:56:32vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg394164

stage: patch review -> resolved
2021-05-21 22:50:18vstinnersetmessages: + msg394163
2021-05-21 21:44:39Anthony Sottilesetmessages: + msg394161
2021-05-21 17:25:06vstinnersetmessages: + msg394131
2021-05-21 17:20:11miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request24894
2021-05-21 09:30:12erlendaaslandsetmessages: + msg394100
2021-05-21 00:17:48vstinnersetmessages: + msg394086
2021-05-21 00:14:17pablogsalsetmessages: + msg394085
2021-05-21 00:08:28vstinnersetfiles: + debug_subtype_dealloc.patch
2021-05-21 00:08:17vstinnersetfiles: + ref.py

messages: + msg394082
2021-05-20 23:46:17vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request24880
2021-05-20 23:45:28vstinnersetmessages: + msg394080
2021-05-20 22:55:14Anthony Sottilesetmessages: + msg394076
2021-05-20 22:45:09steve.dowersetmessages: + msg394075
2021-05-20 21:51:12vstinnersetmessages: + msg394073
2021-05-20 21:34:35erlendaaslandsetmessages: + msg394070
2021-05-20 21:32:12pablogsalsetmessages: + msg394069
2021-05-20 21:31:14pablogsalsetmessages: + msg394068
2021-05-20 21:27:30ammar2setmessages: + msg394067
2021-05-20 21:20:35erlendaaslandsetnosy: + erlendaasland
2021-05-20 21:15:29stestaggsetmessages: + msg394066
2021-05-20 21:08:03stestaggsetmessages: + msg394065
2021-05-20 20:48:45ammar2setmessages: + msg394063
2021-05-20 19:20:02pablogsalsetmessages: + msg394053
2021-05-20 19:19:35pablogsalsetmessages: + msg394051
2021-05-20 19:19:10stestaggsetmessages: + msg394050
2021-05-20 18:35:14ammar2setmessages: + msg394048
2021-05-20 18:00:21stestaggsetnosy: + stestagg
2021-05-20 17:23:37shreyanavigyansetmessages: + msg394042
2021-05-20 17:22:17Anthony Sottilesetmessages: + msg394041
2021-05-20 17:21:03vstinnersetnosy: + vstinner
messages: + msg394040
2021-05-20 17:12:52ammar2setnosy: + ammar2
messages: + msg394039
2021-05-20 16:57:09Anthony Sottilesetmessages: + msg394037
2021-05-20 16:55:19pablogsalsetmessages: + msg394035
2021-05-20 16:26:11pablogsalsetpriority: deferred blocker -> release blocker
2021-05-20 14:43:23christian.heimessetpriority: normal -> deferred blocker

assignee: steve.dower
versions: + Python 3.11
keywords: + 3.10regression
nosy: + pablogsal

messages: + msg394028
2021-05-20 14:28:19Anthony Sottilesetmessages: + msg394026
2021-05-20 07:19:36shreyanavigyansetmessages: + msg394004
2021-05-20 06:55:13christian.heimessetnosy: + christian.heimes
messages: + msg394003
2021-05-20 06:41:39shreyanavigyansettitle: crash on windows invoking flake8 under tox -> crash on windows invoking flake8
nosy: + shreyanavigyan

messages: + msg394002

type: crash
2021-05-20 03:28:01Anthony Sottilesetmessages: + msg394000
2021-05-20 03:26:06xtreaksetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2021-05-20 03:24:15Anthony Sottilecreate