classification
Title: Access violations in C extension modules on Python 3.9.3
Type: crash Stage: resolved
Components: Extension Modules, Windows Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, cgohlke, gregory.p.smith, lukasz.langa, paul.moore, steve.dower, terry.reedy, tim.golden, zach.ware
Priority: release blocker Keywords: patch

Created on 2021-04-03 03:09 by cgohlke, last changed 2021-05-03 21:42 by terry.reedy. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 25160 closed Mark.Shannon, 2021-04-03 12:10
PR 25179 merged gregory.p.smith, 2021-04-04 04:17
Messages (5)
msg390116 - (view) Author: Christoph Gohlke (cgohlke) Date: 2021-04-03 03:09
First reported at https://github.com/numpy/numpy/issues/18720

After upgrading to Python 3.9.3, 32-bit on Windows, importing numpy (installed via `pip install numpy`) crashes:

```
Microsoft Windows [Version 10.0.19041.906]
C:\Python39-32>python -X dev
Python 3.9.3 (tags/v3.9.3:e723086, Apr  2 2021, 11:01:03) [MSC v.1928 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Windows fatal exception: access violation

Current thread 0x0000105c (most recent call first):
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1116 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1116 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1116 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "C:\Python39-32\lib\site-packages\numpy\random\_pickle.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
  File "C:\Python39-32\lib\site-packages\numpy\random\__init__.py", line 179 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
  File "C:\Python39-32\lib\site-packages\numpy\__init__.py", line 156 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<stdin>", line 1 in <module>
```

Rebuilding numpy from source against Python 3.9.3 fixes this crash, but many other packages compiled against older versions of Python 3.9.x still crash.

I think this might be due to <https://github.com/python/cpython/commit/8b795ab5541d8a4e69be4137dfdc207714270b77#diff-7285576bdacf86fe37274d3d0d399c29b4be2959005f60ac0729615d8fca9186>.

In `bpo-42500: Fix recursion in or after except (GH-23568) (#24501)`, the public `PyThreadState` struct changed. This breaks ABI compatibility for code that is directly accessing the `PyThreadState` struct members, which is not uncommon. E.g. in Cython <https://github.com/cython/cython/blob/0.29.x/Cython/Utility/Exceptions.c#L93-L115>. This code will fail on 32-bit. On 64-bit it passes because of fortunate struct alignment AFAICT.
msg390134 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-04-03 18:14
Tomorrow evening CEST (around 30 hours from now) I intend to release a hotfix 3.9.4 with Mark's fix.
msg390172 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-04-04 04:19
I really suggest a simple rollback (revert) of the change rather than a redo-fix for 3.9.4.  Unless you like the idea of a possible 3.9.5 if the redo-fix itself has issues that haven't been given due thought in the rush to undo the 3.9.3 mistake.

This leaves more time for a proper fix of bpo-42500 in 3.9 not crafted under duress.
msg390181 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-04-04 11:02
New changeset c7b0feca25fc68ec3e0884b82e5f45a4da011e8e by Gregory P. Smith in branch '3.9':
[3.9] bpo-43710: Rollback the 3.9 bpo-42500 fix, it broke the ABI in 3.9.3 (#25179)
https://github.com/python/cpython/commit/c7b0feca25fc68ec3e0884b82e5f45a4da011e8e
msg390212 - (view) Author: Christoph Gohlke (cgohlke) Date: 2021-04-04 21:07
Thank you for fixing this issue so fast! Python 3.9.4 works well.
History
Date User Action Args
2021-05-03 21:42:14terry.reedysetassignee: terry.reedy ->
2021-05-03 21:41:49terry.reedysetcomponents: + Extension Modules, Windows, - IDLE
2021-05-03 16:33:41mohansaisetassignee: terry.reedy

components: + IDLE, - Extension Modules, Windows
nosy: + terry.reedy
2021-04-30 15:18:32zach.waresetfiles: - logs.gz
2021-04-30 15:18:09zach.waresetfiles: - Screenshot_20210426-104915_Lucknow Public School.jpg
2021-04-30 15:11:20anjalisingh8225setfiles: + Screenshot_20210426-104915_Lucknow Public School.jpg
2021-04-19 16:59:58Sunday234setfiles: + logs.gz
2021-04-04 21:07:05cgohlkesetmessages: + msg390212
2021-04-04 11:03:35lukasz.langasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-04-04 11:02:51lukasz.langasetmessages: + msg390181
2021-04-04 04:19:30gregory.p.smithsetmessages: + msg390172
2021-04-04 04:17:53gregory.p.smithsetnosy: + gregory.p.smith
pull_requests: + pull_request23920
2021-04-03 18:14:11lukasz.langasetpriority: normal -> release blocker

messages: + msg390134
2021-04-03 12:10:43Mark.Shannonsetkeywords: + patch
stage: patch review
pull_requests: + pull_request23907
2021-04-03 03:39:59xtreaksetnosy: + lukasz.langa, Mark.Shannon
2021-04-03 03:09:40cgohlkecreate