Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Access violations in C extension modules on Python 3.9.3 #87876

Closed
cgohlke mannequin opened this issue Apr 3, 2021 · 5 comments
Closed

Access violations in C extension modules on Python 3.9.3 #87876

cgohlke mannequin opened this issue Apr 3, 2021 · 5 comments
Labels
3.9 only security fixes extension-modules C modules in the Modules dir OS-windows release-blocker type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@cgohlke
Copy link
Mannequin

cgohlke mannequin commented Apr 3, 2021

BPO 43710
Nosy @terryjreedy, @gpshead, @pfmoore, @tjguk, @ambv, @markshannon, @zware, @zooba
PRs
  • [3.9] bpo-43710: Change layout of pystate struct to preserve intra-version ABI compatibiliy. #25160
  • [3.9] bpo-43710: Rollback the 3.9 bpo-42500 fix, it broke the ABI in 3.9.3 #25179
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-04-04.11:03:35.837>
created_at = <Date 2021-04-03.03:09:40.678>
labels = ['extension-modules', 'release-blocker', 'OS-windows', 'type-crash', '3.9']
title = 'Access violations in C extension modules on Python 3.9.3'
updated_at = <Date 2021-05-03.21:42:14.320>
user = 'https://bugs.python.org/cgohlke'

    bugs.python.org fields:

    activity = <Date 2021-05-03.21:42:14.320>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-04.11:03:35.837>
closer = 'lukasz.langa'
components = ['Extension Modules', 'Windows']
creation = <Date 2021-04-03.03:09:40.678>
creator = 'cgohlke'
dependencies = []
files = []
hgrepos = []
issue_num = 43710
keywords = ['patch']
message_count = 5.0
messages = ['390116', '390134', '390172', '390181', '390212']
nosy_count = 9.0
nosy_names = ['terry.reedy', 'gregory.p.smith', 'paul.moore', 'tim.golden', 'cgohlke', 'lukasz.langa', 'Mark.Shannon', 'zach.ware', 'steve.dower']
pr_nums = ['25160', '25179']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue43710'
versions = ['Python 3.9']
    @cgohlke
    Copy link
    Mannequin Author

    cgohlke mannequin commented Apr 3, 2021

    First reported at numpy/numpy#18720

    After upgrading to Python 3.9.3, 32-bit on Windows, importing numpy (installed via pip install numpy) crashes:

    Microsoft Windows [Version 10.0.19041.906]
C:\Python39-32>python -X dev
Python 3.9.3 (tags/v3.9.3:e723086, Apr  2 2021, 11:01:03) [MSC v.1928 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Windows fatal exception: access violation

Current thread 0x0000105c (most recent call first):
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1116 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1116 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1116 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "C:\Python39-32\lib\site-packages\numpy\random\_pickle.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
  File "C:\Python39-32\lib\site-packages\numpy\random\__init__.py", line 179 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
  File "C:\Python39-32\lib\site-packages\numpy\__init__.py", line 156 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<stdin>", line 1 in <module>

    Rebuilding numpy from source against Python 3.9.3 fixes this crash, but many other packages compiled against older versions of Python 3.9.x still crash.

    I think this might be due to <8b795ab#diff-7285576bdacf86fe37274d3d0d399c29b4be2959005f60ac0729615d8fca9186%5C%3E.

    In bpo-42500: Fix recursion in or after except (GH-23568) (bpo-24501), the public PyThreadState struct changed. This breaks ABI compatibility for code that is directly accessing the PyThreadState struct members, which is not uncommon. E.g. in Cython https://github.com/cython/cython/blob/0.29.x/Cython/Utility/Exceptions.c#L93-L115. This code will fail on 32-bit. On 64-bit it passes because of fortunate struct alignment AFAICT.

    @cgohlke cgohlke mannequin added extension-modules C modules in the Modules dir OS-windows 3.9 only security fixes type-crash A hard crash of the interpreter, possibly with a core dump labels Apr 3, 2021
    @ambv
    Copy link
    Contributor

    ambv commented Apr 3, 2021

    Tomorrow evening CEST (around 30 hours from now) I intend to release a hotfix 3.9.4 with Mark's fix.

    @gpshead
    Copy link
    Member

    gpshead commented Apr 4, 2021

    I really suggest a simple rollback (revert) of the change rather than a redo-fix for 3.9.4. Unless you like the idea of a possible 3.9.5 if the redo-fix itself has issues that haven't been given due thought in the rush to undo the 3.9.3 mistake.

    This leaves more time for a proper fix of bpo-42500 in 3.9 not crafted under duress.

    @ambv
    Copy link
    Contributor

    ambv commented Apr 4, 2021

    New changeset c7b0fec by Gregory P. Smith in branch '3.9':
    [3.9] bpo-43710: Rollback the 3.9 bpo-42500 fix, it broke the ABI in 3.9.3 (bpo-25179)
    c7b0fec

    @ambv ambv closed this as completed Apr 4, 2021
    @ambv ambv closed this as completed Apr 4, 2021
    @cgohlke
    Copy link
    Mannequin Author

    cgohlke mannequin commented Apr 4, 2021

    Thank you for fixing this issue so fast! Python 3.9.4 works well.

    @sai sai mannequin added topic-IDLE and removed extension-modules C modules in the Modules dir OS-windows labels May 3, 2021
    @sai sai mannequin assigned terryjreedy May 3, 2021
    @sai sai mannequin added topic-IDLE and removed extension-modules C modules in the Modules dir OS-windows labels May 3, 2021
    @sai sai mannequin assigned terryjreedy May 3, 2021
    @terryjreedy terryjreedy added extension-modules C modules in the Modules dir OS-windows and removed topic-IDLE labels May 3, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes extension-modules C modules in the Modules dir OS-windows release-blocker type-crash A hard crash of the interpreter, possibly with a core dump
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @ambv @gpshead @terryjreedy