Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

help("modules") segfaults on 3.11, MacOS #90099

Closed
iritkatriel opened this issue Nov 30, 2021 · 16 comments
Closed

help("modules") segfaults on 3.11, MacOS #90099

iritkatriel opened this issue Nov 30, 2021 · 16 comments
Labels
3.11 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@iritkatriel
Copy link
Member

BPO 45941
Nosy @tiran, @markshannon, @erlend-aasland, @iritkatriel

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-12-01.21:57:56.526>
created_at = <Date 2021-11-30.22:45:55.399>
labels = ['interpreter-core', 'invalid', 'type-crash', '3.11']
title = 'help("modules") segfaults on 3.11, MacOS'
updated_at = <Date 2021-12-01.21:57:56.525>
user = 'https://github.com/iritkatriel'

bugs.python.org fields:

activity = <Date 2021-12-01.21:57:56.525>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-12-01.21:57:56.526>
closer = 'iritkatriel'
components = ['Interpreter Core']
creation = <Date 2021-11-30.22:45:55.399>
creator = 'iritkatriel'
dependencies = []
files = []
hgrepos = []
issue_num = 45941
keywords = ['3.11regression']
message_count = 16.0
messages = ['407403', '407404', '407405', '407406', '407442', '407443', '407447', '407450', '407451', '407452', '407455', '407477', '407479', '407480', '407481', '407495']
nosy_count = 4.0
nosy_names = ['christian.heimes', 'Mark.Shannon', 'erlendaasland', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue45941'
versions = ['Python 3.11']

@iritkatriel
Copy link
Member Author

Python 3.11.0a2+ (heads/main:8a45ca542a, Nov 30 2021, 22:40:56) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> help("modules")

Please wait a moment while I gather a list of all available modules...

/Users/iritkatriel/src/cpython/Lib/pkgutil.py:92: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP-632 for potential alternatives
__import__(info.name)
/Users/iritkatriel/src/cpython/Lib/pkgutil.py:92: DeprecationWarning: lib2to3 package is deprecated and may not be able to parse Python 3.10+
__import__(info.name)
test_sqlite3: testing with version '2.6.0', sqlite_version '3.32.3'
/Users/iritkatriel/src/cpython/Lib/distutils/command/build_ext.py:13: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
from distutils.sysconfig import customize_compiler, get_python_version
zsh: segmentation fault ./python.exe

@iritkatriel iritkatriel added 3.11 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump labels Nov 30, 2021
@iritkatriel
Copy link
Member Author

On the debugger:

>> help("modules")

Please wait a moment while I gather a list of all available modules...

/Users/iritkatriel/src/cpython/Lib/pkgutil.py:92: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP-632 for potential alternatives
__import__(info.name)
/Users/iritkatriel/src/cpython/Lib/pkgutil.py:92: DeprecationWarning: lib2to3 package is deprecated and may not be able to parse Python 3.10+
__import__(info.name)
test_sqlite3: testing with version '2.6.0', sqlite_version '3.32.3'
/Users/iritkatriel/src/cpython/Lib/distutils/command/build_ext.py:13: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
from distutils.sysconfig import customize_compiler, get_python_version
Process 88596 stopped

  • thread Support "bpo-" in Misc/NEWS #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000100160046 python.exe`_PyObject_GenericGetAttrWithDict(obj=0x0000000106acfa50, name=0x0000000100fe91c0, dict=0x0000000000000000, suppress=0) at object.c:1305:17
    1302 PyDictValues **values_ptr = _PyObject_ValuesPointer(obj);
    1303 if (values_ptr && *values_ptr) {
    1304 if (PyUnicode_CheckExact(name)) {
    -> 1305 assert(_PyObject_DictPointer(obj) == NULL);
    1306 res = _PyObject_GetInstanceAttribute(obj, *values_ptr, name);
    1307 if (res != NULL) {
    1308 goto done;
    Target 0: (python.exe) stopped.
    (lldb) p _PyObject_Dump(name)
    object address : 0x100fe91c0
    object refcount : 165
    object type : 0x10051aec0
    object type name: str
    object repr : 'string'
    (lldb) p _PyObject_Dump(
    _PyObject_DictPointer(obj))
    error: Execution was interrupted, reason: Attempted to dereference an invalid pointer..
    The process has been returned to the state before expression evaluation.
    (lldb) p _PyObject_Dump(obj)
    object address : 0x106acfa50
    object refcount : 7
    object type : 0x105add590
    object type name: _cffi_backend.FFI
    object repr : <_cffi_backend.FFI object at 0x106acfa50>

@iritkatriel iritkatriel changed the title help("modules") segfaults on 3.11 help("modules") segfaults on 3.11, MacOS Nov 30, 2021
@iritkatriel iritkatriel changed the title help("modules") segfaults on 3.11 help("modules") segfaults on 3.11, MacOS Nov 30, 2021
@iritkatriel
Copy link
Member Author

It works for me on 3.10.

@iritkatriel
Copy link
Member Author

It's failing in this assertion which was added in #28802 :

assert(*_PyObject_DictPointer(obj) == NULL);

@markshannon
Copy link
Member

Works for me on Ubuntu.

Python 3.11.0a2+ (heads/main:0aa0bd0563, Dec  1 2021, 11:39:40) [GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help("modules")

[lots of output, but no assertion failure or segfault]

>>

I note that your debug output includes _cffi_backend.FFI object
Is this a cffi bug?

@markshannon
Copy link
Member

When I say "cffi bug", I should say "is cffi not ported to 3.11 yet?"

@erlend-aasland
Copy link
Contributor

Unable to reproduce on macOS 12 with latest 3.11 official build, and my most recent dev build (HEAD at b394af1).

@erlend-aasland
Copy link
Contributor

Also unable to reproduce with HEAD at 49444fb.

@iritkatriel
Copy link
Member Author

I'm using MacOS 11.6.

@erlend-aasland
Copy link
Contributor

Also unable to reproduce with HEAD at 8a45ca5. Perhaps it is related to your build/dev environment?

@erlend-aasland
Copy link
Contributor

I'm unable to reproduce on macOS 11.6 as well (head at 49444fb).

@iritkatriel
Copy link
Member Author

  • thread Support "bpo-" in Misc/NEWS #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x000000010015fd26 python.exe`_PyObject_GenericGetAttrWithDict(obj=0x000000013feb7a50, name=0x0000000100fe9350, dict=0x0000000000000000, suppress=0) at object.c:1305:17
    1302 PyDictValues **values_ptr = _PyObject_ValuesPointer(obj);
    1303 if (values_ptr && *values_ptr) {
    1304 if (PyUnicode_CheckExact(name)) {
    -> 1305 assert(_PyObject_DictPointer(obj) == NULL);
    1306 res = _PyObject_GetInstanceAttribute(obj, *values_ptr, name);
    1307 if (res != NULL) {
    1308 goto done;
    Target 0: (python.exe) stopped.
    (lldb) p obj
    (PyObject *) $0 = 0x000000013feb7a50
    (lldb) p _PyObject_Dump((PyObject
    )obj->ob_type)
    object address : 0x13ff25590
    object refcount : 32
    object type : 0x100518680
    object type name: type
    object repr : <class '_cffi_backend.FFI'>
    (lldb) p (PyObject*)obj->ob_type
    (PyObject *) $1 = 0x000000013ff25590
    (lldb) p (PyObject*)obj->ob_type->tp_dict
    (PyObject *) $2 = 0x000000013fedd910
    (lldb) p _PyObject_Dump((PyObject*)obj->ob_type->tp_dict)
    object address : 0x13fedd910
    object refcount : 1
    object type : 0x100511d50
    object type name: dict
    object repr : {'__new__': <built-in method __new__ of type object at 0x13ff25590>, '__getattribute__': <slot wrapper '__getattribute__' of '_cffi_backend.FFI' objects>, '__init__': <slot wrapper '__init__' of '_cffi_backend.FFI' objects>, 'addressof': <method 'addressof' of '_cffi_backend.FFI' objects>, 'alignof': <method 'alignof' of '_cffi_backend.FFI' objects>, 'def_extern': <method 'def_extern' of '_cffi_backend.FFI' objects>, 'callback': <method 'callback' of '_cffi_backend.FFI' objects>, 'cast': <method 'cast' of '_cffi_backend.FFI' objects>, 'dlclose': <method 'dlclose' of '_cffi_backend.FFI' objects>, 'dlopen': <method 'dlopen' of '_cffi_backend.FFI' objects>, 'from_buffer': <method 'from_buffer' of '_cffi_backend.FFI' objects>, 'from_handle': <method 'from_handle' of '_cffi_backend.FFI' objects>, 'gc': <method 'gc' of '_cffi_backend.FFI' objects>, 'getctype': <method 'getctype' of '_cffi_backend.FFI' objects>, 'init_once': <method 'init_once' of '_cffi_backend.FFI' objects>, 'integer_const': <method 'integer_const' of '_cffi_backend.FFI' objects>, 'list_types': <method 'list_types' of '_cffi_backend.FFI' objects>, 'memmove': <method 'memmove' of '_cffi_backend.FFI' objects>, 'new': <method 'new' of '_cffi_backend.FFI' objects>, 'new_allocator': <method 'new_allocator' of '_cffi_backend.FFI' objects>, 'new_handle': <method 'new_handle' of '_cffi_backend.FFI' objects>, 'offsetof': <method 'offsetof' of '_cffi_backend.FFI' objects>, 'release': <method 'release' of '_cffi_backend.FFI' objects>, 'sizeof': <method 'sizeof' of '_cffi_backend.FFI' objects>, 'string': <method 'string' of '_cffi_backend.FFI' objects>, 'typeof': <method 'typeof' of '_cffi_backend.FFI' objects>, 'unpack': <method 'unpack' of '_cffi_backend.FFI' objects>, 'errno': <attribute 'errno' of '_cffi_backend.FFI' objects>, '__doc__': None, 'NULL': <cdata 'void *' NULL>, 'error': <class 'ffi.error'>, 'CType': <class '_cffi_backend.CType'>, 'CData': <class '_cffi_backend._CDataBase'>, 'buffer': <class '_cffi_backend.buffer'>, 'RTLD_LAZY': 1, 'RTLD_NOW': 2, 'RTLD_GLOBAL': 8, 'RTLD_LOCAL': 4, 'RTLD_NODELETE': 128, 'RTLD_NOLOAD': 16}
    (lldb) p (PyObject*)obj->ob_type->tp_dictoffset
    (PyObject *) $3 = NULL

Because the tp_dictoffset is 0, _PyObject_DictPointer(obj) returns NULL and in the assertion, where the return value is dereferenced, it crashes.

(Note: it's not that the assertion is failing, we don't get that far).

@tiran
Copy link
Member

tiran commented Dec 1, 2021

This looks like an issue with 3rd party software cffi. Are you using a fresh build of cffi? The internal ABI of an alpha build changes very often.

@iritkatriel
Copy link
Member Author

How do I check that?

@tiran
Copy link
Member

tiran commented Dec 1, 2021

The easiest way is to remove and rebuild 3rd party software.

@iritkatriel
Copy link
Member Author

Christian - that's what it was. I uninstalled cffi and now it works.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Development

No branches or pull requests

4 participants