Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refcount issues in import #86160

Closed
serhiy-storchaka opened this issue Oct 10, 2020 · 7 comments
Closed

Refcount issues in import #86160

serhiy-storchaka opened this issue Oct 10, 2020 · 7 comments
Labels
3.9 only security fixes 3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage

Comments

@serhiy-storchaka
Copy link
Member

BPO 41994
Nosy @brettcannon, @ncoghlan, @encukou, @ericsnowcurrently, @serhiy-storchaka, @indygreg
PRs
  • bpo-41994: Fix refcount issues in Python/import.c #22632
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-01-12.14:47:15.737>
created_at = <Date 2020-10-10.11:14:48.425>
labels = ['interpreter-core', '3.9', '3.10', 'performance']
title = 'Refcount issues in import'
updated_at = <Date 2021-09-28.01:02:00.403>
user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2021-09-28.01:02:00.403>
actor = 'indygreg'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-12.14:47:15.737>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2020-10-10.11:14:48.425>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 41994
keywords = ['patch']
message_count = 7.0
messages = ['378384', '384943', '384947', '402688', '402696', '402698', '402755']
nosy_count = 6.0
nosy_names = ['brett.cannon', 'ncoghlan', 'petr.viktorin', 'eric.snow', 'serhiy.storchaka', 'indygreg']
pr_nums = ['22632']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue41994'
versions = ['Python 3.9', 'Python 3.10']
    @serhiy-storchaka
    Copy link
    Member Author

    There is a reference leak in import_add_module(). PyDict_GetItemWithError() returns a borrowed reference, but PyObject_GetItem() return a non-borrowed reference. If sys.modules is not a dict, there is a reference leak. import_add_module() and several other function which return a borrowed reference should be made returning a non-borrowed reference, because there are no guaranties that general mapping keeps reference to value.

    It is still not guarantee correctness of PyImport_AddModuleObject().

    @serhiy-storchaka serhiy-storchaka added 3.9 only security fixes 3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage labels Oct 10, 2020
    @encukou
    Copy link
    Member

    encukou commented Jan 12, 2021

    New changeset 4db8988 by Serhiy Storchaka in branch 'master':
    bpo-41994: Fix refcount issues in Python/import.c (GH-22632)
    4db8988

    @serhiy-storchaka
    Copy link
    Member Author

    Thank you Petr!

    @indygreg
    Copy link
    Mannequin

    indygreg mannequin commented Sep 27, 2021

    While testing 3.10.0rc2, I noticed that commit 4db8988 removed _PyImport_FindExtensionObject() from libpython. This breaks both PyOxidizer and py2exe, which rely on this function for a custom implementation of Loader.create_module() to enable loading Windows DLLs from memory.

    It can probably be worked around. But I wanted to note it here in case it was an unwanted regression.

    @encukou
    Copy link
    Member

    encukou commented Sep 27, 2021

    Please don't use private API. The underscore at the beginning marks the function as private.
    (This has been the case for a long time; in 3.10 it is finally officially documented: https://docs.python.org/3.10/c-api/stable.html#stable )

    It has been removed in alpha5; I'm afraid rc2 is too late to put it back. (Especially since we'd need to put it back as public API, as CPython doesn't need it any more.)

    Please, if you see the need for any other private APIs, could you file bugs to make them public (or better, to support the use case they're needed for)? If they are useful, they should have documentation, tests, and better backwards compatibility expecations.

    @serhiy-storchaka
    Copy link
    Member Author

    _PyImport_FindExtensionObject is a private API. It was added in 3.3 because import.c and importdl.c needed to share code. Since 3.5 it was only used in import.c, so there is no longer need to expose it. It was removed in 3.10 because there was an issue with this API: it returned a borroved reference which could be invalid at the time of returning.

    I can restore and fix _PyImport_FindExtensionObject in 3.10, but is not it too later to do after 3.10.0rc2?

    If it is essential to you, propose to add a new public API (perhaps better designed).

    @indygreg
    Copy link
    Mannequin

    indygreg mannequin commented Sep 28, 2021

    I didn't want to derail this old issue too much. So I filed bpo-45307 to track a solution.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    This was referenced Jun 19, 2023
    Closed
    Closed
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes 3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @encukou @serhiy-storchaka