Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[subinterpreters] Add --experimental-isolated-subinterpreters build option #84694

Closed
vstinner opened this issue May 5, 2020 · 10 comments
Closed
Labels
3.10 only security fixes build The build process and cross-build topic-subinterpreters

Comments

@vstinner
Copy link
Member

vstinner commented May 5, 2020

BPO 40514
Nosy @vstinner, @ericsnowcurrently, @corona10
PRs
  • bpo-40514: Add --with-experimental-isolated-subinterpreters #19926
  • [3.9] bpo-40514: Remove --with-experimental-isolated-subinterpreters in 3.9 #20228
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-10-31.22:59:17.920>
    created_at = <Date 2020-05-05.13:13:51.880>
    labels = ['expert-subinterpreters', 'build', '3.10']
    title = '[subinterpreters] Add --experimental-isolated-subinterpreters build option'
    updated_at = <Date 2020-10-31.22:59:17.920>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2020-10-31.22:59:17.920>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-10-31.22:59:17.920>
    closer = 'vstinner'
    components = ['Build', 'Subinterpreters']
    creation = <Date 2020-05-05.13:13:51.880>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 40514
    keywords = ['patch']
    message_count = 6.0
    messages = ['368141', '368156', '368159', '368172', '369401', '370745']
    nosy_count = 3.0
    nosy_names = ['vstinner', 'eric.snow', 'corona10']
    pr_nums = ['19926', '20228']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue40514'
    versions = ['Python 3.10']

    @vstinner
    Copy link
    Member Author

    vstinner commented May 5, 2020

    Modifying Python internals to have one GIL per interpreter (bpo-40512) is a large project which requires to modify many small things (again, see bpo-40512).

    I propose to add a temporary build --experimental-isolated-subinterpreters build option to configure for developers who want to hack on isolated subinterpreters. The intent is to speedup bpo-40512 development by making some practice compromises, to have more time to properly design the real fixes.

    For example, tuple, dict and frame have free lists which are shared by subinterpreters. A practical solution is to simply disable them at build time to avoid the need to have per-interpreter free lists.

    Another example is pymalloc which is shared by all subinterpreters and rely on the unique global interpreter lock (GIL) to protect its internal states. A practical solution is to disable it and force the usage of libc malloc() function instead.

    Some compromosises cannot be the default since they have a significant negative impact on performances. So I propose to add a temporary build option until all these small issues will be fixed.

    @vstinner vstinner added 3.9 only security fixes build The build process and cross-build labels May 5, 2020
    @vstinner
    Copy link
    Member Author

    vstinner commented May 5, 2020

    New changeset c5fa364 by Victor Stinner in branch 'master':
    bpo-40514: Add --with-experimental-isolated-subinterpreters (GH-19926)
    c5fa364

    @vstinner
    Copy link
    Member Author

    vstinner commented May 5, 2020

    --with-experimental-isolated-subinterpreters option name is very long on purpose: to advertize that you must not use it, unless you fully understand its purpose :-)

    I didn't document the build option in Misc/SpecialBuilds.txt or Doc/whatsnew/3.9.rst on purpose: you should not use this special build :-) Basically, this option is only for Eric Snow and me :-D

    @ericsnowcurrently
    Copy link
    Member

    It would probably make sense to remove the build option in the 3.9 release. We can leave it in master, but remove it in the 3.9 branch once it has been created.

    @vstinner vstinner changed the title Add --experimental-isolated-subinterpreters build option [subinterpreters] Add --experimental-isolated-subinterpreters build option May 15, 2020
    @vstinner vstinner changed the title Add --experimental-isolated-subinterpreters build option [subinterpreters] Add --experimental-isolated-subinterpreters build option May 15, 2020
    @vstinner
    Copy link
    Member Author

    New changeset 9512ad7 by Victor Stinner in branch '3.9':
    [3.9] bpo-40514: Remove --with-experimental-isolated-subinterpreters in 3.9 (GH-20228)
    9512ad7

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 5, 2020

    I started to *remove* a few"#ifdef EXPERIMENTAL_ISOLATED_SUBINTERPRETERS":

    • commit 88ec919: Make list free list per-interpreter
    • commit 3744ed2: Make frame free list per-interpreter
    • PR 20645 (under review): Make dict free lists per-interpreter

    @vstinner vstinner added 3.10 only security fixes and removed 3.9 only security fixes labels Jun 5, 2020
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @kumaraditya303
    Copy link
    Contributor

    Reopened for #93185

    @erlend-aasland
    Copy link
    Contributor

    Reopened for #93185

    Why do you need to reopen it? The title says "add --experimental-isolated-subinterpreters build option". If you are to remove it, it can be done in the new issue, no?

    @kumaraditya303
    Copy link
    Contributor

    I reopened it as #93185 uses this issue in title, if that PR will use a new issue, this can be closed.

    @erlend-aasland
    Copy link
    Contributor

    Use #93185 for the removal; there is no need for this to be reopened.

    ericsnowcurrently added a commit that referenced this issue May 27, 2022
    This was added for bpo-40514 (gh-84694) to test out a per-interpreter GIL. However, it has since proven unnecessary to keep the experiment in the repo. (It can be done as a branch in a fork like normal.) So here we are removing:
    
    * the configure option
    * the macro
    * the code enabled by the macro
    ericsnowcurrently added a commit to ericsnowcurrently/cpython that referenced this issue May 28, 2022
    …-93185)
    
    This was added for bpo-40514 (pythongh-84694) to test out a per-interpreter GIL. However, it has since proven unnecessary to keep the experiment in the repo. (It can be done as a branch in a fork like normal.) So here we are removing:
    
    * the configure option
    * the macro
    * the code enabled by the macro
    miss-islington pushed a commit that referenced this issue May 28, 2022
    … (GH-93306)
    
    (cherry picked from commit caa279d)
    
    This was added for bpo-40514 (gh-84694) to test out a per-interpreter GIL.  However, it has since proven unnecessary to keep the experiment in the repo.  (It can be done as a branch in a fork like normal.)  So here we are removing:
    
    * the configure option
    * the macro
    * the code enabled by the macro
    
    Automerge-Triggered-By: GH:ericsnowcurrently
    CAM-Gerlach added a commit to CAM-Gerlach/cpython that referenced this issue Oct 16, 2022
     / pythongh-93306
    
    Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
    CAM-Gerlach added a commit to CAM-Gerlach/cpython that referenced this issue Oct 16, 2022
     / pythongh-93306
    
    Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
    CAM-Gerlach added a commit to CAM-Gerlach/cpython that referenced this issue Oct 16, 2022
     / pythongh-93306
    
    Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
    miss-islington pushed a commit that referenced this issue Oct 17, 2022
    Part of #95913
    Forward port of #93306, which was a backport of #93185, to address #84694
    
    This adds the What's New entry for the removal of the subinterpreter-related env variable, build-time flag, etc. As @ericsnowcurrently  was author of the original changes, I added him as a co-author to the commit.
    
    This addition to the Python 3.11 What's New document were only made to the Python 3.11 branch during the backport process, and not added to the version in `main`. Forward-porting it ensures the docs retain these additions for the future, rather than being lost in a legacy Python versions, allows it to be be edited as part of #95913 , and avoids merge conflicts with routine back-ports of PRs touching it.
    
    I've pulled in the addition exactly as-is with no modifications; any editing will be done in future PRs (and therefore can be reviewed and backported accordingly).
    
    The one other such addition is forward-ported in #98344
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes build The build process and cross-build topic-subinterpreters
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants