Remove the usage of the C stack in Python to Python calls #89419

pablogsal · 2021-09-21T10:42:00Z

BPO	45256
Nosy	@gvanrossum, @pmp-p, @markshannon, @serhiy-storchaka, @zooba, @ctismer, @corona10, @pablogsal, @brandtbucher, @Fidget-Spinner, @thesamesam
PRs	bpo-45256: Remove the usage of the C stack in Python to Python calls #28488 bpo-45256: Small cleanups for the code that inlines Python-to-Python calls in ceval.c #28836 bpo-45256: Fix cleanup of stolen locals for Python-to-Python calls #28905 bpo-45256: Avoid C calls for more Python to Python calls. #28937 bpo-45256: Rationalize code around Python-to-Python calls a bit. #29235 bpo-45256: Don't call at the C level when calling bound-methods from Python code. #29238 bpo-45256: Don't track the exact depth of each `InterpreterFrame` #30372

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-06.12:52:24.256>
created_at = <Date 2021-09-21.10:41:59.834>
labels = ['3.11']
title = 'Remove the usage of the C stack in Python to Python calls'
updated_at = <Date 2022-01-06.12:52:24.256>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2022-01-06.12:52:24.256>
actor = 'Mark.Shannon'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-06.12:52:24.256>
closer = 'Mark.Shannon'
components = []
creation = <Date 2021-09-21.10:41:59.834>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 45256
keywords = ['patch']
message_count = 21.0
messages = ['402311', '402429', '402430', '402485', '402491', '402504', '402518', '402546', '402763', '402766', '402767', '402780', '402785', '403538', '403543', '403832', '404166', '405109', '405193', '406462', '409753']
nosy_count = 11.0
nosy_names = ['gvanrossum', 'pmpp', 'Mark.Shannon', 'serhiy.storchaka', 'steve.dower', 'Christian.Tismer', 'corona10', 'pablogsal', 'brandtbucher', 'kj', 'thesamesam']
pr_nums = ['28488', '28836', '28905', '28937', '29235', '29238', '30372']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue45256'
versions = ['Python 3.11']

Linked PRs

The text was updated successfully, but these errors were encountered:

pablogsal · 2021-09-21T10:42:00Z

Removing the usage of the C stack in Python-to-Python calls will allow future optimizations in the eval loop to take place and can yield some speed ups given that we will be removing C function calls and preambles by inlining the callee in the same eval loop as the caller.

serhiy-storchaka · 2021-09-22T12:52:09Z

AFAIK Mark Shannon proposed this idea, but it was rejected.

pablogsal · 2021-09-22T13:04:35Z

What was rejected was https://www.python.org/dev/peps/pep-0651/ which included this idea but had a lot more stuff in it. In particular, it was rejected because it gave semantics to overflow exceptions (two exceptions were proposed), new APIs and it had lack consistent guarantees for different platforms, among other considerations.

This version is just for optimization purposes with no changes in semantics.

ctismer · 2021-09-23T09:39:02Z

Hey guys, you know that you are about to implement the core idea of Stackless Python, right? :-D

markshannon · 2021-09-23T11:05:30Z

I've trying to do this since about 2011 :)

zooba · 2021-09-23T15:59:04Z

I fully support implementing the core idea of Stackless Python :)

I spent a whole EuroPython a couple of years back discussing the idea with (apparently) everyone except Mark.

Though I wouldn't like to lose the ability to extract the Python stack by inspecting native memory alone. That is very useful for debugging, particularly when you've only got a crash dump. So provided all the code objects are only an indirection or two away from a local (or better yet, parameter) value, it should be fine.

pablogsal · 2021-09-23T18:01:18Z

Though I wouldn't like to lose the ability to extract the Python stack by inspecting native memory alone.

Don't worry about it, I am personally making sure that keeps being possible. it will need some changes in the tools, but not any more that any other changes between minor versions of the interpreter

ctismer · 2021-09-24T09:16:19Z

FYI., in Stackless Python I built a chain of frames by a double linked list. This was the replacement for the current frame/stack mix.
Debugging was still quite easy, following this frame chain.

Note that it is a rather easy step to add the capability to squirrel the whole current chain away and replace it by another one. Add some header to such a chain, and you can call it "Tasklet".

markshannon · 2021-09-28T10:03:22Z

PR 28488 has no NEWS entry, or What's New entry.

However, adding multiple entries will be confusing, so that's best left until all calls to Python functions and method don't use the C stack.

ctismer · 2021-09-28T11:11:32Z

FWIW, getting all function to avoid the C stack will most probably take a long time, if it happens at all. Especially functions which might call into Python multiple times must be re-designed heavily, turned into multiple pieces to be in tail position. Been there, long ago... :)

pablogsal · 2021-09-28T11:41:33Z

We are starting to optimize first the easy cases of CALL_FUNCTION and move slowly to add more and more calls into it. This approach has the advantage that allow us to take it in small increments.

ctismer · 2021-09-28T14:57:26Z

Very much appreciated approach.
Too bad that things stop when people are writing extensions as usual. Or do you think we can teach them how to avoid the C stack? Anyway, good luck!

zooba · 2021-09-28T16:09:13Z

The goal is reduced stack depth, not reframing the entire call model around not having a C stack.

We can't even reasonably rewrite getattr() without supporting callbacks from C into Python, so further generalisation is very unlikely.

But if you inspect the native stack of most Python programs, you'll see that it's mostly taken up with calls within Python code. Compressing all of those is a significant advantage, akin to inlining the Python code at compile time, even if it doesn't see "through" native calls.

pablogsal · 2021-10-09T15:51:34Z

New changeset b4903af by Pablo Galindo Salgado in branch 'main':
bpo-45256: Remove the usage of the C stack in Python to Python calls (GH-28488)
b4903af

pablogsal · 2021-10-09T16:52:09Z

New changeset 543acbc by Pablo Galindo Salgado in branch 'main':
bpo-45256: Small cleanups for the code that inlines Python-to-Python calls in ceval.c (GH-28836)
543acbc

markshannon · 2021-10-13T13:38:50Z

New changeset 3901c08 by Pablo Galindo Salgado in branch 'main':
bpo-45256: Fix cleanup of stolen locals for Python-to-Python calls (GH-28905)
3901c08

markshannon · 2021-10-18T08:57:29Z

New changeset 70945d5 by Mark Shannon in branch 'main':
bpo-45256: Avoid C calls for most Python to Python calls. (GH-28937)
70945d5

pablogsal · 2021-10-27T16:22:27Z

Unfortunately, seems that #28937 has broken the AMD64 FreeBSD Shared 3.x buildbot:

https://buildbot.python.org/all/#/builders/483/builds/1003/steps/5/logs/stdio

The buildbot was green until we merged this

markshannon · 2021-10-28T15:15:07Z

New changeset 7f61d9d by Mark Shannon in branch 'main':
bpo-45256: Rationalize code around Python-to-Python calls a bit. (GH-29235)
7f61d9d

markshannon · 2021-11-17T09:52:38Z

https://bugs.python.org/issue45829 is the related issue for special methods

markshannon · 2022-01-05T11:30:34Z

New changeset 332e6b9 by Brandt Bucher in branch 'main':
bpo-45256: Don't track the exact depth of each InterpreterFrame (GH-30372)
332e6b9

…frame is available (#100611)

… no frame is available ``` Unable to read information on python frame Python Exception <class 'AttributeError'>: 'NoneType' object has no attribute 'co_name' ``` Regression in commit b4903af. While refactoring the code into a while loop, the previous early return when no frame exists went missing. We have just printed a message that we cannot get information about this, so the frame will be None, and we cannot attempt to use it. Discovered on python 3.11, in python 3.12a2 this should error out with `.is_shim()` instead of `co_name`. (cherry picked from commit 8586949)

… no frame is available (pythonGH-100611) ``` Unable to read information on python frame Python Exception <class 'AttributeError'>: 'NoneType' object has no attribute 'co_name' ``` Regression in commit b4903af. While refactoring the code into a while loop, the previous early return when no frame exists went missing. We have just printed a message that we cannot get information about this, so the frame will be None, and we cannot attempt to use it. Discovered on python 3.11, in python 3.12a2 this should error out with `.is_shim()` instead of `co_name`. (cherry picked from commit 8586949)

…n no frame is available (GH-100611) (#100738) gh-89419: gdb: fix bug causing AttributeError in py-locals when no frame is available (GH-100611) ``` Unable to read information on python frame Python Exception <class 'AttributeError'>: 'NoneType' object has no attribute 'co_name' ``` Regression in commit b4903af. While refactoring the code into a while loop, the previous early return when no frame exists went missing. We have just printed a message that we cannot get information about this, so the frame will be None, and we cannot attempt to use it. Discovered on python 3.11, in python 3.12a2 this should error out with `.is_shim()` instead of `co_name`. (cherry picked from commit 8586949)

* Rename Lib/test/crashers/ to Lib/test/test_crashers/. * Move Lib/test/test_crashers.py to Lib/test/test_crashers/__init__.py. * test_crashers is no longer skipped and makes sure that scripts do crash, and no simply fail with a non-zero exit code. * Update bogus_code_obj.py to use CodeType.replace(). * Remove Lib/test/crashers/ scripts which no longer crash: * recursive_call.py: fixed by pythongh-89419 * mutation_inside_cyclegc.py: fixed by pythongh-97922 * trace_at_recursion_limit.py: fixed by Python 3.7

* Rename Lib/test/crashers/ to Lib/test/test_crashers/. * Move Lib/test/test_crashers.py to Lib/test/test_crashers/__init__.py. * test_crashers is no longer skipped and makes sure that scripts do crash, and no simply fail with a non-zero exit code. * Update bogus_code_obj.py to use CodeType.replace(). * Scripts crashing Python now uses SuppressCrashReport of test.support to not create coredump files. * Remove Lib/test/crashers/ scripts which no longer crash: * recursive_call.py: fixed by pythongh-89419 * mutation_inside_cyclegc.py: fixed by pythongh-97922 * trace_at_recursion_limit.py: fixed by Python 3.7

pablogsal added 3.11 only security fixes labels Sep 21, 2021

vstinner changed the title ~~Remove the usage of the cstack in Python to Python calls~~ Remove the usage of the C stack in Python to Python calls Sep 22, 2021

markshannon closed this as completed Jan 6, 2022

ezio-melotti transferred this issue from another repository Apr 10, 2022

fantix mentioned this issue Aug 24, 2022

edgedb-python test fails with uvloop 0.15.0 edgedb/edgedb#2222

Closed

kumaraditya303 mentioned this issue Sep 8, 2022

Undefined behaviour in main and 3.11 #96678

Closed

bedevere-bot mentioned this issue Dec 30, 2022

gh-89419: gdb: fix printing AttributeError in py-locals when no frame is available #100611

Merged

kumaraditya303 pushed a commit that referenced this issue Jan 3, 2023

gh-89419: gdb: fix bug causing AttributeError in py-locals when no …

8586949

…frame is available (#100611)

bedevere-bot mentioned this issue Jan 4, 2023

[3.11] gh-89419: gdb: fix bug causing AttributeError in py-locals when no frame is available (GH-100611) #100738

Merged

vstinner mentioned this issue Aug 22, 2023

gh-108297: Update test_crashers #108299

Closed

jackyk02 mentioned this issue Sep 1, 2023

Python 3.11 support lf-lang/lingua-franca#1902

Open

krfricke mentioned this issue Nov 15, 2023

Python 3.11 --native profiling fails because of bpo-45256 benfred/py-spy#634

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove the usage of the C stack in Python to Python calls #89419

Remove the usage of the C stack in Python to Python calls #89419

pablogsal commented Sep 21, 2021 •

edited by bedevere-bot

pablogsal commented Sep 21, 2021

serhiy-storchaka commented Sep 22, 2021

pablogsal commented Sep 22, 2021

ctismer commented Sep 23, 2021

markshannon commented Sep 23, 2021

zooba commented Sep 23, 2021

pablogsal commented Sep 23, 2021

ctismer commented Sep 24, 2021

markshannon commented Sep 28, 2021

ctismer commented Sep 28, 2021

pablogsal commented Sep 28, 2021

ctismer commented Sep 28, 2021

zooba commented Sep 28, 2021

pablogsal commented Oct 9, 2021

pablogsal commented Oct 9, 2021

markshannon commented Oct 13, 2021

markshannon commented Oct 18, 2021

pablogsal commented Oct 27, 2021

markshannon commented Oct 28, 2021

markshannon commented Nov 17, 2021

markshannon commented Jan 5, 2022

Remove the usage of the C stack in Python to Python calls #89419

Remove the usage of the C stack in Python to Python calls #89419

Comments

pablogsal commented Sep 21, 2021 • edited by bedevere-bot

Linked PRs

pablogsal commented Sep 21, 2021

serhiy-storchaka commented Sep 22, 2021

pablogsal commented Sep 22, 2021

ctismer commented Sep 23, 2021

markshannon commented Sep 23, 2021

zooba commented Sep 23, 2021

pablogsal commented Sep 23, 2021

ctismer commented Sep 24, 2021

markshannon commented Sep 28, 2021

ctismer commented Sep 28, 2021

pablogsal commented Sep 28, 2021

ctismer commented Sep 28, 2021

zooba commented Sep 28, 2021

pablogsal commented Oct 9, 2021

pablogsal commented Oct 9, 2021

markshannon commented Oct 13, 2021

markshannon commented Oct 18, 2021

pablogsal commented Oct 27, 2021

markshannon commented Oct 28, 2021

markshannon commented Nov 17, 2021

markshannon commented Jan 5, 2022

pablogsal commented Sep 21, 2021 •

edited by bedevere-bot