classification
Title: co_stacksize estimate can be highly off
Type: resource usage Stage: resolved
Components: Interpreter Core Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, benjamin.peterson, inada.naoki, nascheme, pitrou, serhiy.storchaka, yselivanov
Priority: normal Keywords: patch

Created on 2015-05-31 18:29 by arigo, last changed 2018-01-11 19:28 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1938 closed serhiy.storchaka, 2017-06-04 09:38
PR 5076 merged serhiy.storchaka, 2018-01-01 20:06
Messages (8)
msg244552 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2015-05-31 18:29
The computation of `co_stacksize' by the compiler is known to give only an upper bound estimate.  http://bugs.python.org/issue1754094 is an example of fixing a "leak" where every repetition of a statement makes `co_stacksize' bigger by 1.  However, in the whole 3.x series (from at least 3.2), the following simple code, which at runtime requires about 4 or 5 stack slots, leaks 14(!) slots for every `try:' block.  Maybe this should be improved to reduce the extreme size of the leak.

def g():
    try: pass
    except ImportError as e: pass
    try: pass
    except ImportError as e: pass
    try: pass
    except ImportError as e: pass
    ...

i.e. any function that is big enough to contain 6 try: blocks in sequence will have its stack size overestimated by about 70.
msg295118 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-04 09:44
PR 1938 tries to balance the stack effect computation for try/except and try/finally. I'm not sure that it is worth backporting.

Noised Antoine and Benjamin as people related to issue3021.
msg295119 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-04 09:45
I'm against backporting performance improvements which don't fix a severe regression.
msg295122 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-04 11:06
This isn't so easy. Seems the simplest way to solve this issue is implementing issue17611.
msg305677 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-06 21:03
The WIP pull request PR# 2827 seems to help. The following code prints 86 on python3.6 and 25 with PR 2827 applied.

def g():
    try: pass
    except ImportError as e: pass
    try: pass
    except ImportError as e: pass
    try: pass
    except ImportError as e: pass
    try: pass
    except ImportError as e: pass
    try: pass
    except ImportError as e: pass
    try: pass
    except ImportError as e: pass
print(g.__code__.co_stacksize)
msg309346 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-01-01 20:08
With PR 5076 the result of the above example is 10.
msg309347 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-01-01 20:10
Tests originally based on Antoine's tests added for PR 2827.
msg309733 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-01-09 19:54
New changeset d4864c61e3e27e337762dc45e504977299bd5b46 by Serhiy Storchaka in branch 'master':
bpo-24340: Fix estimation of the code stack size. (#5076)
https://github.com/python/cpython/commit/d4864c61e3e27e337762dc45e504977299bd5b46
History
Date User Action Args
2018-01-11 19:28:17serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-01-09 19:54:55serhiy.storchakasetmessages: + msg309733
2018-01-03 18:24:32serhiy.storchakalinkissue31113 dependencies
2018-01-03 18:19:39serhiy.storchakalinkissue32455 dependencies
2018-01-01 20:13:02serhiy.storchakalinkissue17611 dependencies
2018-01-01 20:10:37serhiy.storchakasetmessages: + msg309347
2018-01-01 20:08:55serhiy.storchakasetmessages: + msg309346
2018-01-01 20:06:58serhiy.storchakasetkeywords: + patch
pull_requests: + pull_request4950
2017-11-06 21:03:15naschemesetmessages: + msg305677
2017-11-06 19:51:18naschemesetnosy: + nascheme
2017-08-30 12:26:52serhiy.storchakasetpull_requests: - pull_request3289
2017-08-30 12:18:41serhiy.storchakasetpull_requests: + pull_request3289
2017-06-04 11:06:19serhiy.storchakasetmessages: + msg295122
2017-06-04 09:45:47pitrousetmessages: + msg295119
2017-06-04 09:44:13serhiy.storchakasetversions: - Python 3.6
nosy: + pitrou, benjamin.peterson

messages: + msg295118

stage: needs patch -> patch review
2017-06-04 09:38:43serhiy.storchakasetpull_requests: + pull_request2017
2017-06-04 06:01:43arigosetversions: + Python 3.7
2017-02-21 09:51:04serhiy.storchakasetnosy: + serhiy.storchaka
2017-02-21 08:44:58inada.naokisetnosy: + inada.naoki
2015-06-01 18:28:51arigosetnosy: - arigo
2015-06-01 17:21:00Arfreversetnosy: + Arfrever
2015-06-01 07:23:47pitrousetstage: needs patch
type: resource usage
versions: + Python 3.6
2015-05-31 20:51:18yselivanovsetnosy: + yselivanov
2015-05-31 18:29:54arigocreate