Title: __debug__ is not optimized out at compile time for anything but `if:` and `while:` blocks
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.7, Python 3.6
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: SilentGhost, arigo, eryksun, josh.r, nascheme, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2016-05-31 18:50 by josh.r, last changed 2017-12-18 13:11 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4880 merged serhiy.storchaka, 2017-12-15 09:53
PR 4882 merged serhiy.storchaka, 2017-12-15 11:02
PR 4916 merged serhiy.storchaka, 2017-12-18 12:10
PR 4918 merged python-dev, 2017-12-18 12:30
Messages (7)
msg266769 - (view) Author: Josh Rosenberg (josh.r) * Date: 2016-05-31 18:50
Issue #22091 points out a quirk in the compile function and use of the __debug__ "constant" causing inconsistent behavior when the optimize level of the compile call differs from that of the main interpreter; __debug__ in an `if` or `while` statement is compiled out, but all other uses load it dynamically (at runtime), so in a mixed environment (interpreter at optimization=0, compile at optimize=2), you get non-obvious behavior.

This behavior appears to be a consequence of __debug__ being handled by a special case for the `if` and `while` statements in compile.c that statements of that form to be compiled out, but *not* for similar constructs, e.g. `a if __debug__ else b` and `__debug__ or a` are always evaluated at runtime, whether or not `compile` is involved. The `expr_constant` function here is responsible for this (and only called in the same file, for `if` and `while` statements).

I'm not sure I understand the peephole optimizer, but if it can operate recursively (that is, an initial replacement of A->B where B could be optimized from B->C is optimized in a subsequent pass, turning all uses of A to C eventually), it seems like the "correct" solution would be to piggyback on optimizations for True and False, by having the peephole optimizer replace LOAD_NAME/LOAD_GLOBAL for __debug__ with an appropriate LOAD_CONST, True or False, based on the compile environment. This would fix this bug (making __debug__ evaluate in the `compile` call, so the environment when the compiled code is executed doesn't matter), and it would optimize all the other cases that the current special cases for `if` and `while` don't cover by letting the recursive pass optimize them out the same way uses of literal True and False is optimized, so uses of ternary expressions, non-`if`/`while` boolean operations, and all other operations using __debug__ are optimized using a constant (and possibly optimized out of existence) without needing to independently maintain separate optimizations all over the codebase for __debug__. Might also allow the removal of the explicit special case for __debug__ in compile.c for `if` and `while`, simplifying that code a bit.

This would fix the problem from #22091 and also make __debug__ reliably useful for "optimization", since all uses of it would "compile out" to optimized runtime code, where right now only two cases do so.

Does this seem reasonable?

Note: I added the nosy list from #22091 here, since that bug is really just a special case of this one for `compile` only.
msg266771 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-31 19:09
I think that the complete solution of this problem is making __debug__ a named constant like None, True, False.

This needs changing the grammar of Python.
msg266780 - (view) Author: Josh Rosenberg (josh.r) * Date: 2016-05-31 21:06
That would also work. The argument I'd give in favor of performing a pass that replaces it with a literal True or False is that you don't have update as many places, don't have to worry about missing a place, and you don't have to decide if __debug__ is a reference to True or False, or a new object entirely.

It's just too easy to miss a case where __debug__ should be special and not notice (because optimizations aren't heavily tested for specific byte code outputs or anything), where a missed optimization for the True or False constant is much less likely to go unnoticed.
msg308378 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-12-15 10:35
New changeset 3325a6780c81f1ea51190370b5454879c4862a37 by Serhiy Storchaka in branch 'master':
bpo-27169: The __debug__ constant is now optimized out at compile time. (#4880)
msg308383 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-12-15 11:26
New changeset b82da9ebb20053637e731fd40589e1e52f9f8f6e by Serhiy Storchaka in branch '3.6':
[3.6] bpo-27169: The __debug__ constant is now optimized out at compile time. (GH-4880) (#4882)
msg308550 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-12-18 12:29
New changeset bd6ec4d79e8575df3d08f8a89ba721930032714c by Serhiy Storchaka in branch 'master':
bpo-32365: Fix a reference leak when compile __debug__. (#4916)
msg308552 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-12-18 13:11
New changeset 5659743b5693c9e23313a74117948294e35013f4 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-32365: Fix a reference leak when compile __debug__. (GH-4916) (#4918)
Date User Action Args
2017-12-18 13:11:58serhiy.storchakasetmessages: + msg308552
2017-12-18 12:30:32python-devsetpull_requests: + pull_request4813
2017-12-18 12:29:16serhiy.storchakasetmessages: + msg308550
2017-12-18 12:10:27serhiy.storchakasetpull_requests: + pull_request4811
2017-12-15 11:26:53serhiy.storchakasetstatus: open -> closed
dependencies: - Build-out an AST optimizer, moving some functionality out of the peephole optimizer
resolution: fixed
stage: patch review -> resolved
versions: + Python 3.6
2017-12-15 11:26:28serhiy.storchakasetmessages: + msg308383
2017-12-15 11:02:27serhiy.storchakasetpull_requests: + pull_request4778
2017-12-15 10:35:51serhiy.storchakasetmessages: + msg308378
2017-12-15 09:53:13serhiy.storchakasetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request4774
2017-11-06 19:50:59naschemesetnosy: + nascheme
2017-10-05 07:48:41serhiy.storchakasetdependencies: + Build-out an AST optimizer, moving some functionality out of the peephole optimizer
2017-10-05 07:47:40serhiy.storchakasetassignee: serhiy.storchaka
stage: needs patch
type: behavior
versions: + Python 3.7, - Python 3.6
2016-05-31 21:06:04josh.rsetmessages: + msg266780
2016-05-31 19:09:39serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg266771
2016-05-31 18:50:58josh.rcreate