classification
Title: __debug__ is not optimized out at compile time for anything but `if:` and `while:` blocks
Type: behavior Stage: needs patch
Components: Interpreter Core Versions: Python 3.7
process
Status: open Resolution:
Dependencies: 11549 Superseder:
Assigned To: serhiy.storchaka Nosy List: SilentGhost, arigo, eryksun, josh.r, nascheme, serhiy.storchaka
Priority: normal Keywords:

Created on 2016-05-31 18:50 by josh.r, last changed 2017-11-06 19:50 by nascheme.

Messages (3)
msg266769 - (view) Author: Josh Rosenberg (josh.r) * Date: 2016-05-31 18:50
Issue #22091 points out a quirk in the compile function and use of the __debug__ "constant" causing inconsistent behavior when the optimize level of the compile call differs from that of the main interpreter; __debug__ in an `if` or `while` statement is compiled out, but all other uses load it dynamically (at runtime), so in a mixed environment (interpreter at optimization=0, compile at optimize=2), you get non-obvious behavior.

This behavior appears to be a consequence of __debug__ being handled by a special case for the `if` and `while` statements in compile.c that statements of that form to be compiled out, but *not* for similar constructs, e.g. `a if __debug__ else b` and `__debug__ or a` are always evaluated at runtime, whether or not `compile` is involved. The `expr_constant` function here https://hg.python.org/cpython/file/fd0ac7ba091e/Python/compile.c#l3542 is responsible for this (and only called in the same file, for `if` and `while` statements).

I'm not sure I understand the peephole optimizer, but if it can operate recursively (that is, an initial replacement of A->B where B could be optimized from B->C is optimized in a subsequent pass, turning all uses of A to C eventually), it seems like the "correct" solution would be to piggyback on optimizations for True and False, by having the peephole optimizer replace LOAD_NAME/LOAD_GLOBAL for __debug__ with an appropriate LOAD_CONST, True or False, based on the compile environment. This would fix this bug (making __debug__ evaluate in the `compile` call, so the environment when the compiled code is executed doesn't matter), and it would optimize all the other cases that the current special cases for `if` and `while` don't cover by letting the recursive pass optimize them out the same way uses of literal True and False is optimized, so uses of ternary expressions, non-`if`/`while` boolean operations, and all other operations using __debug__ are optimized using a constant (and possibly optimized out of existence) without needing to independently maintain separate optimizations all over the codebase for __debug__. Might also allow the removal of the explicit special case for __debug__ in compile.c for `if` and `while`, simplifying that code a bit.

This would fix the problem from #22091 and also make __debug__ reliably useful for "optimization", since all uses of it would "compile out" to optimized runtime code, where right now only two cases do so.

Does this seem reasonable?

Note: I added the nosy list from #22091 here, since that bug is really just a special case of this one for `compile` only.
msg266771 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-31 19:09
I think that the complete solution of this problem is making __debug__ a named constant like None, True, False.

This needs changing the grammar of Python.
msg266780 - (view) Author: Josh Rosenberg (josh.r) * Date: 2016-05-31 21:06
That would also work. The argument I'd give in favor of performing a pass that replaces it with a literal True or False is that you don't have update as many places, don't have to worry about missing a place, and you don't have to decide if __debug__ is a reference to True or False, or a new object entirely.

It's just too easy to miss a case where __debug__ should be special and not notice (because optimizations aren't heavily tested for specific byte code outputs or anything), where a missed optimization for the True or False constant is much less likely to go unnoticed.
History
Date User Action Args
2017-11-06 19:50:59naschemesetnosy: + nascheme
2017-10-05 07:48:41serhiy.storchakasetdependencies: + Build-out an AST optimizer, moving some functionality out of the peephole optimizer
2017-10-05 07:47:40serhiy.storchakasetassignee: serhiy.storchaka
stage: needs patch
type: behavior
versions: + Python 3.7, - Python 3.6
2016-05-31 21:06:04josh.rsetmessages: + msg266780
2016-05-31 19:09:39serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg266771
2016-05-31 18:50:58josh.rcreate