classification
Title: PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: Perlence, eric.snow, ncoghlan
Priority: normal Keywords: patch

Created on 2017-10-23 11:25 by Perlence, last changed 2017-11-05 04:58 by ncoghlan. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4105 merged ncoghlan, 2017-10-24 11:46
PR 4286 merged ncoghlan, 2017-11-05 04:49
Messages (8)
msg304794 - (view) Author: Sviatoslav Abakumov (Perlence) Date: 2017-10-23 11:25
Setting PYTHONDONTWRITEBYTECODE doesn't seem to have an effect in Python 3.7.0a2:

    $ python -V
    Python 3.7.0a2
    $ env PYTHONDONTWRITEBYTECODE=1 python -c 'import sys; print(sys.dont_write_bytecode)'
    False
msg304796 - (view) Author: Sviatoslav Abakumov (Perlence) Date: 2017-10-23 12:04
Looks like PYTHONOPTIMIZE has no effect either:

    $ python -V
    Python 3.7.0a2
    $ env PYTHONOPTIMIZE=1 python -c 'import sys; print(sys.flags.optimize)'
    0
msg304798 - (view) Author: Sviatoslav Abakumov (Perlence) Date: 2017-10-23 12:43
It seems 1abcf67[1] is the first bad commit.

[1]https://github.com/python/cpython/commit/1abcf6700b4da6207fe859de40c6c1bada6b4fec
msg304899 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-10-24 10:41
Huh, it appears the tests for these features are relying solely on the command line options, and not checking that the environment variables are also setting the flags properly. We should be able to account for that omission with a single new test in `test_cmd_line` that sets every relevant environment variable and checks for the expected `sys.flags` contents.

As far as what's actually going wrong goes, it's this sequence of events in Py_Main:

    _Py_InitializeCore(&core_config);
    ... other code ...
    apply_command_line_and_environment(&cmdline);

_Py_InitializeCore is setting the internal flags appropriately based on the runtime environment, but then Py_Main is stomping over those environmental settings with the settings from the command line.

Historically, these operations happened the other way around, so it was solely up to the code reading the environment variables to ensure they played nice with each other. Now the command line processing logic needs to be updated to also ensure that it only ever increases these values and never reduces them.
msg304955 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-10-25 02:11
New changeset d7ac06126db86f76ba92cbca4cb702852a321f78 by Nick Coghlan in branch 'master':
bpo-31845: Fix reading flags from environment (GH-4105)
https://github.com/python/cpython/commit/d7ac06126db86f76ba92cbca4cb702852a321f78
msg304956 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-10-25 02:14
We could still use some more comprehensive test cases for the env var handling and the way that interacts with the command line settings, but the merged PR includes at least a rudimentary check for the four that directly affect sys.flags without any tricky side effects (PYTHONDEBUG, PYTHONVERBOSE, PYTHONOPTIMIZE, PYTHONDONTWRITEBYTECODE).
msg305582 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-05 04:26
Cross-linking to the work-in-progress RFE that introduced the error: https://bugs.python.org/issue22257
msg305584 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-05 04:58
New changeset 1b46131ae423f43d45947bb48844cf82f6fd82b8 by Nick Coghlan in branch 'master':
bpo-22257: Mention startup refactoring in What's New (GH-4286)
https://github.com/python/cpython/commit/1b46131ae423f43d45947bb48844cf82f6fd82b8
History
Date User Action Args
2017-11-05 04:58:48ncoghlansetmessages: + msg305584
2017-11-05 04:49:12ncoghlansetpull_requests: + pull_request4249
2017-11-05 04:26:51ncoghlansetmessages: + msg305582
2017-10-25 02:14:21ncoghlansetstatus: open -> closed
type: behavior
messages: + msg304956

resolution: fixed
stage: patch review -> resolved
2017-10-25 02:11:34ncoghlansetmessages: + msg304955
2017-10-24 11:46:36ncoghlansetkeywords: + patch
stage: patch review
pull_requests: + pull_request4075
2017-10-24 11:28:00ncoghlansetassignee: ncoghlan
2017-10-24 10:41:13ncoghlansetmessages: + msg304899
2017-10-23 16:47:55serhiy.storchakasetnosy: + ncoghlan
2017-10-23 16:16:11skrahsetnosy: + eric.snow
2017-10-23 12:43:56Perlencesetmessages: + msg304798
2017-10-23 12:04:23Perlencesetmessages: + msg304796
title: Envvar PYTHONDONTWRITEBYTECODE is not respected -> PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect
2017-10-23 11:25:30Perlencecreate