Title: Document startup option/environment interaction
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Todd.Rovito, asvetlov, bukzor, docs@python, python-dev, r.david.murray, terry.reedy
Priority: normal Keywords: patch

Created on 2010-08-12 22:54 by bukzor, last changed 2012-10-17 18:41 by asvetlov. This issue is now closed.

File name Uploaded Description Edit
EnvironmentVariables.patch Todd.Rovito, 2012-10-16 04:13 review
EnvironmentVariables.patch Todd.Rovito, 2012-10-16 05:17 review
Messages (14)
msg113717 - (view) Author: Buck Evan (bukzor) * Date: 2010-08-12 22:54
In our environment, we have a wrapper which enables optimization by default (-OO). Most commandline tools which have a mode-changing flag such as this, also have a flag to do the opposite ( see: ls -t -U, wget -nv -v,  ).

I'd like to implement one or both of:
1) Add a -D option which is the opposite of -O. python -OO -D gives an optimization level of 1.
2) Honor PYTHONOPTIMIZE = 0. At the least, the man page needs to describe how these two methods interact.
msg114021 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-08-15 23:01
How did you conclude that PYTHONOPTIMIZE = 0 is not honored? Can you provide a minimal example or demonstration.

In any case, 2.6.6 is nearly out so bad behavior needs to be demonstrated with 2.7/3.x.
msg116996 - (view) Author: Buck Evan (bukzor) * Date: 2010-09-20 22:59
Minimal demo:

$ python3.1 -OO -c "print(__debug__)"

I've used this code to get the desired functionality:

if [[ $TESTING == 1 || ${PYTHONOPTIMIZE-2} =~ '^(0*|)$' ]]; then
    #someone is requesting no optimization
    export -n PYTHONOPTIMIZE
elif [[ $PYTHONOPTIMIZE ]]; then
    #someone is setting their own optimization
    #optimization by default

exec $INSTALL_BASE/bin/python2.6 $opt "$@"
msg117006 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-09-20 23:39
I am following the suggestion in the last sentence of the original post and turning this into a doc issue. I believe it is standard that command-line switches override environmental variables when there is a conflict, but 'Using Python', section 1.2. 'Environment variables' could gain a sentence that says so, or whatever the case is. If I am correct, "These environment variables influence Python’s behavior." could be followed by "They are processed *before* the command-line switches other than -E."

As I read the doc, PYTHONOPTIMIZE = 0, does not seem to mean to the interpreter what you expect it to. "PYTHONOPTIMIZE  If this is set to a non-empty string it is equivalent to specifying the -O option. If set to an integer, it is equivalent to specifying -O multiple times." (which is to say, equivalent to -OO.) But the doc could be misleading. You can check the source if you wish.

In general there are lots of switches to alter defaults and turn things on but none to restore defaults and turn things off. The one exception is "-E  Ignore all PYTHON* environment variables, e.g. PYTHONPATH and PYTHONHOME, that might be set." The obvious intent is to start fresh and only specify what one wants. I do not think one idiosyncratic wrapper is sufficient reason to add one or more other off switches. There are already about 20 options. If you do not want something turned on, do not turn it on.
msg117010 - (view) Author: Buck Evan (bukzor) * Date: 2010-09-21 00:07
If I understand this code, it means that PYTHONOPTIMIZE set to 1 or 2 works as expected, but set to 0, gives a flag value of 1.

static int
add_flag(int flag, const char *envs)
        int env = atoi(envs);
        if (flag < env)
                flag = env;
        if (flag < 1)
                flag = 1;
        return flag;

Read literally, the man page indicates that any integer value will give a flag value of 2. 

I agree my shell script is probably unusual, but I believe setting this environment value to zero and expecting the feature to be off (given no contradicting options) is reasonable.

I petition to remove the second if statement above (environment value of "0" yields no flag).

I'd also love to provide a numeric argument to -O, to dynamically set this value more readily, but that is lower importance.

I can implement these and run the unit tests if required.
msg117014 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-09-21 01:45
I understood the documentation snippet Terry quoted as saying exactly what you say the code does.  The doc could be clearer by replacing "multiple times" with "that number of times".  Not that more than two has any meaning currently...

Changing the behavior of PYTHONOPTIMIZE as you suggest would be a backward compatibility change that would probably require a deprecation cycle.  (And the message would have to not really be a DeprecationWarning, since DeprecationWarnings are now silent by default but this is a user-triggered event, not a developer-triggered event and so should be user visible.)

Unsetting an environment variable to remove its effect is a fairly common idiom.  So I don't think it is worth the effort of making the change you suggest, even though I agree it would be more intuitive if PYTHONOPTIMIZE=0 meant what you thought it meant.
msg117015 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-09-21 01:49
I do not deny that the behavior described in the doc is a bit strange (keying off of type, ignoring value) is a bit bizarre and contrary to reasonable expectation based on other programs. But to alter behavior before 3.2 or later would require a clearly demonstrated discrepancy between doc and behavior, which I do not see. For startup behavior, some discussion on pydev or python-ideas lists would probably be needed.

If the C code you posted is a quote from the Python source, please give the file path, preferably for current trunk version (the py3k branch). The second 'if' looks a bit strange at first glance, but I will not judge without seeing more.

> I believe setting this environment value to zero and expecting the feature to be off (given no contradicting options) is reasonable.

Optimization is off by default. So there is no need to have a environment value that means to keep it off.  Just do not set it. Or unset it if needed.

That said, the code you showed suggests that the option/environment interaction is more complicated than I thought, at least for this pair.
msg117019 - (view) Author: Buck Evan (bukzor) * Date: 2010-09-21 02:11
"that number of times" isn't exactly accurate either, since "0" is effectively interpreted as "1".

This change would only adversely affect people who use no -O option, set PYTHONOPTIMIZE to '0', and need optimization.
I feel like that falls into the realm of version differences, but that's your decision.
msg117020 - (view) Author: Buck Evan (bukzor) * Date: 2010-09-21 02:20
The file is here:

The second if statement is doing exactly what I find troubling: set the flag even if the incoming value is 0.
I guess this is to handle the empty string case, such as:

msg173014 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2012-10-16 04:13
Applies to Python 3.4 in development and Python 2.7.  I recently ran into a similar conflict where I was overriding a environment variable with a command line option and didn't realize it.  I first checked the documentation and noticed that it was not clear.  Then I found this bug report and made a patch which is attached.  The patch iterates what Terry Reedy suggested which I thought made the documentation clearer.  This is my first patch to Python I hope it is helpful as Python has served me well over the years.
msg173017 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2012-10-16 05:17
After another review I decided to submit a newer patch that uses a comma so I think the documentation flows better.
msg173181 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-10-17 14:33
New changeset 2f812212599b by Andrew Svetlov in branch '2.7':
Issue #9583: Document startup option/environment interaction.

New changeset a8052ad9752b by Andrew Svetlov in branch '3.2':
Issue #9583: Document startup option/environment interaction.

New changeset f0bb61cbc046 by Andrew Svetlov in branch '3.3':
Merge issue #9583: Document startup option/environment interaction.

New changeset def0e0bb06ef by Andrew Svetlov in branch 'default':
Merge issue #9583: Document startup option/environment interaction.
msg173182 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-10-17 14:35
Applied patch.
Thanks, Todd.
I think the issue can be closed.
msg173193 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-10-17 18:41
Close the issue.
Feel free to reopen if clarification needed.
Date User Action Args
2012-10-17 18:41:42asvetlovsetstatus: open -> closed
resolution: fixed
messages: + msg173193

stage: needs patch -> resolved
2012-10-17 14:35:27asvetlovsetnosy: + asvetlov

messages: + msg173182
versions: + Python 3.2, Python 3.3
2012-10-17 14:33:41python-devsetnosy: + python-dev
messages: + msg173181
2012-10-16 05:17:23Todd.Rovitosetfiles: + EnvironmentVariables.patch
type: enhancement
messages: + msg173017
2012-10-16 04:13:36Todd.Rovitosetfiles: + EnvironmentVariables.patch
versions: + Python 3.4, - Python 3.1, Python 3.2
nosy: + Todd.Rovito

messages: + msg173014

keywords: + patch
2010-09-21 02:20:27bukzorsetmessages: + msg117020
2010-09-21 02:11:36bukzorsetmessages: + msg117019
2010-09-21 01:50:00terry.reedysetmessages: + msg117015
2010-09-21 01:45:03r.david.murraysetnosy: + r.david.murray
messages: + msg117014
2010-09-21 00:07:22bukzorsetmessages: + msg117010
2010-09-20 23:40:38terry.reedysettitle: PYTHONOPTIMIZE = 0 is not honored -> Document startup option/environment interaction
nosy: + docs@python

assignee: docs@python
components: + Documentation, - Interpreter Core
type: behavior -> (no value)
2010-09-20 23:39:45terry.reedysetmessages: + msg117006
stage: needs patch
2010-09-20 22:59:20bukzorsetmessages: + msg116996
2010-09-20 21:29:03eric.araujosetversions: + Python 3.1
2010-08-15 23:01:08terry.reedysetnosy: + terry.reedy

messages: + msg114021
versions: + Python 2.7, Python 3.2, - Python 2.6
2010-08-12 22:54:27bukzorcreate