classification
Title: custom PYTHONPATH may break apps embedding Python
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Kevin.Harms, RomainGeissler1A, brett.cannon, dmalcolm, docs@python, eric.araujo, jankratochvil, meador.inge, ncoghlan, r.david.murray, vstinner
Priority: normal Keywords:

Created on 2012-05-29 19:56 by jankratochvil, last changed 2019-05-27 14:51 by vstinner. This issue is now closed.

Messages (16)
msg161906 - (view) Author: Jan Kratochvil (jankratochvil) Date: 2012-05-29 19:56
People use custom Python builds setting PYTHONHOME and PYTHONPATH to these custom builds. This may be expected to break running system Python binary but it also unexpectedly breaks various applications which also embed Python:

$ echo foo >site.py; export PYTHONPATH=.
$ gdb
Traceback (most recent call last):
  File "./site.py", line 1, in <module>
    foo
NameError: name 'foo' is not defined
$ _

It is not obvious to the user who even already forgot about custom PYTHONPATH why GDB breaks.

Standard Fedora applications may link with system libpython incompatible with the PYTHONHOME/PYTHONPATH files for the other Python version. Python does not keep the scripts too compatible across Python versions.

This issues was discussed first for GDB upstream:
[RFA] ignore PYTHONHOME environment variable.
http://sourceware.org/ml/gdb-patches/2010-11/msg00328.html
http://sourceware.org/ml/gdb-patches/2010-12/msg00214.html
http://sourceware.org/ml/gdb-patches/2011-01/msg00307.html
http://sourceware.org/ml/gdb-patches/2012-05/msg00710.html
(first mail of the thread in each month)
Then also in Fedora:
https://fedorahosted.org/fesco/ticket/858
This Python issue would not be filed by the user if it gets resolved:
Issue12140

Exactly the same problem exists for many other apps, yum for all

$ echo foo >site.py; PYTHONPATH=. yum list bash
Traceback (most recent call last):
  File "./site.py", line 1, in <module>
    foo
NameError: name 'foo' is not defined
$ _

Various proposal have been, applicable possibly in some combination together:

 * Ignore PYTHONHOME/PYTHONPATH for embedded applications. (dot)
 * Ignore PYTHONHOME, use only GDB_PYTHONHOME (if set). This proposal is not applicable for all applications in general.
 * Print warning if PYTHONHOME/PYTHONPATH is set.
 * Recover from Python startup failure and suppress the environment vars.
   or Recover from Python startup failure and disable the Python support, if possible.
 * Should the special care affect PYTHONHOME? Differently for PYTHONPATH? What about PYTHONVERBOSE and other PYTHON* settings? 

How should get PYTHONHOME/PYTHONPATH inherited into child processes, if any (debuggees in the GDB case).

Another question is whether the behavior should be the same if upstream application has been linked with non-system libpython location.
msg161907 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-05-29 19:58
An alternate idea would be to recommend more strongly in the doc that people should not shadow standard library module names (in your example, site).
msg161908 - (view) Author: Jan Kratochvil (jankratochvil) Date: 2012-05-29 20:10
This site.py is only an example how it can happen.  In reality it is due to .py files intended for incompatible version of Python.  I am not a Python programmer to have some appropriate incompatible code at hand.
msg161909 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-05-29 20:12
> This site.py is only an example how it can happen.
site is special, as it’s one of the modules imported during Python startup.  Can you reproduce the bug with the same file named os.py?  (I expect you can.)  Can you reproduce with spam.py?  (I think you can’t.)
msg161911 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-05-29 20:19
I think it is up to each embedding application to decide if it wants to respect the environment variables or not.  I don't think that's a decision that core can make.

Now, if we aren't providing an easy way for an embedding application to make that decision, that might be something we could fix.  But I suspect we are (I don't know the C API well enough to say for sure).
msg161912 - (view) Author: Jan Kratochvil (jankratochvil) Date: 2012-05-29 20:24
I fully agree with site.py/os.py/spam.py but I find it offtopic for this Issue.

I do not find too important if some unsetenv/setenv gets called by the app or by libpython.  But the rules should apply for every embedded app the same, the Fedora ticket is suggesting new PEP (Informational) for it.
msg161913 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-05-29 20:31
An informational PEP sounds like exactly what I said: each application needs to decide.  The PEP would provide the information on which to base that decision, and suggestions for how to do it.

An informational PEP on best practices for embedding sounds like a great idea to me.  I hope someone will write one.
msg161918 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-05-29 22:40
It wouldn't be a PEP, it would be an addition to the embedding docs:
http://docs.python.org/extending/embedding.html
msg161919 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-05-29 22:46
For the more general breakage due to PYTHONHOME and PYTHONPATH, yeah, global environment variables are bad, particularly when an OS relies on tools written in (or embedding) Python.

That's the reason virtualenv (and 3.3's forthcoming venv) are a preferred alternative - they give you a space to play in that shouldn't break your system Python or apps that embed it.
msg161955 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-05-30 14:12
> I fully agree with site.py/os.py/spam.py but I find it offtopic for this Issue.

I don’t understand this message :)  There is nothing to agree with or judge on or off-topic; I was trying to understand the root of the bug and really asking you in good faith to try two things to see if my idea was right.
msg161957 - (view) Author: Jan Kratochvil (jankratochvil) Date: 2012-05-30 14:21
While it should be documented this is not only a docs issue.  It should be solved in some way during runtime.
msg161959 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-05-30 14:32
No it shouldn't.  As mentioned in the Fedora thread you linked, this is no different than the user setting LD_LIBRARY_PATH to something that screws up a system installed program.
msg161970 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-05-30 20:57
If we don't expose the mechanism behind -E to embedding applications via
the C API, then a non-docs change may be needed. However, writing (or at
least trying to write) the relevant docs is a good way to check whether or
not that is the case.
msg194137 - (view) Author: Kevin Harms (Kevin.Harms) Date: 2013-08-01 23:44
As far as I can tell, this issue is not resolved. It would be very nice for the C API to have the equivalent for -E for security reasons. I would like to make a setgid binary and do not want users to be able to set PYTHON* variables to force loading of custom code.
msg328727 - (view) Author: Romain Geissler (RomainGeissler1A) Date: 2018-10-28 18:50
Hi,

Just updating this 6 year old bug report. The last comment says it's not possible to ignore environment when using the C-API. I don't know back then, but today it is possible. I have just built gdb 8.2 against python 3.7.1, and patched gdb with this simple patch, which apparently works:

--- gdb/python/python.c
+++ gdb/python/python.c
@@ -1726,6 +1726,9 @@
 #endif
 #endif

+  // Force using the toolchain python without being troubled by $PYTHONHOME or $PYTHONPATH.
+  Py_IgnoreEnvironmentFlag = 1;
+
   Py_Initialize ();
   PyEval_InitThreads ();

Cheers,
Romain
msg343633 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-27 14:51
This issue has been fixed in 36763 by the PEP 587 which provides a new "Isolated Configuration" and a way to tune the Python configuration without impacting subprocesses. For example, the Isolated Configuration ignores environment variables by default.

In Python 3.7 and older, you can set Py_IgnoreEnvironmentFlag to 1 (the flag exists also in Python 2.7), or even set Py_IsolatedFlag to 1 (new in Python 3.4).
History
Date User Action Args
2019-05-27 14:51:32vstinnersetstatus: open -> closed

nosy: + vstinner
messages: + msg343633

resolution: fixed
stage: needs patch -> resolved
2018-10-28 18:50:52RomainGeissler1Asetnosy: + RomainGeissler1A
messages: + msg328727
2013-08-01 23:44:13Kevin.Harmssetnosy: + Kevin.Harms
messages: + msg194137
2012-05-30 20:57:07ncoghlansetmessages: + msg161970
2012-05-30 14:32:55r.david.murraysetmessages: + msg161959
2012-05-30 14:21:28jankratochvilsetmessages: + msg161957
2012-05-30 14:12:46eric.araujosetmessages: + msg161955
2012-05-29 22:46:05ncoghlansetassignee: docs@python
components: + Documentation, - Interpreter Core
versions: + Python 3.2, Python 3.3
nosy: + docs@python

messages: + msg161919
stage: needs patch
2012-05-29 22:44:54meador.ingesetnosy: + meador.inge
2012-05-29 22:40:54ncoghlansetmessages: + msg161918
2012-05-29 20:31:42r.david.murraysetmessages: + msg161913
2012-05-29 20:24:08jankratochvilsetmessages: + msg161912
2012-05-29 20:19:51r.david.murraysetnosy: + r.david.murray
messages: + msg161911
2012-05-29 20:12:35eric.araujosettitle: custom PYTHONPATH breaks Python-embedded apps -> custom PYTHONPATH may break apps embedding Python
2012-05-29 20:12:12eric.araujosetmessages: + msg161909
2012-05-29 20:10:27jankratochvilsetmessages: + msg161908
2012-05-29 19:59:05eric.araujosetnosy: + brett.cannon, ncoghlan
2012-05-29 19:58:51eric.araujosetnosy: + eric.araujo
messages: + msg161907
2012-05-29 19:56:51jankratochvilcreate