classification
Title: PATH is not honored in subprocess.Popen in win32
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.6, Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: subprocess PATH semantics and portability
View: 8557
Assigned To: docs@python Nosy List: Grissiom.Gu, cvrebert, docs@python, eryksun, gjb1002, loewis, martin.panter, sbt, scopatz, terry.reedy
Priority: normal Keywords:

Created on 2012-07-26 00:57 by Grissiom.Gu, last changed 2017-02-04 10:45 by eryksun. This issue is now closed.

Messages (10)
msg166447 - (view) Author: Grissiom Gu (Grissiom.Gu) Date: 2012-07-26 00:57
My system is 32 bit win7 and python is 2.7.2.

Sample code is here:
=======================
import os, sys

nenv = {}
nenv['PATH'] = 'C:\\Windows\\System32\\;e:\\projects\\teest\\python\\exe\\'
nenv['SystemRoot'] = 'C:\\Windows\\'

#os.putenv('PATH', nenv['PATH'])
import subprocess
proc = subprocess.Popen('test.bat', stdin=subprocess.PIPE, stdout=subprocess.PIPE,
    stderr=subprocess.PIPE, shell = False, env=nenv)
data, err = proc.communicate()
rv = proc.wait()

if data:
    print data
if err:
    print err
sys.exit(rv)
==========================
in E:\projects\teest\python I have two folders: 'exe/' have a bat file named 'test.bat' and a 'launch/' folder have the above script. When execute the above script, python yield "WindowsError: [Error 2]" which means could not find the executable.

However, if I un-comment the line "#os.putenv('PATH', nenv['PATH'])", it works like a charm. It seems _subprocess.CreateProcess does not honer the PATH.

FYI, if I use the CreateProcess of win32 extensions, thing works fine without the need to export PATH.
msg166473 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-07-26 11:11
I think env is used for specifying the environment used by the  child process once it has started.  env['PATH'] is not used by the parent process to find program to run.

Having said that, maybe if you used "shell=True" as well then the shell would find the program using env['PATH'].
msg166612 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-07-27 22:15
Richard is correct. From 17.1.1.2. Popen Constructor
"If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of inheriting the current process’ environment, which is the default behavior." 
(followed by Window note about SystemRoot.)

The paragraph about shell=True in 17.1.1.1. Frequently Used Arguments suggests that that might work (but see warning).
msg166715 - (view) Author: Grissiom Gu (Grissiom.Gu) Date: 2012-07-29 04:38
Thank you for everyone participated in this issue. I learnt a lot from you.

But I observe that the same script(with proper modification of file names) works very well under Linux. After I dive into the source code, I found Python use execvpe to invoke the child process which _will_ use the PATH variable to search the executable.

So I think this is a inconsistent behavior at least. So I think we should either notify the user that this API is inconsistent or just fix the Popen call in windows. Prepending the new PATH to the env of current process and than call CreateProcess is just a patch within 10 lines, I think.
msg166761 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-07-29 14:04
> But I observe that the same script(with proper modification of file names) 
> works very well under Linux. After I dive into the source code, I found 
> Python use execvpe to invoke the child process which _will_ use the PATH 
> variable to search the executable.

The posix execvpe function does not use PATH from the passed environment, but (surprisingly to me) os.execvpe does.  subprocess in Python 3 no longer uses os.execvpe, but still goes out of its way to use envp['PATH'] on posix systems when searching for the executable.

So I think you have a legitimate complaint of inconsistency.

However, I don't think your suggested fix is a good idea since it (temporarily) changes the environment of the current process which might be problematic in a threaded program.

I am sure that Python 2.7 will not be patched to address this, but you might want to make a feature request for Python 3.4.
msg166803 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-07-29 19:09
What I understand you two as saying is that there seems to be an undocumented difference in execxxe between unix and windows which has been carried over to subprocess. In particular, for both, the PATH component of the env parameter is used to search for the file on *nix but not on windows, even though this is not part of posix behavior. So reopening -- as a doc issue -- unless an os expert like Martin declares this to be a behavior bug.

Grissiom, can you verify that the difference still exists in 3.2/3?
msg166817 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-07-29 21:51
> What I understand you two as saying is that there seems to be an 
> undocumented difference in execxxe between unix and windows which has been 
> carried over to subprocess.

No.  There is no difference between the platforms in the behaviour of os.execvpe().  os.execvpe() is simply implemented by trying execve() with a list of candidate executable paths until one of them works -- see _execvpe() in os.py.

Whether this difference from the behaviour of Posix's execvpe() was deliberate, I don't know.

The difference between the platforms in Python 2.x is caused by the fact that subprocess uses os.execvpe() on Unix and CreateProcess() on Windows.

In Python 3.x, os.execvpe() is no longer used on Unix, but the old behaviour has been deliberately maintained.  So the difference in behaviour is still present in Python 3.x.
msg286901 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-02-04 00:16
> difference from the behaviour of Posix's execvpe() was deliberate

POSIX doesn't define execvpe [1]. GNU glibc implemented it in 2009 [2]. On Windows, MSC has had execvpe and spawnvpe since at least 5.0 [3], and I think it arrived in 4.0 in 1986. Guido added os._execvpe in 1.2b4 [4] in 1995. I think it's the only implementation of execvpe that searches the passed environment. 

[1]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/execvp.html
[2]: https://sourceware.org/ml/libc-alpha/2009-10/msg00063.html
[3]: https://openlibrary.org/works/OL2028669W
[4]: https://hg.python.org/cpython/file/534a97c400cc/Lib/os.py
msg286930 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-02-04 08:43
Perhaps this is a duplicate of Issue 8557
msg286954 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-02-04 10:45
Sure, let's close this in favor of the older documentation issue 8557.
History
Date User Action Args
2017-02-04 10:45:05eryksunsetstatus: open -> closed
superseder: subprocess PATH semantics and portability
messages: + msg286954

resolution: duplicate
stage: needs patch -> resolved
2017-02-04 08:43:01martin.pantersetnosy: + martin.panter
messages: + msg286930
2017-02-04 00:16:42eryksunsetnosy: + eryksun
messages: + msg286901
2017-02-03 14:58:01scopatzsetnosy: + scopatz

versions: + Python 3.6
2014-09-03 16:54:29terry.reedysetversions: + Python 3.4, Python 3.5, - Python 3.2, Python 3.3
2014-09-03 09:38:58gjb1002setnosy: + gjb1002
2012-07-29 21:51:11sbtsetmessages: + msg166817
2012-07-29 19:09:32terry.reedysetstatus: closed -> open

assignee: docs@python
components: + Documentation
versions: + Python 3.2, Python 3.3
nosy: + loewis, docs@python

messages: + msg166803
resolution: not a bug -> (no value)
stage: resolved -> needs patch
2012-07-29 14:04:54sbtsetmessages: + msg166761
2012-07-29 04:38:36Grissiom.Gusetmessages: + msg166715
2012-07-27 22:15:27terry.reedysetstatus: open -> closed

nosy: + terry.reedy
messages: + msg166612

resolution: not a bug
stage: resolved
2012-07-26 11:11:56sbtsetnosy: + sbt
messages: + msg166473
2012-07-26 07:06:38cvrebertsetnosy: + cvrebert
2012-07-26 00:57:09Grissiom.Gucreate