classification
Title: subprocess PATH semantics and portability
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.2, Python 3.1, Python 2.7, Python 2.6
process
Status: languishing Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: RubyTuesdayDONO, brian.curtin, dabrahams, docs@python, mark, r.david.murray
Priority: normal Keywords:

Created on 2010-04-28 10:44 by dabrahams, last changed 2014-01-17 02:29 by RubyTuesdayDONO.

Files
File name Uploaded Description Edit
probe.py dabrahams, 2010-05-02 03:43 demonstrates portable Popen wrapper
Messages (17)
msg104422 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-04-28 10:44
On POSIX systems, the PATH environment variable is always used to
look up directory-less executable names passed as the first argument to Popen(...), but on Windows, PATH is only considered when shell=True is also passed.  

Actually I think it may be slightly weirder than that when
shell=False, because the following holds for me:

C:\>rem ##### Prepare minimal PATH #####
C:\>set "PATH=C:\Python26\Scripts;C:\Python26;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem"

C:\>rem ##### Prepare a minimal, clean environment #####
C:\>virtualenv --no-site-packages e:\zzz
New python executable in e:\zzz\Scripts\python.exe
Installing setuptools................done.

C:\>rem ##### Show that shell=True makes the difference in determining whether PATH is respected #####
C:\>python
Python 2.6.5 (r265:79096, Mar 19 2010, 18:02:59) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.Popen(['python', '-c', 'import sys; print sys.executable'])
<subprocess.Popen object at 0x0000000001DBE080>
>>> C:\Python26\python.exe

>>> subprocess.Popen(['python', '-c', 'import sys; print sys.executable'], env={'PATH':r'e:\zzz\Scripts'})
<subprocess.Popen object at 0x0000000001F05A90>
>>> C:\Python26\python.exe

>>> subprocess.Popen(['python', '-c', 'import sys; print sys.executable'], env={'PATH':r'e:\zzz\Scripts'}, shell=True)
<subprocess.Popen object at 0x0000000001F05B00>
>>> e:\zzz\Scripts\python.exe

That is, it looks like the environment at the time Python is invoked is what counts unless I pass shell=True.  I don't even seem to be able to override this behavior by changing os.environ: you can clear() it and pass env={} and subprocess.Popen(['python']) still succeeds.

This is a very important problem for portable code and one that took me hours to suss out.  I think:

a) the current behavior needs to be documented
b) it needs to be fixed if possible
c) otherwise, shell=True should be the default
msg104429 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-04-28 13:22
It's worse than I thought; there isn't even one setting for shell that works everywhere.  This is what happens on POSIX (tested on Mac and Ubuntu):

$ mkdir /tmp/xxx
$ cd /tmp/xxx
xxx $ virtualenv /tmp/zzz
xxx $ python
Python 2.6.5 (r265:79063, Mar 23 2010, 08:10:08) 
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess import *
>>> p = Popen(['python', '-c', 'import sys;print sys.executable'], 
...           stdin=PIPE,stdout=PIPE,stderr=PIPE,
...           env={'PATH':'/tmp/zzz/bin'})
>>> stdout,stderr = p.communicate(None)
>>> print stdout
/tmp/zzz/bin/python

>>> print stderr

>>> p = Popen(['python', '-c', 'import sys;print sys.executable'], shell=True,
...           stdin=PIPE,stdout=PIPE,stderr=PIPE,
...           env={'PATH':'/tmp/zzz/bin'})
>>> stdout,stderr = p.communicate(None)
>>> print stdout

>>> print stderr
msg104437 - (view) Author: Mark Summerfield (mark) Date: 2010-04-28 15:13
IMO there's another problem with subprocess portablity---the lack of control over encodings: see issue 6135.
msg104527 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-04-29 15:49
Changing the default value of shell is not an option anyway.

The behavior you describe is exactly what one should expect: the environment in which the executable is located is the environment of the process calling Popen, not the environment passed to Popen.  The environment passed to Popen is the environment in which the subprocess executes.  When using shell=True, this is the environment in which the shell executes, and the *shell* then looks up the executable in that new environment.  As far as I know this behavior is the same on both Windows and Unix, and thus is not a portability issue.  (How the shell works can be a portability issue, though.)

I'm not sure that this needs to be documented explicitly, as it is a logical consequence of how subprocesses work, but if you want to suggest a doc update I'll take a look at it.

I suspect your Unix example is about the fragility of the rules for computing sys.executable (there's an issue in the tracker about that...you may get a different result on trunk), but I haven't checked it.
msg104611 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-04-30 02:47
I wrote a Python script (enclosed) to methodically test how these things work, that doesn't rely on peculiarities of sys.executable.  The tests did reveal some notable differences on *nix and 'doze:

* When shell=False on windows you must launch the process using a full filename (e.g. "foo.exe", not just "foo", pass --invoke-filename to the script to enable that).  This may seem obvious to you, but for me it was surprising that one executable lookup function (looking in PATH) is in effect but not the other (extending unqualified executable names).  This should be spelled out in the docs.

* On *nix, with shell=False and the executable is neither in the PATH in the environment at the time of Python's launch nor in os.environ at the time of Popen, passing Popen an explicit env whose PATH includes the executable is enough to cause it to be found.  Not so on 'doze. 

* On 'doze, when the executable is in the PATH of os.environ but not in that of Popen's explicit env argument, even with shell=False, no Exception is raised (but returncode is nonzero)
msg104646 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-04-30 15:30
Well, it seems I was mistaken when I thought I knew how this worked :)
Checking the os.exec documentation linked from the subprocess page, I see that when an environment is supplied PATH is indeed checked in it.  The documentation for CreateProcess, however, indicates that PATH is ignored, and any extension must be supplied explicitly.

At the very least the docs should be updated to clarify that execvpe is used when an environment is passed on posix, and to link to the CreateProcess docs.  A discussion of PATH could perhaps be put in a note or footnote (probably footnote, there are enough notes already in those docs!)

I'm not sure how one creates a good portability story out of these pieces.  It doesn't seem as though there is any way to harmonize the two, since we are dealing with the semantics of system calls over which we have no control.

For reference, here is (a?) link to CreateProcess docs that I found via Google:

http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx

It doesn't look like the kind of link that one could trust to be stable, though, so I'm not sure if we should include it in the docs.

I'm adding Brian Curtin as nosy to see if he knows whether or not there are permalink-like links to the relevant Microsoft documentation that we could use.
msg104647 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-04-30 15:49
You could take the "(VS8.5)" part out of the link which will give the latest version, which may not always be the relevant version (although I doubt this specific API would change).

That's about the best permalink-like feature you'll find, but overall, leaving the link as-is is pretty safe in my experience.
msg104738 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-01 19:04
@r.david.murray: did you try running my test?  I think it shows that we are pretty darned close to fully portable.  I believe we could fix Popen to make it fully portable pretty easily.  In fact, there may be a pure-python fix.  Documenting the differences would also not be hard.  I would discourage you from relying *solely* on a description such as "uses execvpe on POSIX" to describe the semantics.  Aside from being a nonportable description, it doesn't help anyone who isn't familiar with the POSIX system calls.
msg104752 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-05-01 21:57
I didn't run the script.  I have now, but I'm not clear from its output what each test is actually doing, and don't really have the time to figure it out from the code right now.

I think it is probably more efficient to just ask you what your suggestion is for making things more portable?

As for the docs, the docs link to the os.exec python docs, which explain the PATH semantics.  Linking to the Microsoft documentation would equivalently explain the Windows semantics.  An explicit footnote discussing the differences in PATH behavior in the subprocess context would probably be helpful.
msg104766 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-02 03:43
I've uploaded a new probe.py that contains a win32 Popen wrapper that I think acts just like *nix's Popen w.r.t. PATH and environment (pass --fix to demonstrate).  I suggest using this or an equivalent wrapper for Win32, and documenting the fact that with shell=False, filename extensions need to be supplied explicitly on windows.
msg104902 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-03 23:02
Not to appear impatient, but...<bump>.
It's a fairly tidy answer, I think :-)
msg104904 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-05-03 23:37
Sorry for my Windows ignorance, but if CreateProcess ignores the PATH, how does updating the PATH fix the problem?
msg104908 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-04 01:22
I'm probably as ignorant as you are of Windows issues. I just know what my experiments tell me: if you force the contents of any explicit 'env' argument into os.environ before calling Popen, you get the same behavior as on *nix.
msg104909 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-05-04 02:37
Well, it wouldn't be the first time the microsoft docs were wrong.

There are two questions here: (1) is this behavior consistent across all microsoft platforms we support?  (2) is this *change* in behavior of Popen acceptable?

For (1) we need a unit test added to the subprocess unit tests that can check this.

For (2)...well, I think it would be good for the behavior to be as consistent as practical, so I'd be in favor.  We need some second opinions, though, to make sure we aren't overlooking some negative consequence.  I'm also not sure that this qualifies as a bug fix, so it may only be possible to get it into 3.2, assuming it is acceptable.

Note that I have not tested your program on Windows myself, I'm taking your word for it that it works ;)  I'll be more inclined to test things if the tests are in the form of unit tests, which should be much easier to understand than your test program.
msg104912 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-04 05:38
R. David Murray wrote:
> There are two questions here: (1) is this behavior consistent across all microsoft platforms we support?  

I'll be honest: I don't know.

> (2) is this *change* in behavior of Popen acceptable?

I don't know that either.

> I'll be more inclined to
> test things if the tests are in the form of unit tests, which should
> be much easier to understand than your test program.

I guess no good deed goes unpunished ;-)

I also guess that whether you think unit tests will be easier to
understand depends on what kind of information you expect to glean
from the code.  My script was designed to probe for all
inconsistencies between ‘doze and POSIX behaviors, and it is more
revealing in that respect than a unit test would be.  The unit test
that would prompt the particular code change I'm suggesting would look
more like:

    put directory X in the env argument's PATH (but not in os.environ)
    attempt to launch X/some_executable as simply “some_executable”
    assert that X/some_executable actually ran

I don't know what Popen's unit tests look like, and to be honest,
right now I just don't have any more time to pour into this particular
issue.  Even if it doesn't get fixed in Python I'm going to be using
my wrapper for uniformity.  I hope everything I've done so far is
useful to the community but if not, I still have what I need.
msg104925 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-05-04 12:06
Fair enough.  Thank you for your detective work, and hopefully someone will be interested enough to pick this up again later.
msg105867 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-16 14:16
New data point: in some contexts on Windows (not sure of the exact cause but I was dealing with multiple drives), even this workaround isn't enough.  I ended up having to do something like this (i.e. manually search the path) on win32:


    def full_executable_path(invoked, environ):

        if os.path.splitext(invoked)[1]:
            return invoked
        
        explicit_dir = os.path.dirname(invoked)

        if explicit_dir:
            path = [ explicit_dir ]
        else:
            path = environ.get('PATH').split(os.path.pathsep)

        extensions = environ.get(
            'PATHEXT',
            # Use *something* in case the environment variable is
            # empty.  These come from my machine's defaults
            '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.PSC1'
            ).split(os.path.pathsep)

        for dir in path:
            for ext in extensions:
                full_path = os.path.join(dir, invoked+ext)
                if os.path.exists( full_path ):
                    return full_path

        return invoked # Not found; invoking it will likely fail

    class Popen(subprocess.Popen):
        def __init__(
            self, args, bufsize=0, executable=None,
            stdin=None, stdout=None, stderr=None,
            preexec_fn=None, close_fds=False, shell=False, 
            cwd=None, env=None, 
            *args_, **kw):

            if executable is None and not shell:
                executable = full_executable_path(args[0], env or os.environ)

            super(Popen,self).__init__(
                args, bufsize, executable, stdin, stdout, stderr, 
                preexec_fn, close_fds, shell, cwd, env, *args_, **kw)
History
Date User Action Args
2014-01-17 02:29:07RubyTuesdayDONOsetnosy: + RubyTuesdayDONO
2010-05-16 14:16:28dabrahamssetmessages: + msg105867
2010-05-04 12:06:32r.david.murraysetstatus: open -> languishing

messages: + msg104925
2010-05-04 05:38:39dabrahamssetmessages: + msg104912
2010-05-04 02:37:55r.david.murraysetmessages: + msg104909
2010-05-04 01:22:54dabrahamssetmessages: + msg104908
2010-05-03 23:37:48r.david.murraysetmessages: + msg104904
2010-05-03 23:02:00dabrahamssetmessages: + msg104902
2010-05-03 23:01:01dabrahamssetfiles: - probe.py
2010-05-02 03:43:26dabrahamssetfiles: + probe.py

messages: + msg104766
2010-05-01 21:57:07r.david.murraysetmessages: + msg104752
2010-05-01 19:04:19dabrahamssetmessages: + msg104738
2010-04-30 15:49:08brian.curtinsetmessages: + msg104647
2010-04-30 15:30:45r.david.murraysetnosy: + brian.curtin
title: subprocess PATH semantics -> subprocess PATH semantics and portability
messages: + msg104646

versions: + Python 3.1, Python 2.7, Python 3.2
stage: needs patch
2010-04-30 02:47:57dabrahamssetfiles: + probe.py

messages: + msg104611
2010-04-29 15:49:44r.david.murraysetnosy: + r.david.murray

messages: + msg104527
title: subprocess portability issue -> subprocess PATH semantics
2010-04-28 15:13:52marksetnosy: + mark
messages: + msg104437
2010-04-28 13:22:12dabrahamssetmessages: + msg104429
2010-04-28 10:44:53dabrahamscreate