Title: subprocess PATH semantics and portability
Type: enhancement Stage: needs patch
Components: Documentation Versions: Python 3.10, Python 3.9, Python 3.8
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Anthony Sottile, RubyTuesdayDONO, brian.curtin, dabrahams, docs@python, eryksun, mark, pepalogik, r.david.murray
Priority: normal Keywords:

Created on 2010-04-28 10:44 by dabrahams, last changed 2021-03-05 15:23 by eryksun.

File name Uploaded Description Edit dabrahams, 2010-05-02 03:43 demonstrates portable Popen wrapper
Messages (21)
msg104422 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-04-28 10:44
On POSIX systems, the PATH environment variable is always used to
look up directory-less executable names passed as the first argument to Popen(...), but on Windows, PATH is only considered when shell=True is also passed.  

Actually I think it may be slightly weirder than that when
shell=False, because the following holds for me:

C:\>rem ##### Prepare minimal PATH #####
C:\>set "PATH=C:\Python26\Scripts;C:\Python26;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem"

C:\>rem ##### Prepare a minimal, clean environment #####
C:\>virtualenv --no-site-packages e:\zzz
New python executable in e:\zzz\Scripts\python.exe
Installing setuptools................done.

C:\>rem ##### Show that shell=True makes the difference in determining whether PATH is respected #####
Python 2.6.5 (r265:79096, Mar 19 2010, 18:02:59) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.Popen(['python', '-c', 'import sys; print sys.executable'])
<subprocess.Popen object at 0x0000000001DBE080>
>>> C:\Python26\python.exe

>>> subprocess.Popen(['python', '-c', 'import sys; print sys.executable'], env={'PATH':r'e:\zzz\Scripts'})
<subprocess.Popen object at 0x0000000001F05A90>
>>> C:\Python26\python.exe

>>> subprocess.Popen(['python', '-c', 'import sys; print sys.executable'], env={'PATH':r'e:\zzz\Scripts'}, shell=True)
<subprocess.Popen object at 0x0000000001F05B00>
>>> e:\zzz\Scripts\python.exe

That is, it looks like the environment at the time Python is invoked is what counts unless I pass shell=True.  I don't even seem to be able to override this behavior by changing os.environ: you can clear() it and pass env={} and subprocess.Popen(['python']) still succeeds.

This is a very important problem for portable code and one that took me hours to suss out.  I think:

a) the current behavior needs to be documented
b) it needs to be fixed if possible
c) otherwise, shell=True should be the default
msg104429 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-04-28 13:22
It's worse than I thought; there isn't even one setting for shell that works everywhere.  This is what happens on POSIX (tested on Mac and Ubuntu):

$ mkdir /tmp/xxx
$ cd /tmp/xxx
xxx $ virtualenv /tmp/zzz
xxx $ python
Python 2.6.5 (r265:79063, Mar 23 2010, 08:10:08) 
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess import *
>>> p = Popen(['python', '-c', 'import sys;print sys.executable'], 
...           stdin=PIPE,stdout=PIPE,stderr=PIPE,
...           env={'PATH':'/tmp/zzz/bin'})
>>> stdout,stderr = p.communicate(None)
>>> print stdout

>>> print stderr

>>> p = Popen(['python', '-c', 'import sys;print sys.executable'], shell=True,
...           stdin=PIPE,stdout=PIPE,stderr=PIPE,
...           env={'PATH':'/tmp/zzz/bin'})
>>> stdout,stderr = p.communicate(None)
>>> print stdout

>>> print stderr
msg104437 - (view) Author: Mark Summerfield (mark) * Date: 2010-04-28 15:13
IMO there's another problem with subprocess portablity---the lack of control over encodings: see issue 6135.
msg104527 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-04-29 15:49
Changing the default value of shell is not an option anyway.

The behavior you describe is exactly what one should expect: the environment in which the executable is located is the environment of the process calling Popen, not the environment passed to Popen.  The environment passed to Popen is the environment in which the subprocess executes.  When using shell=True, this is the environment in which the shell executes, and the *shell* then looks up the executable in that new environment.  As far as I know this behavior is the same on both Windows and Unix, and thus is not a portability issue.  (How the shell works can be a portability issue, though.)

I'm not sure that this needs to be documented explicitly, as it is a logical consequence of how subprocesses work, but if you want to suggest a doc update I'll take a look at it.

I suspect your Unix example is about the fragility of the rules for computing sys.executable (there's an issue in the tracker about may get a different result on trunk), but I haven't checked it.
msg104611 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-04-30 02:47
I wrote a Python script (enclosed) to methodically test how these things work, that doesn't rely on peculiarities of sys.executable.  The tests did reveal some notable differences on *nix and 'doze:

* When shell=False on windows you must launch the process using a full filename (e.g. "foo.exe", not just "foo", pass --invoke-filename to the script to enable that).  This may seem obvious to you, but for me it was surprising that one executable lookup function (looking in PATH) is in effect but not the other (extending unqualified executable names).  This should be spelled out in the docs.

* On *nix, with shell=False and the executable is neither in the PATH in the environment at the time of Python's launch nor in os.environ at the time of Popen, passing Popen an explicit env whose PATH includes the executable is enough to cause it to be found.  Not so on 'doze. 

* On 'doze, when the executable is in the PATH of os.environ but not in that of Popen's explicit env argument, even with shell=False, no Exception is raised (but returncode is nonzero)
msg104646 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-04-30 15:30
Well, it seems I was mistaken when I thought I knew how this worked :)
Checking the os.exec documentation linked from the subprocess page, I see that when an environment is supplied PATH is indeed checked in it.  The documentation for CreateProcess, however, indicates that PATH is ignored, and any extension must be supplied explicitly.

At the very least the docs should be updated to clarify that execvpe is used when an environment is passed on posix, and to link to the CreateProcess docs.  A discussion of PATH could perhaps be put in a note or footnote (probably footnote, there are enough notes already in those docs!)

I'm not sure how one creates a good portability story out of these pieces.  It doesn't seem as though there is any way to harmonize the two, since we are dealing with the semantics of system calls over which we have no control.

For reference, here is (a?) link to CreateProcess docs that I found via Google:

It doesn't look like the kind of link that one could trust to be stable, though, so I'm not sure if we should include it in the docs.

I'm adding Brian Curtin as nosy to see if he knows whether or not there are permalink-like links to the relevant Microsoft documentation that we could use.
msg104647 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-04-30 15:49
You could take the "(VS8.5)" part out of the link which will give the latest version, which may not always be the relevant version (although I doubt this specific API would change).

That's about the best permalink-like feature you'll find, but overall, leaving the link as-is is pretty safe in my experience.
msg104738 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-01 19:04
@r.david.murray: did you try running my test?  I think it shows that we are pretty darned close to fully portable.  I believe we could fix Popen to make it fully portable pretty easily.  In fact, there may be a pure-python fix.  Documenting the differences would also not be hard.  I would discourage you from relying *solely* on a description such as "uses execvpe on POSIX" to describe the semantics.  Aside from being a nonportable description, it doesn't help anyone who isn't familiar with the POSIX system calls.
msg104752 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-05-01 21:57
I didn't run the script.  I have now, but I'm not clear from its output what each test is actually doing, and don't really have the time to figure it out from the code right now.

I think it is probably more efficient to just ask you what your suggestion is for making things more portable?

As for the docs, the docs link to the os.exec python docs, which explain the PATH semantics.  Linking to the Microsoft documentation would equivalently explain the Windows semantics.  An explicit footnote discussing the differences in PATH behavior in the subprocess context would probably be helpful.
msg104766 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-02 03:43
I've uploaded a new that contains a win32 Popen wrapper that I think acts just like *nix's Popen w.r.t. PATH and environment (pass --fix to demonstrate).  I suggest using this or an equivalent wrapper for Win32, and documenting the fact that with shell=False, filename extensions need to be supplied explicitly on windows.
msg104902 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-03 23:02
Not to appear impatient, but...<bump>.
It's a fairly tidy answer, I think :-)
msg104904 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-05-03 23:37
Sorry for my Windows ignorance, but if CreateProcess ignores the PATH, how does updating the PATH fix the problem?
msg104908 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-04 01:22
I'm probably as ignorant as you are of Windows issues. I just know what my experiments tell me: if you force the contents of any explicit 'env' argument into os.environ before calling Popen, you get the same behavior as on *nix.
msg104909 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-05-04 02:37
Well, it wouldn't be the first time the microsoft docs were wrong.

There are two questions here: (1) is this behavior consistent across all microsoft platforms we support?  (2) is this *change* in behavior of Popen acceptable?

For (1) we need a unit test added to the subprocess unit tests that can check this.

For (2)...well, I think it would be good for the behavior to be as consistent as practical, so I'd be in favor.  We need some second opinions, though, to make sure we aren't overlooking some negative consequence.  I'm also not sure that this qualifies as a bug fix, so it may only be possible to get it into 3.2, assuming it is acceptable.

Note that I have not tested your program on Windows myself, I'm taking your word for it that it works ;)  I'll be more inclined to test things if the tests are in the form of unit tests, which should be much easier to understand than your test program.
msg104912 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-04 05:38
R. David Murray wrote:
> There are two questions here: (1) is this behavior consistent across all microsoft platforms we support?  

I'll be honest: I don't know.

> (2) is this *change* in behavior of Popen acceptable?

I don't know that either.

> I'll be more inclined to
> test things if the tests are in the form of unit tests, which should
> be much easier to understand than your test program.

I guess no good deed goes unpunished ;-)

I also guess that whether you think unit tests will be easier to
understand depends on what kind of information you expect to glean
from the code.  My script was designed to probe for all
inconsistencies between ‘doze and POSIX behaviors, and it is more
revealing in that respect than a unit test would be.  The unit test
that would prompt the particular code change I'm suggesting would look
more like:

    put directory X in the env argument's PATH (but not in os.environ)
    attempt to launch X/some_executable as simply “some_executable”
    assert that X/some_executable actually ran

I don't know what Popen's unit tests look like, and to be honest,
right now I just don't have any more time to pour into this particular
issue.  Even if it doesn't get fixed in Python I'm going to be using
my wrapper for uniformity.  I hope everything I've done so far is
useful to the community but if not, I still have what I need.
msg104925 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-05-04 12:06
Fair enough.  Thank you for your detective work, and hopefully someone will be interested enough to pick this up again later.
msg105867 - (view) Author: Dave Abrahams (dabrahams) Date: 2010-05-16 14:16
New data point: in some contexts on Windows (not sure of the exact cause but I was dealing with multiple drives), even this workaround isn't enough.  I ended up having to do something like this (i.e. manually search the path) on win32:

    def full_executable_path(invoked, environ):

        if os.path.splitext(invoked)[1]:
            return invoked
        explicit_dir = os.path.dirname(invoked)

        if explicit_dir:
            path = [ explicit_dir ]
            path = environ.get('PATH').split(os.path.pathsep)

        extensions = environ.get(
            # Use *something* in case the environment variable is
            # empty.  These come from my machine's defaults

        for dir in path:
            for ext in extensions:
                full_path = os.path.join(dir, invoked+ext)
                if os.path.exists( full_path ):
                    return full_path

        return invoked # Not found; invoking it will likely fail

    class Popen(subprocess.Popen):
        def __init__(
            self, args, bufsize=0, executable=None,
            stdin=None, stdout=None, stderr=None,
            preexec_fn=None, close_fds=False, shell=False, 
            cwd=None, env=None, 
            *args_, **kw):

            if executable is None and not shell:
                executable = full_executable_path(args[0], env or os.environ)

                args, bufsize, executable, stdin, stdout, stderr, 
                preexec_fn, close_fds, shell, cwd, env, *args_, **kw)
msg262382 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2016-03-24 22:39
Here's the workaround I'm opting for:

if sys.platform =='win32':
    distutils.spawn.find_executable(cmd[0]) + cmd[1:]
msg262399 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-03-25 04:43
As is documented for CreateProcess [1], the search path always includes the following directories:

    * The directory from which the application loaded.
    * The current directory for the parent process.
    * The Windows system directory. Use the
      GetSystemDirectory function to get the path of
      this directory.
    * The 16-bit Windows system directory. There is no
      function that obtains the path of this directory,
      but it is searched. The name of this directory is
    * The Windows directory. Use the GetWindowsDirectory
      function to get the path of this directory.
    * The directories that are listed in the PATH
      environment variable.

The value of PATH comes from the calling process environment, not from the environment passed in the lpEnvironment parameter. If you need to search some other list of paths, you can use shutil.which to find the fully qualified path of the target executable.

Note that in Vista+ you can remove the current directory from the search list by defining the environment variable "NoDefaultCurrentDirectoryInExePath" [2]. 

The following examples show the minimum search path that CreateProcess uses when PATH isn't defined.

    >>> 'PATH' in os.environ

    >>>'python -Sc "import sys; print(sys.prefix)"')
    Breakpoint 0 hit
    00007ff9`cf4b5860 488bc4          mov     rax,rsp
    0:000> du @rcx
    0000006c`a7074410  "C:\Program Files\Python35;.;C:\W"
    0000006c`a7074450  "indows\SYSTEM32\;C:\Windows\syst"
    0000006c`a7074490  "em;C:\Windows"
    0:000> g
    C:\Program Files\Python35

    >>> os.environ['NoDefaultCurrentDirectoryInExePath'] = '1'

    >>>'python -Sc "import sys; print(sys.prefix)"')
    Breakpoint 0 hit
    00007ff9`cf4b5860 488bc4          mov     rax,rsp
    0:000> du @rcx
    0000006c`a6560710  "C:\Program Files\Python35;C:\Win"
    0000006c`a6560750  "dows\SYSTEM32\;C:\Windows\system"
    0000006c`a6560790  ";C:\Windows"
    0:000> g
    C:\Program Files\Python35

Note that in the 2nd case the current directory ('.') is no longer present between the application directory ("C:\Program Files\Python35") and the system directory ("C:\Windows\SYSTEM32\").

CreateProcess executes PE executables and batch files (run via the %ComSpec% interpreter). It automatically appends the .exe extension when searching for an executable. It does this via the lpExtension parameter of SearchPath [3]. 

Some .com files are PE executables (e.g. Otherwise it's not really usefully to loop over the PATHEXT extensions unless you're using shell=True, since most are filetypes that CreateProcess doesn't support [4]. 

[4]: If Microsoft's Windows team cared at all about cross-platform
     idioms they'd add shebang support to CreateProcess, which
     would make all scripts, not just batch files, directly
     executable without requiring ShellExecuteEx and registered
     filetypes. ShellExecuteEx doesn't support a lot of useful
     creation flags that are only available by calling
     CreateProcess directly, and it also doesn't check ACLs to
     prevent executing a file. So scripts are second class
     citizens in Windows, which is why Python has to embed 
     scripts in .exe wrappers.
msg320098 - (view) Author: Jan Lachnitt (pepalogik) Date: 2018-06-20 16:53
A related issue exists with cwd: #15533.
msg388155 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-05 15:23
The Popen() docs begin by explaining that it has "os.execvp()-like" behavior in POSIX and uses CreateProcess() in Windows. Personally, I do not think it's proper for Python's documentation to discuss details of  how CreateProcess() handles lpCommandLine (args), lpApplicationName (executable), lpCurrentDirectory (cwd), and lpEnvironment (env). So maybe all this needs is to clearly map Popen() parameters to the corresponding CreateProcess() parameters.

If Popen() implements a parameter on its own, then it makes sense to me to document the behavior. For example, in POSIX the behavior of `cwd` is implemented by Popen(), and documented as follows:

    In particular, the function looks for executable (or for the first 
    item in args) relative to cwd if the executable path is a relative

This claim is not always true in POSIX since a base filename without a slash in it, which is a relative path, is not searched for in the current directory unless "." is in PATH. But the main problem with the above sentence is the lack of a disclaimer that it only applies to POSIX. In Windows, `cwd` is passed through directly as the lpCurrentDirectory of CreateProcess(). This parameter sets the working directory of the child process and has nothing to do with searching for an executable parsed out of lpCommandLine or resolving a relative path in lpApplicationName. It may affect the result with shell=True, but even in that case there are caveats. Regardless, Python doesn't do anything with `cwd` in Windows except pass it to CreateProcess(), so the cwd -> lpCurrentDirectory parameter mapping is all there is to document.
Date User Action Args
2021-03-05 15:23:08eryksunsetstatus: languishing -> open
type: behavior -> enhancement
messages: + msg388155

versions: + Python 3.8, Python 3.9, Python 3.10, - Python 2.7, Python 3.5, Python 3.6, Python 3.7
2018-06-20 16:53:27pepalogiksetnosy: + pepalogik
messages: + msg320098
2017-02-04 10:48:48eryksunsetversions: + Python 3.5, Python 3.6, Python 3.7, - Python 2.6, Python 3.1, Python 3.2
2017-02-04 10:45:05eryksunlinkissue15451 superseder
2017-02-04 07:48:01martin.panterlinkissue13196 superseder
2016-03-25 04:43:25eryksunsetnosy: + eryksun
messages: + msg262399
2016-03-24 22:39:05Anthony Sottilesetnosy: + Anthony Sottile
messages: + msg262382
2014-01-17 02:29:07RubyTuesdayDONOsetnosy: + RubyTuesdayDONO
2010-05-16 14:16:28dabrahamssetmessages: + msg105867
2010-05-04 12:06:32r.david.murraysetstatus: open -> languishing

messages: + msg104925
2010-05-04 05:38:39dabrahamssetmessages: + msg104912
2010-05-04 02:37:55r.david.murraysetmessages: + msg104909
2010-05-04 01:22:54dabrahamssetmessages: + msg104908
2010-05-03 23:37:48r.david.murraysetmessages: + msg104904
2010-05-03 23:02:00dabrahamssetmessages: + msg104902
2010-05-03 23:01:01dabrahamssetfiles: -
2010-05-02 03:43:26dabrahamssetfiles: +

messages: + msg104766
2010-05-01 21:57:07r.david.murraysetmessages: + msg104752
2010-05-01 19:04:19dabrahamssetmessages: + msg104738
2010-04-30 15:49:08brian.curtinsetmessages: + msg104647
2010-04-30 15:30:45r.david.murraysetnosy: + brian.curtin
title: subprocess PATH semantics -> subprocess PATH semantics and portability
messages: + msg104646

versions: + Python 3.1, Python 2.7, Python 3.2
stage: needs patch
2010-04-30 02:47:57dabrahamssetfiles: +

messages: + msg104611
2010-04-29 15:49:44r.david.murraysetnosy: + r.david.murray

messages: + msg104527
title: subprocess portability issue -> subprocess PATH semantics
2010-04-28 15:13:52marksetnosy: + mark
messages: + msg104437
2010-04-28 13:22:12dabrahamssetmessages: + msg104429
2010-04-28 10:44:53dabrahamscreate