This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: subprocess.Popen(cwd) documentation: Posix vs Windows
Type: behavior Stage: needs patch
Components: Documentation, Library (Lib), Windows Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: asvetlov, ban, chris.jerdonek, cvrebert, damon-atkins, docs@python, eryksun, gregory.p.smith, ned.deily, paul.moore, pepalogik, python-dev, steve.dower, tim.golden, wolma, zach.ware
Priority: normal Keywords: needs review, patch

Created on 2012-08-02 05:44 by chris.jerdonek, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
issue-15533-test-cases-1.patch chris.jerdonek, 2012-08-27 16:08 review
issue-15533-2-default.patch chris.jerdonek, 2012-08-27 20:23 review
issue-15533-3-default.patch chris.jerdonek, 2012-08-28 00:28 review
issue-15533-4-default.patch asvetlov, 2012-09-03 18:49 review
issue-15533-5-default.patch chris.jerdonek, 2012-09-04 12:50 review
issue-15533-6-default.patch chris.jerdonek, 2012-09-04 13:12 review
issue-15533-7-default.patch chris.jerdonek, 2012-09-04 19:09 review
issue-15533-8-default.patch chris.jerdonek, 2012-09-05 14:25 review
Messages (45)
msg167194 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-02 05:44
The sentence describing Popen()'s cwd argument in the subprocess documentation seems reversed to me:

http://docs.python.org/dev/library/subprocess.html#subprocess.Popen

It says, "If cwd is not None, the child’s current directory will be changed to cwd before it is executed. Note that this directory is not considered when searching the executable, so you can’t specify the program’s path relative to cwd."

However, when cwd is not None, it seems like you *must* specify the program's path relative to cwd.  For example, when running a script containing the following using `./python.exe` from a source checkout--

    p = Popen(['./python.exe', '-V'], stdout=PIPE, stderr=PIPE, cwd='temp')
    
you get an: "OSError: [Errno 2] No such file or directory."

In contrast, when you *do* specify the program's path relative to cwd, it works--

    p = Popen(['../python.exe', '-V'], stdout=PIPE, stderr=PIPE, cwd='temp')

Issue 6374 seems to have made the same point in its second bullet, but the issue was closed without addressing that part of it.
msg169206 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-27 16:08
Attached are a few test cases showing that Popen *does* consider cwd when searching for the executable (as well as for args[0]), and in particular that you *can* specify the program's path relative to cwd.

I also moved the test_cwd test to be adjacent to the other cwd test (the one that tests cwd with the executable argument).

I can also prepare the documentation changes for addition to the patch.
msg169215 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-27 20:23
Here is a full patch for the default branch (documentation correction and test cases for the documented behavior).

If this patch looks acceptable, I can prepare a separate patch for 2.7.
msg169216 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-27 20:37
python_dir = os.path.dirname(os.path.realpath(sys.executable))
wrong_cwd = os.path.join(python_dir, 'Doc')

Actually, is there a better directory to be using for this?  I'd like a directory that is guaranteed to exist that is in the same directory as sys.executable -- so that I can construct a simple relative path from that directory to sys.executable.
msg169236 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-27 22:44
Because tests should be runnable from installed Pythons (including binary -only installations), tests should not assume that a Python source directory is available nor make any assumptions about the location of the Python executable itself.
msg169240 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-28 00:28
Here is a new patch that makes no assumptions about the contents of the directory containing sys.executable.
msg169724 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-09-02 21:13
Maybe better to check cwd in _call_popen_and_assert for child process (like test_cwd does) instead of just checking for successful child execution?
msg169795 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-09-03 18:49
Update patch.
msg169796 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-03 19:08
Thanks, Andrew.  Regarding your comment, it was a deliberate choice not to do the additional check because I wanted each test to check only one thing.  But I am okay with adding the additional check.

Regarding the patch, should all of the methods now do something similar to what test_cwd() does?

+        # We cannot use os.path.realpath to canonicalize the path,
+        # since it doesn't expand Tru64 {memb} strings. See bug 1063571.
+        cwd = os.getcwd()
+        os.chdir(tmpdir)
+        tmpdir = os.getcwd()

It looks like test_cwd() may have needed to apply this treatment to have more reliable string-matching in the assert.  Otherwise, why is tmpdir being re-assigned to?
msg169820 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-09-04 09:27
I believe it's trick for Tru64 platform.
I've asked to support of this in python-dev.
msg169821 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-04 10:23
For future reference, here is the beginning of the e-mail thread on python-dev:

http://mail.python.org/pipermail/python-dev/2012-September/121584.html

We also need to know whether the Tru64 trick needs to be used in 2.7, since this documentation issue also affects 2.7.
msg169824 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-04 12:21
Andrew, I seem to be getting a test failure for test_executable_with_cwd() with your updated patch (the child process is outputting an absolute path rather than '').

I will update the patch to fix.  There are also some stylistic changes I would like to make to the helper method (updated code comment, etc).
msg169825 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-04 12:50
Here is an updated patch.  The changes I made are:

(1) Update code comments in _call_popen_and_assert().
(2) Fix test failure.
(3) Rename _call_popen_and_assert() to _assert_cwd() since it is a simpler
    name and the old name did not reflect that the method is specific to
    the test_cwd_* methods.
(4) Add _split_python_path() helper method so that we do not need to call
    os.path.realpath(sys.executable) in every method.

Andrew, you can make changes re: Tru64 (removing old code, etc).  Thanks.
msg169829 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-04 13:12
Updating the patch again to tweak the original documentation change.

I was concerned that the previous language could be construed to mean that Popen will look in *two* places for the executable (both relative to the current directory and relative to the cwd argument).  The change I'm uploading makes this a little more clear.
msg169838 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-04 19:09
Updating the doc portion of the patch one more time.
msg169871 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-05 14:25
Here is a proposed patch that attempts to minimize the chance of test breakage for Tru64.  The patch follows Martin's recommendation on python-dev of being cautious by following existing code.
msg170291 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-11 11:35
Andrew, do you think my changes to the patch are adequate given the response on python-dev to your question?
msg171611 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-09-30 05:53
Chris, please commit your patch. It's fine for me.
msg171620 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-09-30 09:24
New changeset abfaa4368263 by Chris Jerdonek in branch '3.2':
Issue #15533: Clarify docs and add tests for subprocess.Popen()'s cwd argument.
http://hg.python.org/cpython/rev/abfaa4368263

New changeset f66ff96f0030 by Chris Jerdonek in branch '3.3':
Issue #15533: Merge fix from 3.2.
http://hg.python.org/cpython/rev/f66ff96f0030

New changeset 37f4aa15a1c6 by Chris Jerdonek in branch 'default':
Issue #15533: Merge fix from 3.3.
http://hg.python.org/cpython/rev/37f4aa15a1c6
msg171621 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-30 09:26
I will commit to 2.7 separately.
msg171622 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-30 10:03
Two of the tests fail on at least some of the Windows bots.  I am investigating.

======================================================================
ERROR: test_cwd_with_relative_arg (test.test_subprocess.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\Buildslave\3.2.moore-windows\build\lib\test\test_subprocess.py", line 222, in test_cwd_with_relative_arg
    self._assert_cwd(python_dir, rel_python, cwd=python_dir)
  File "D:\Buildslave\3.2.moore-windows\build\lib\test\test_subprocess.py", line 195, in _assert_cwd
    **kwargs)
  File "D:\Buildslave\3.2.moore-windows\build\lib\subprocess.py", line 745, in __init__
    restore_signals, start_new_session)
  File "D:\Buildslave\3.2.moore-windows\build\lib\subprocess.py", line 964, in _execute_child
    startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

======================================================================
ERROR: test_cwd_with_relative_executable (test.test_subprocess.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\Buildslave\3.2.moore-windows\build\lib\test\test_subprocess.py", line 240, in test_cwd_with_relative_executable
    cwd=python_dir)
  File "D:\Buildslave\3.2.moore-windows\build\lib\test\test_subprocess.py", line 195, in _assert_cwd
    **kwargs)
  File "D:\Buildslave\3.2.moore-windows\build\lib\subprocess.py", line 745, in __init__
    restore_signals, start_new_session)
  File "D:\Buildslave\3.2.moore-windows\build\lib\subprocess.py", line 964, in _execute_child
    startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.2/builds/210
msg171649 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-30 16:33
So it seems the cwd argument to Popen() currently works differently on Windows from Mac OS X.  For example, the following doesn't work on Windows (but does on Mac).  Windows doesn't look for arg0 relative to arg_cwd:

def test_cwd(arg0, arg_cwd):
    os.chdir('foo')  # Make sure we're in a different directory from arg0.
    p = subprocess.Popen([arg0, "-c",
                          "import os, sys; "
                          "sys.stdout.write(os.getcwd()); "
                          "sys.exit(47)"],
                          stdout=subprocess.PIPE,
                          cwd=arg_cwd)
    p.wait()
    print("stdout: " + p.stdout.read().decode("utf-8"))
    print("return_code: %s" % p.returncode)

python_path = os.path.realpath(sys.executable)
python_dir, python_base = os.path.split(python_path)
rel_python = os.path.join(os.curdir, python_base)

# Raises: WindowsError: [Error 2] The system cannot find the file specified
test_cwd(rel_python, python_dir)

I'm going to mark the two tests as "skipped" on Windows pending a resolution.
msg171651 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-09-30 16:49
New changeset d8d52b5b4bc2 by Chris Jerdonek in branch '3.2':
Issue #15533: Skip test_cwd_with_relative_*() tests on Windows pending resolution of issue.
http://hg.python.org/cpython/rev/d8d52b5b4bc2

New changeset 17d709f0b69b by Chris Jerdonek in branch '3.3':
Issue #15533: Merge update from 3.2.
http://hg.python.org/cpython/rev/17d709f0b69b

New changeset d10a7c1ac3a7 by Chris Jerdonek in branch 'default':
Issue #15533: Merge update from 3.3.
http://hg.python.org/cpython/rev/d10a7c1ac3a7
msg171692 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-01 05:28
I propose addressing the remainder of this issue by:

1) Documenting the difference in behavior between Windows and non-Windows, adjusting the tests to reflect this difference, and then closing this issue, and then

2) Creating a new issue to discuss whether and in what version to make the behavior of the cwd argument the same across Windows and non-Windows.
msg177694 - (view) Author: Jan Lachnitt (pepalogik) Date: 2012-12-18 16:16
Hi,

I am using Python 3.2.3 on Windows XP. I encountered a problem with subprocess.call when using the cwd parameter. I used Google to look for a solution and I found this issue in Python tracker. But this issue seems absolutely reversed!

The subprocess documentation says this: "In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path." But this is NOT true. If I use the path relative to cwd, I get Windows Error 2 (system cannot find the file). If I change the executable's path to be relative to the current directory, where the script is running (i.e. NOT the cwd parameter passed to subprocess.call), it works fine.
msg177695 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-12-18 16:38
Hi Jan, yes, the documentation now describes the non-Windows behavior, which is different from the Windows behavior.  See the comment just before yours on what still needs to be done to resolve this issue.  I was in the middle of creating a bunch of test cases for the various combinations, as well as documenting the results.  But I got busy with other things.  I will see if I can revisit my work on this sometime soon and upload a diff.
msg177696 - (view) Author: Jan Lachnitt (pepalogik) Date: 2012-12-18 16:41
EDIT:

No, it doesn't work fine: the executable reports stack overflow. I thought this had nothing to do with Python, hence I didn't mention it. But If I run the executable without Python, THEN it works fine. But this may be another issue.

I'll update to 3.3.0 and then tell you if this has changed.
msg177697 - (view) Author: Jan Lachnitt (pepalogik) Date: 2012-12-18 16:53
Hi Chris, thank for your reply, I didn't see it while writing the edit. Does it mean that on Linux, it will use the cwd?
msg177747 - (view) Author: Jan Lachnitt (pepalogik) Date: 2012-12-19 15:28
Hi all,

I have solved the problem by using absolute path of the executable. The reason why the executable didn't work properly may be that the executable's relative path was inconsistent with current directory. See the following example (I have made an executable which shows its argv and cwd). If it is called normally, then:

argv[0] = phsh0.exe
cwd = D:\Jenda\AutoLEED\TESTING\default

But if it is called by Python's subprocess.call from "D:\Jenda\AutoLEED\TESTING" as I want, then:

argv[0] = default\phsh0.exe
cwd = D:\Jenda\AutoLEED\TESTING\default

The executable may be confused by this inconsistency. So it is not the documentation, but Python itself what should be changed. The executable should be searched in cwd on any platform to avoid the inconsistency.

I have not yet updated my Python installation, so my results apply to 3.2.3.
msg185549 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-03-30 07:19
Note, that test_executable_without_cwd now fails when the tests are run from an installed Python.  See Issue17046.
msg281658 - (view) Author: Wolfgang Maier (wolma) * Date: 2016-11-24 21:42
Ping. This still isn't fixed several years later, i.e., the documentation still describes the POSIX, but not the Windows behavior.
See also issue20927, which reports this as a bug.
msg281686 - (view) Author: Wolfgang Maier (wolma) * Date: 2016-11-25 08:55
Before I forget again what I've gathered yesterday about this issue, here's a summary of the problem:

When the the first element of args or the executable argument of subprocess.Popen does not specify an absolute path, the way the executable gets discovered is platform-dependent.
On POSIX platforms, if the argument contains a directory part, the executable gets looked for relative to the current working directory. If the argument is just a basename, the PATH is searched.
On Windows, the executable argument and the first element of args are treated in different ways: If the executable is specified through the executable argument, it is always looked for relative to the current working directory only, but if it's specified through args, it's searched for using Windows-specific rules as documented for the underlying CreateProcess function (see https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx).

Now, on top of all this, if the cwd argument of Popen is used, then, in Python3 on POSIX-platforms, the current working directory gets changed to cwd *before* the above interpretation happens, i.e., if executable or args[0] contains a dirname, the executable is looked for relative to cwd.
On Windows, however, cwd becomes the current working directory of the new process, but is *not* used during the executable lookup.

I guess with this being rather complicated it would be nice to have a dedicated section explaining the interpretation of relative paths as arguments instead of trying to get only the cwd wording right.
msg282050 - (view) Author: Wolfgang Maier (wolma) * Date: 2016-11-29 22:26
Just found issue15451, which reports a similar inconsistency between Windows and POSIX for 'PATH' provided through the Popen env parameter as for cwd. It seems that, on POSIX-platforms, the PATH environment variable passed through env affects the executable lookup if executable does *not* contain a dirname, but on Windows the new PATH never affects executable lookup. So, again, relative executable paths are causing platform-specific behavior.
msg282052 - (view) Author: Jan Lachnitt (pepalogik) Date: 2016-11-29 23:12
Thank Wolfgang Maier for reminding this issue and providing various details and observations. Having taken a look at my old comments (and at others' comments, too), I feel that the cwd issue deserves a clearer description.

Let's use the following simple C program as the callee:

#include <stdio.h>
#include <unistd.h>
int main(int argc, char* argv[]) {
    char cwd[FILENAME_MAX+1];
    for (int i = 0; i < argc; ++i)
        printf("argv[%d] = %s\n", i, argv[i]);
    getcwd(cwd, FILENAME_MAX);
    printf("cwd = %s\n", cwd);
    return 0;
}

As is evident, this program merely prints its arguments and working directory. I have built it using gcc, called it "print_argv+cwd", and placed it in the "subdir" subdirectory of the current directory.

Next, let's use the following Python 3 script for testing:

import os
from subprocess import run  # substitute run->call in Python < 3.5
prg_name = 'print_argv+cwd'
if os.name == 'nt':
    prg_name += '.exe'
else:
    prg_name = os.path.join('.',prg_name)
dir_name = 'subdir'
def execute(path, cwd):
    print('Executing "{}" in "{}":'.format(path,cwd))
    try:
        run([path], cwd=cwd)  # substitute run->call in Python < 3.5
    except Exception as err:
        print(type(err).__qualname__+':', err)
print('Script\'s cwd =', os.getcwd())
execute(prg_name, dir_name)
execute(os.path.join(dir_name,prg_name), dir_name)
execute(os.path.abspath(os.path.join(dir_name,prg_name)), dir_name)

Output on Linux with Python 3.5.2:

Script's cwd = /home/jenda/Bug reports/Python/subprocess
Executing "./print_argv+cwd" in "subdir":
argv[0] = ./print_argv+cwd
cwd = /home/jenda/Bug reports/Python/subprocess/subdir
Executing "subdir/./print_argv+cwd" in "subdir":
FileNotFoundError: [Errno 2] No such file or directory: 'subdir/./print_argv+cwd'
Executing "/home/jenda/Bug reports/Python/subprocess/subdir/print_argv+cwd" in "subdir":
argv[0] = /home/jenda/Bug reports/Python/subprocess/subdir/print_argv+cwd
cwd = /home/jenda/Bug reports/Python/subprocess/subdir

Output on Windows with Python 3.5.2:

Script's cwd = C:\Users\Jenda\Bug reports\Python\subprocess
Executing "print_argv+cwd.exe" in "subdir":
FileNotFoundError: [WinError 2] Systém nemůže nalézt uvedený soubor
Executing "subdir\print_argv+cwd.exe" in "subdir":
argv[0] = subdir\print_argv+cwd.exe
cwd = C:\Users\Jenda\Bug reports\Python\subprocess\subdir
Executing "C:\Users\Jenda\Bug reports\Python\subprocess\subdir\print_argv+cwd.exe" in "subdir":
argv[0] = C:\Users\Jenda\Bug reports\Python\subprocess\subdir\print_argv+cwd.exe
cwd = C:\Users\Jenda\Bug reports\Python\subprocess\subdir

Summary: On Linux, subprocess.run (or call or Popen) behaves correctly, in accordance with current documentation. On Windows, both possible relative paths produce incorrect results. With the first one, relative to "subdir", Python fails to find the executable. With the other one, relative to the script's cwd, Python actually executes the program, but argv[0] is inconsistent with cwd. Imagine that the called program wants to resolve its own path: It joins cwd and argv[0] and gets "C:\Users\Jenda\Bug reports\Python\subprocess\subdir\subdir\print_argv+cwd.exe", which is an incorrect (and nonexistent) path. This is why the cwd issue is not just a documentation issue.

The only option working correctly on Windows is the last one, using absolute path of the executable.
msg286967 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-02-04 14:55
The Unix implementation of subprocess.Popen follows the behavior of os.execvpe, which is an outlier. Other execvpe implementations, such as the one added to glibc in 2009, search PATH in the current environment instead of the passed environment. As such, and given the natural expectations of a Windows programmer, I do not see the current behavior of the Windows implementation as incorrect. It's a documentation bug.

On a related note, the Popen documentation for Windows should also mention that defining the environment variable NoDefaultCurrentDirectoryInExePath removes the current directory from the executable search path, in both CreateProcess and cmd.exe (i.e. w/ shell=True). This feature was introduced in Windows Vista, so it applies to Python 3.5+. 

> Python actually executes the program, but argv[0] is inconsistent with
> cwd. Imagine that the called program wants to resolve its own path:
> It joins cwd and argv[0] and gets 
> "C:\Users\Jenda\Bug reports\Python\subprocess\subdir\subdir\print_argv+cwd.exe"

A Windows program would call GetModuleFileName with hModule as NULL, which returns the path of the process executable. There's also the pseudo-environment variable __APPDIR__. 

Using argv[0] from the command line would be unreliable. For example:

    >>> _ = run('"spam & eggs" /c echo %__APPDIR__%',
    ...         executable=os.environ['ComSpec'])
    C:\Windows\system32\

    >>> _ = run('"spam & eggs" -m calendar 2017 2',
    ...         executable=sys.executable)
       February 2017
    Mo Tu We Th Fr Sa Su
           1  2  3  4  5
     6  7  8  9 10 11 12
    13 14 15 16 17 18 19
    20 21 22 23 24 25 26
    27 28
msg319446 - (view) Author: Damon Atkins (damon-atkins) Date: 2018-06-13 10:11
I see from this. That this is still an issue
https://github.com/python/cpython/blob/master/Lib/subprocess.py#L1146

Is it not a solution to
save current directory location
chdir(cwd) before calling  _winapi.CreateProcess()
restore the original directory.

This will result in the cwd being searched for the executable, which most people would expect to happen.  It seems CreateProcess does not change to cwd until after the file is checked for existence or loaded.
msg319447 - (view) Author: Damon Atkins (damon-atkins) Date: 2018-06-13 10:14
See also https://bugs.python.org/msg262399
msg319467 - (view) Author: Jan Lachnitt (pepalogik) Date: 2018-06-13 14:08
@eryksun: Sorry for my late reply, apparently I did not have time to reply in 2017. I see your point, but still I think that Python is conceptually multi-platform, so its behavior on Linux and Windows should be as much consistent as possible.

I am not the one to decide which one of the two possible behaviors shall be the correct one. The current documentation <https://docs.python.org/3/library/subprocess.html#subprocess.Popen> describes the behavior on Linux: "In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path." If this is chosen as the correct behavior, then the behavior on Windows is incorrect.

@Damon Atkins: Thank you for reminding this issue, but I suspect your proposed solution of being thread-unsafe. I propose another solution: On Windows, Python should resolve the executable path itself (taking cwd and env into account) and then pass the absolute path to CreateProcess().
msg319492 - (view) Author: Damon Atkins (damon-atkins) Date: 2018-06-14 03:33
From https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425.aspx

Note Python is using CreateProcess(), consider using CreateProcessW()

The Unicode version of this function, CreateProcessW, can modify the contents of this string. Therefore, this parameter cannot be a pointer to read-only memory (such as a const variable or a literal string). If this parameter is a constant string, the function may cause an access violation.
msg319530 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-06-14 17:14
Thanks for pointing me at this issue Ned.  It sounds like there is a behavior difference between Windows and POSIX systems related to the current directory and/or which process environment is used by the system call that launches the new process to find the executable.

It seems to have existed "forever" in subprocess module API terms, so I don't know if we should reconcile the corner cases when cwd and/or env are supplied to a single cross platform behavior as that could break existing code.  Such a behavior change _could_ be made but be 3.8 specific.  BUT:  If we did that, it becomes a challenge for people writing code that needs to work on multiple Python versions.  Popen growing yet another bool flag parameter to choose the new behavior is possible, but quite ugly (and unusable on multi-python version code).

I think we should start out by living with the difference - document these platform specific corner case behaviors to minimize surprise.

If we want to provide a way for people to have the same behavior on both, we should document a recommended way to do that.  I believe that entails telling people get an absolute path to their executable themselves before launching the subprocess as that should work the same no matter the cwd or environment?
msg320097 - (view) Author: Jan Lachnitt (pepalogik) Date: 2018-06-20 16:47
Nobody responds yet, so I will.

I think that the basic proposal was made by Chris Jerdonek in msg171692 already on 2012-10-01: First document both behaviors and then discuss the possible harmonization. I think the proposal was good and further discussion has confirmed this.

Chris Jerdonek, to whom this issue is assigned, last commented it on 2012-12-18. Shouldn't the issue be assigned to somebody else?

By the way, the related env issue is #8557.
msg388902 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-17 01:02
> Python is conceptually multi-platform, so its behavior on 
> Linux and Windows should be as much consistent as possible.

It's not expected for the behavior of all Popen() parameters to be the same on all platforms. For example, the behavior and capabilities of shell=True are different in Windows vs POSIX. But I think a basic set of parameters should have been singled out for cross-platform consistency -- at least in the default case. Unfortunately it wasn't designed that way. New behavior that's consistent with POSIX can be implemented, but at this point it would have to be enabled by a parameter. 

> "In particular, the function looks for executable (or for the 
> first item in args) relative to cwd if the executable path is 
> a relative path." 

For POSIX, this should be stated as a "relative path without a slash in it" or a "relative path without a directory in it". An unqualified filename is a relative path that won't be resolved against `cwd`, unless there's a "." entry in PATH.

For Windows, the use of CreateProcess() is documented. It could be stated more explicitly that `executable`, `args` / list2cmdline(args), `env`, and `cwd` are passed directly to CreateProcess() as lpApplicationName, lpCommandLine, lpEnvironment, and lpCurrentDirectory.

Here are some notes and corrections about the documentation of lpCommandLine, in particular the following paragraph:

    If the file name does not contain an extension, .exe is appended.
    Therefore, if the file name extension is .com, this parameter must
    include the .com extension. If the file name ends in a period (.)
    with no extension, or if the file name contains a path, .exe is
    not appended. If the file name does not contain a directory path,
    the system searches for the executable file in the following
    sequence [1][2][3]:

        * %__APPDIR__%
        * %__CD__% [4]
        * %SystemRoot%\System32
        * %SystemRoot%\System
        * %SystemRoot%
        * %PATH% (machine/user extended search sequence)

    [1] The search sequence is rewritten here succinctly using
        environment variables in the current process, including the
        virtual variables __APPDIR__ (application directory) and
        __CD__ (current directory), which are supported by WinAPI
        GetEnvironmentVariableW().

    [2] A path name is resolved by searching for it if it isn't fully
        qualified and doesn't explicitly begin with the current
        directory (".") or its parent (".."). Note that, unlike POSIX,
        a relative path name is resolved by searching for it even if
        it contains a directory component (i.e. a slash or backslash).
        For example, "spam\eggs.exe" is resolved by looking for
        r"%__APPDIR__%\spam\eggs.exe" and so on.

    [3] If a path name has to be resolved by searching, and its final
        component does not contain a "." character, then ".exe" is
        appended to the name. On the other hand, if a path name does
        not need to be resolved by searching, because it's fully
        qualified or the first component is "." or "..", then if the 
        given path name doesn't exist, it also looks for the name with
        ".exe" appended, even if the final component of the path name
        contains a "." character.

    [4] If "NoDefaultCurrentDirectoryInExePath" is defined in the
        environment and the path name does not contain a directory
        component (i.e. no slash or backslash), then the current
        directory is excluded from the search sequence.

That %__APPDIR__% always takes precedence means that subprocess.Popen(['python']) runs the Python version of the current process, regardless of PATH, unless shell=True is used. 

The implementation of lpApplicationName (executable) and lpCurrentDirectory (cwd) means that argv[0] in the child process, as parsed from its command line, does not necessarily resolve to a name in the filesystem. Windows supports GetModuleFileNameW(NULL, ...) to get the path of the application executable.
msg388903 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-03-17 01:56
For our subprocess docs, Eryk's text:

"""
For POSIX, ``executable`` should be stated as a "relative path without a slash in it" or a "relative path without a directory in it". An unqualified filename is a relative path that won't be resolved against ``cwd``, unless there's a "." entry in PATH.

For Windows, the use of CreateProcess() is documented. It could be stated more explicitly that ``executable``, ``args`` / ``list2cmdline(args)``, ``env``, and ``cwd`` are passed directly to CreateProcess() as lpApplicationName, lpCommandLine, lpEnvironment, and lpCurrentDirectory.
"""

is quite reasonable.  I wouldn't include your long notes.  But a link to a MSDN article explaining that would be useful at the end of the Windows paragraph.

For the POSIX case we should describe which PATH is used.  The current one, or the one set in a ``env`` dict.
msg399464 - (view) Author: (ban) Date: 2021-08-12 15:10
>> "In particular, the function looks for executable (or for the 
>> first item in args) relative to cwd if the executable path is 
>> a relative path." 
>
> For POSIX, this should be stated as a "relative path without a slash in
> it" or a "relative path without a directory in it". An unqualified
> filename is a relative path that won't be resolved against `cwd`,
> unless there's a "." entry in PATH.

While I don't understand the wording proposed (that seem backwards to me?), I do think it would be important to fix this.

I just got puzzled and spent some effort writing a workaround *not to look in cwd* for a bare command name, before I got so skeptical I actually tried empirically what Python would do -- and see the behavior was sensible and `cwd` was taken into account *only if the executable had a path component in it*.

Any other behavior is annoying IMO as it means using a `cwd` *requires* one to pass in an absolute path to the executable not to risk running an unexpected executable, which basically makes support for looking up executable in the system PATH unusable if using a `cwd`.
It would also be somewhat inconsistent with the idea that `cwd` only *changes* the current directory prior to execution, as it would suggest the behavior is not the same when using `cwd=None` and `cwd=os.getcwd()`.

So I'd suggest amending the wording in some way, maybe something like
"In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable has an explicit relative path."
or something like that, possibly even dropping in an example.
The problem with the current wording is that usually an unqualified name *is* a relative path, whereas here one should somehow understand that it means "if it has a path component and that path component is relative", or the more down-to-earthy "if the executable starts with './' or '../'".
msg399479 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-08-12 19:21
> I don't understand the wording proposed (that seem backwards to me?)

Thanks. Looks like I inverted the logic of the quoted paragraph. It should have been a "relative path with a slash in it" is resolved against the current working directory, not "without a slash".
History
Date User Action Args
2022-04-11 14:57:33adminsetgithub: 59738
2021-08-12 19:21:42eryksunsetmessages: + msg399479
2021-08-12 15:10:31bansetnosy: + ban
messages: + msg399464
2021-03-17 01:56:51gregory.p.smithsetmessages: + msg388903
2021-03-17 01:02:39eryksunsetmessages: + msg388902
versions: + Python 3.9, Python 3.10, - Python 3.6, Python 3.7
2018-06-20 22:54:25gregory.p.smithsetassignee: gregory.p.smith
2018-06-20 20:29:47chris.jerdoneksetassignee: chris.jerdonek -> (no value)
2018-06-20 16:47:14pepalogiksetmessages: + msg320097
2018-06-14 17:14:41gregory.p.smithsetmessages: + msg319530
2018-06-14 14:55:32ned.deilysetnosy: + gregory.p.smith

versions: + Python 3.8, - Python 3.5
2018-06-14 03:33:29damon-atkinssetmessages: + msg319492
2018-06-13 14:08:31pepalogiksetmessages: + msg319467
2018-06-13 10:14:18damon-atkinssetmessages: + msg319447
2018-06-13 10:11:45damon-atkinssetnosy: + damon-atkins
messages: + msg319446
2017-02-04 14:55:52eryksunsetnosy: + eryksun
messages: + msg286967
2017-02-04 08:45:05martin.pantersettitle: subprocess.Popen(cwd) documentation -> subprocess.Popen(cwd) documentation: Posix vs Windows
components: + Windows
stage: test needed -> needs patch
2017-02-04 08:09:30martin.panterlinkissue20927 superseder
2016-11-29 23:12:42pepalogiksetmessages: + msg282052
2016-11-29 22:26:10wolmasetmessages: + msg282050
2016-11-29 03:00:00ned.deilysetnosy: + paul.moore, tim.golden, zach.ware, steve.dower

versions: - Python 3.4
2016-11-25 08:55:54wolmasettype: behavior
messages: + msg281686
components: + Library (Lib)
2016-11-24 21:42:43wolmasetnosy: + wolma

messages: + msg281658
versions: + Python 3.5, Python 3.6, Python 3.7, - Python 2.7, Python 3.2, Python 3.3
2013-03-30 07:19:51ned.deilysetmessages: + msg185549
2012-12-19 15:28:47pepalogiksetmessages: + msg177747
2012-12-18 16:53:15pepalogiksetmessages: + msg177697
2012-12-18 16:41:58pepalogiksetmessages: + msg177696
2012-12-18 16:38:48chris.jerdoneksetmessages: + msg177695
2012-12-18 16:16:19pepalogiksetnosy: + pepalogik
messages: + msg177694
2012-10-01 05:28:53chris.jerdoneksetmessages: + msg171692
stage: patch review -> test needed
2012-09-30 16:49:14python-devsetmessages: + msg171651
2012-09-30 16:33:08chris.jerdoneksetkeywords: - easy

messages: + msg171649
2012-09-30 10:03:04chris.jerdoneksetmessages: + msg171622
2012-09-30 09:26:32chris.jerdoneksetmessages: + msg171621
versions: + Python 3.2, Python 3.4
2012-09-30 09:24:38python-devsetnosy: + python-dev
messages: + msg171620
2012-09-30 05:55:33chris.jerdoneksetassignee: docs@python -> chris.jerdonek
2012-09-30 05:53:51asvetlovsetmessages: + msg171611
2012-09-11 11:35:50chris.jerdoneksetmessages: + msg170291
2012-09-05 14:25:51chris.jerdoneksetfiles: + issue-15533-8-default.patch

messages: + msg169871
2012-09-04 19:09:22chris.jerdoneksetfiles: + issue-15533-7-default.patch

messages: + msg169838
2012-09-04 13:12:32chris.jerdoneksetfiles: + issue-15533-6-default.patch

messages: + msg169829
2012-09-04 12:50:20chris.jerdoneksetfiles: + issue-15533-5-default.patch

messages: + msg169825
2012-09-04 12:21:34chris.jerdoneksetmessages: + msg169824
2012-09-04 10:23:20chris.jerdoneksetmessages: + msg169821
2012-09-04 09:27:06asvetlovsetmessages: + msg169820
2012-09-03 19:08:29chris.jerdoneksetmessages: + msg169796
2012-09-03 18:49:07asvetlovsetfiles: + issue-15533-4-default.patch

messages: + msg169795
2012-09-02 21:13:17asvetlovsetmessages: + msg169724
2012-08-28 00:28:48chris.jerdoneksetfiles: + issue-15533-3-default.patch

messages: + msg169240
2012-08-27 22:44:11ned.deilysetnosy: + ned.deily
messages: + msg169236
2012-08-27 20:37:09chris.jerdoneksetmessages: + msg169216
2012-08-27 20:27:59chris.jerdoneksetkeywords: + needs review
stage: patch review
2012-08-27 20:23:46chris.jerdoneksetfiles: + issue-15533-2-default.patch

messages: + msg169215
2012-08-27 16:16:55chris.jerdoneksetnosy: + asvetlov
2012-08-27 16:08:57chris.jerdoneksetfiles: + issue-15533-test-cases-1.patch
keywords: + patch
messages: + msg169206
2012-08-27 09:02:02cvrebertsetnosy: + cvrebert
2012-08-02 05:44:31chris.jerdonekcreate