classification
Title: Different behaviour on Posix and Windows when using subprocess.Popen(..., cwd=path)
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.3
process
Status: closed Resolution: duplicate
Dependencies: Superseder: subprocess.Popen(cwd) documentation: Posix vs Windows
View: 15533
Assigned To: docs@python Nosy List: Jovik, docs@python, eric.smith, kathweaver, r.david.murray, terry.reedy
Priority: normal Keywords:

Created on 2014-03-14 16:59 by Jovik, last changed 2017-02-04 08:09 by martin.panter. This issue is now closed.

Messages (16)
msg213570 - (view) Author: Jovik (Jovik) Date: 2014-03-14 16:59
This works on Linux as expected:
import subprocess
proc = subprocess.Popen(["./app"], stdout=subprocess.PIPE, cwd="workspace")

but on Windows I get:
FileNotFoundError: [WinError 2] The system cannot find the file specified

To successfully execute it on Windows I need to set shell=True first:
proc = subprocess.Popen(["app.exe"], stdout=subprocess.PIPE, cwd="workspace", shell=True)

which is odd since by default it should use cwd when searching for binary:
"If cwd is not None, the function changes the working directory to cwd before executing the child. In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path."
from http://docs.python.org/3.3/library/subprocess.html
msg213571 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-03-14 17:10
Your cwd is relative.  What happens if you make it absolute?  (What I'm thinking is that the non-shell starting cwd may be different on windows than it is on unix...but I don't know windows very well, so this may be irrelevant...)
msg213594 - (view) Author: Jovik (Jovik) Date: 2014-03-14 21:52
I did a test with cwd being set to full path, but the result was the same (still had to use shell=True to execute a command). Let me know if I can provide any more details.
msg213610 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-03-15 01:40
I am sure that using / instead of \, which is to say, not using os.sep, is the problem as / is *not* allowed in Windows command names even though the substitution works for paths given as options. In a Windows console,
> python # works
> .\python # works
> ./python # see /python as an option for the . program, which does not exist.

I suspect that shell=True cause subprocess to execute "shell ./app" (howver 'shell' is spelled on the system), with whatever other options and quotation are needed to make ./app work as an option passed to shell instead of a command to be executed directly.

I also suspect that passing ".\\app" might work, which would mean that you should use .%sapp" % os.sep to get a cross-platform string. If so, please close this issue.
msg213645 - (view) Author: Jovik (Jovik) Date: 2014-03-15 08:01
I'm quite aware of the os.sep issues between the systems, but I checked both out of curiosity. Here are latest results:

All of the following commands raises the same exception:
>>> proc = subprocess.Popen("plink", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace")
>>> proc = subprocess.Popen("plink", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace\\")
>>> proc = subprocess.Popen(".\plink", stdout=subprocess.PIPE, cwd="c:\python33\workspace")
>>> proc = subprocess.Popen(".\\plink", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace")
>>> proc = subprocess.Popen("plink", stdout=subprocess.PIPE, cwd="c:/python33/workspace")

Traceback (most recent call last):
  File "C:\Python33\lib\subprocess.py", line 1104, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python33\lib\subprocess.py", line 819, in __init__
    restore_signals, start_new_session)
  File "C:\Python33\lib\subprocess.py", line 1110, in _execute_child
    raise WindowsError(*e.args)
FileNotFoundError: [WinError 2] The system cannot find the file specified

But, when I set shell=True, then everything works just fine:
>>> proc = subprocess.Popen("plink", stdout=subprocess.PIPE, cwd="c:\python33\workspace", shell=True)
>>> proc = subprocess.Popen(".\plink", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace", shell=True)
>>> proc = subprocess.Popen(".\\plink", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace", shell=True)
>>> proc = subprocess.Popen("plink", stdout=subprocess.PIPE, cwd="c:/python33/workspace", shell=True)
I can get plink's output afterwards with proc.communicate()
msg213650 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2014-03-15 11:52
In the first example, you switch from "./app" to "app.exe" when using shell=True.

What happens to any of your examples if you add ".exe" without shell=True?

Popen eventually calls CreateProcess on Windows. From: http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx, describing the lpApplicationName  parameter: "This parameter must include the file name extension; no default extension is assumed."

Running the shell though, you don't need the extension.
msg213651 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2014-03-15 11:55
Assuming this is the problem, we should at least document this. It does make cross-platform coding difficult.
msg213858 - (view) Author: Jovik (Jovik) Date: 2014-03-17 13:24
As requested I did extra tests with extension. Same result as before:
>>> proc = subprocess.Popen("plink.exe", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace")
>>> proc = subprocess.Popen(".\plink.exe", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace")
>>> proc = subprocess.Popen(".\\plink.exe", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace")
>>> proc = subprocess.Popen(".\plink.exe", stdout=subprocess.PIPE, cwd="c:/python33/workspace")
>>> proc = subprocess.Popen("plink.exe", stdout=subprocess.PIPE, cwd="c:/python33/workspace")

Traceback (most recent call last):
  File "C:\Python33\lib\subprocess.py", line 1104, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python33\lib\subprocess.py", line 819, in __init__
    restore_signals, start_new_session)
  File "C:\Python33\lib\subprocess.py", line 1110, in _execute_child
    raise WindowsError(*e.args)
FileNotFoundError: [WinError 2] The system cannot find the file specified

I believe it's a wider issue, since my colleagues insisted on using shell=True on Windows by default (I didn't understand why, until now)
msg213859 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2014-03-17 13:31
Where is plink.exe? If it's not in cwd (c:\python33\workspace), note that the documentation for cwd says:
"If cwd is not None, the function changes the working directory to cwd before executing the child. In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path."

Although confusingly, the 2.7 documentation says:
"If cwd is not None, the child’s current directory will be changed to cwd before it is executed. Note that this directory is not considered when searching the executable, so you can’t specify the program’s path relative to cwd."
msg213860 - (view) Author: Jovik (Jovik) Date: 2014-03-17 13:40
plink.exe is located in c:\python33\workspace. I'm aware of the docstring difference between Python 2 and 3, thus submitted the bug for Python 3 only.
msg213866 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2014-03-17 14:06
I think the 2.7 documentation is correct: the current directory when the call is made, and not cwd, is included in the search path. I'd suggest specifying args as ["c:\\python33\\workspace\\plink.exe"].

I think I may have mislead you earlier on the search path. The Windows call is:
CreateProcess(lpApplicationName, lpCommandLine, <other stuff>, lpCurrentDirectory, <other stuff>)

The first parameter to Popen ("args") becomes lpCommandLine. The "executable" parameter to Popen, which you're not setting, becomes lpApplicationName. So, you're calling CreateProcess(NULL, "plink.exe, ..., lpCurrentDirectory="c:\\python33\\workspace").

In this case, .exe would be added if missing. But, the search path rules seem to not include the directory pointed to by lpCurrentDirectory (aka cwd).

So I think this would work:
subprocess.Popen(["c:\\python33\\workspace\\plink.exe"], stdout=subprocess.PIPE, cwd="c:\\python33\\workspace")
or
subprocess.Popen(["plink"], executable="c:\\python33\\workspace\\plink.exe", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace")

Unfortunately, I don't have a copy of 3.x on my Windows machine to test with.
msg213870 - (view) Author: Jovik (Jovik) Date: 2014-03-17 14:21
Why this feature works on Posix systems, but not Windows? If my memory is correct, it didn't work anywhere when I used Python 2.7 (according with docs).
msg213873 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2014-03-17 14:35
The underlying APIs are very different. It's debatable how much of a shim we should provide to make all platforms look alike.

I think first we should understand what it currently takes to make something work in both environments. Then we can talk about how or if we can make them look more similar.
msg214072 - (view) Author: Jovik (Jovik) Date: 2014-03-19 10:42
Isn't Python's crossplatform functionality a key feature? A quick fix would be to retry the call by adding cwd to arg[0] in case of FileNotFoundError.
msg214085 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2014-03-19 12:07
We don't always provide fully cross-platform functionality (see the os module for many examples), but we might be able to do better here. It might be some functionality we can add, it might be a documentation issue.

Note, for example, the subprocess.STARTUPINFO and subprocess.Popen creationflags argument. These expose Windows-only functionality.

I'm opposed to trying again with cwd added. There's a long history of security problems doing exactly this. It's why '.' is not in the PATH by default on Unix.

On my list of things to do is trace through exactly which scenarios work and don't work on Windows.

If you really want a more cross-platform solution, Cygwin python might work for you.
msg214336 - (view) Author: Jovik (Jovik) Date: 2014-03-21 08:45
I appreciate your suggestion regarding cygwin, but in the new code base we want to avoid this dependency. Thanks for your time on this issue.
History
Date User Action Args
2017-02-04 08:09:30martin.pantersetstatus: open -> closed
superseder: subprocess.Popen(cwd) documentation: Posix vs Windows
resolution: duplicate
stage: resolved
2014-03-21 08:45:52Joviksetmessages: + msg214336
2014-03-19 12:07:31eric.smithsetmessages: + msg214085
2014-03-19 10:42:26Joviksetmessages: + msg214072
2014-03-17 14:35:44eric.smithsetmessages: + msg213873
2014-03-17 14:21:03Joviksetmessages: + msg213870
2014-03-17 14:06:23eric.smithsetmessages: + msg213866
2014-03-17 13:40:11Joviksetmessages: + msg213860
2014-03-17 13:31:24eric.smithsetmessages: + msg213859
2014-03-17 13:24:15Joviksetmessages: + msg213858
2014-03-15 14:57:29kathweaversetnosy: + kathweaver
2014-03-15 11:55:14eric.smithsetmessages: + msg213651
2014-03-15 11:52:16eric.smithsetmessages: + msg213650
2014-03-15 08:01:34Joviksetmessages: + msg213645
2014-03-15 01:40:24terry.reedysetnosy: + terry.reedy
messages: + msg213610
2014-03-14 21:52:06Joviksetmessages: + msg213594
2014-03-14 18:11:41eric.smithsetnosy: + eric.smith
2014-03-14 17:10:34r.david.murraysetnosy: + r.david.murray
messages: + msg213571
2014-03-14 16:59:45Jovikcreate