Title: subprocess.Popen(cwd) documentation
Type: Stage: test needed
Components: Documentation Versions: Python 3.4, Python 3.2, Python 3.3, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: chris.jerdonek Nosy List: asvetlov, chris.jerdonek, cvrebert, docs@python, ned.deily, pepalogik, python-dev
Priority: normal Keywords: needs review, patch

Created on 2012-08-02 05:44 by chris.jerdonek, last changed 2013-03-30 07:19 by ned.deily.

File name Uploaded Description Edit
issue-15533-test-cases-1.patch chris.jerdonek, 2012-08-27 16:08 review
issue-15533-2-default.patch chris.jerdonek, 2012-08-27 20:23 review
issue-15533-3-default.patch chris.jerdonek, 2012-08-28 00:28 review
issue-15533-4-default.patch asvetlov, 2012-09-03 18:49 review
issue-15533-5-default.patch chris.jerdonek, 2012-09-04 12:50 review
issue-15533-6-default.patch chris.jerdonek, 2012-09-04 13:12 review
issue-15533-7-default.patch chris.jerdonek, 2012-09-04 19:09 review
issue-15533-8-default.patch chris.jerdonek, 2012-09-05 14:25 review
Messages (30)
msg167194 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-02 05:44
The sentence describing Popen()'s cwd argument in the subprocess documentation seems reversed to me:

It says, "If cwd is not None, the child’s current directory will be changed to cwd before it is executed. Note that this directory is not considered when searching the executable, so you can’t specify the program’s path relative to cwd."

However, when cwd is not None, it seems like you *must* specify the program's path relative to cwd.  For example, when running a script containing the following using `./python.exe` from a source checkout--

    p = Popen(['./python.exe', '-V'], stdout=PIPE, stderr=PIPE, cwd='temp')
you get an: "OSError: [Errno 2] No such file or directory."

In contrast, when you *do* specify the program's path relative to cwd, it works--

    p = Popen(['../python.exe', '-V'], stdout=PIPE, stderr=PIPE, cwd='temp')

Issue 6374 seems to have made the same point in its second bullet, but the issue was closed without addressing that part of it.
msg169206 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-27 16:08
Attached are a few test cases showing that Popen *does* consider cwd when searching for the executable (as well as for args[0]), and in particular that you *can* specify the program's path relative to cwd.

I also moved the test_cwd test to be adjacent to the other cwd test (the one that tests cwd with the executable argument).

I can also prepare the documentation changes for addition to the patch.
msg169215 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-27 20:23
Here is a full patch for the default branch (documentation correction and test cases for the documented behavior).

If this patch looks acceptable, I can prepare a separate patch for 2.7.
msg169216 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-27 20:37
python_dir = os.path.dirname(os.path.realpath(sys.executable))
wrong_cwd = os.path.join(python_dir, 'Doc')

Actually, is there a better directory to be using for this?  I'd like a directory that is guaranteed to exist that is in the same directory as sys.executable -- so that I can construct a simple relative path from that directory to sys.executable.
msg169236 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-27 22:44
Because tests should be runnable from installed Pythons (including binary -only installations), tests should not assume that a Python source directory is available nor make any assumptions about the location of the Python executable itself.
msg169240 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-28 00:28
Here is a new patch that makes no assumptions about the contents of the directory containing sys.executable.
msg169724 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-09-02 21:13
Maybe better to check cwd in _call_popen_and_assert for child process (like test_cwd does) instead of just checking for successful child execution?
msg169795 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-09-03 18:49
Update patch.
msg169796 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-03 19:08
Thanks, Andrew.  Regarding your comment, it was a deliberate choice not to do the additional check because I wanted each test to check only one thing.  But I am okay with adding the additional check.

Regarding the patch, should all of the methods now do something similar to what test_cwd() does?

+        # We cannot use os.path.realpath to canonicalize the path,
+        # since it doesn't expand Tru64 {memb} strings. See bug 1063571.
+        cwd = os.getcwd()
+        os.chdir(tmpdir)
+        tmpdir = os.getcwd()

It looks like test_cwd() may have needed to apply this treatment to have more reliable string-matching in the assert.  Otherwise, why is tmpdir being re-assigned to?
msg169820 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-09-04 09:27
I believe it's trick for Tru64 platform.
I've asked to support of this in python-dev.
msg169821 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-04 10:23
For future reference, here is the beginning of the e-mail thread on python-dev:

We also need to know whether the Tru64 trick needs to be used in 2.7, since this documentation issue also affects 2.7.
msg169824 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-04 12:21
Andrew, I seem to be getting a test failure for test_executable_with_cwd() with your updated patch (the child process is outputting an absolute path rather than '').

I will update the patch to fix.  There are also some stylistic changes I would like to make to the helper method (updated code comment, etc).
msg169825 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-04 12:50
Here is an updated patch.  The changes I made are:

(1) Update code comments in _call_popen_and_assert().
(2) Fix test failure.
(3) Rename _call_popen_and_assert() to _assert_cwd() since it is a simpler
    name and the old name did not reflect that the method is specific to
    the test_cwd_* methods.
(4) Add _split_python_path() helper method so that we do not need to call
    os.path.realpath(sys.executable) in every method.

Andrew, you can make changes re: Tru64 (removing old code, etc).  Thanks.
msg169829 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-04 13:12
Updating the patch again to tweak the original documentation change.

I was concerned that the previous language could be construed to mean that Popen will look in *two* places for the executable (both relative to the current directory and relative to the cwd argument).  The change I'm uploading makes this a little more clear.
msg169838 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-04 19:09
Updating the doc portion of the patch one more time.
msg169871 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-05 14:25
Here is a proposed patch that attempts to minimize the chance of test breakage for Tru64.  The patch follows Martin's recommendation on python-dev of being cautious by following existing code.
msg170291 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-11 11:35
Andrew, do you think my changes to the patch are adequate given the response on python-dev to your question?
msg171611 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-09-30 05:53
Chris, please commit your patch. It's fine for me.
msg171620 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-30 09:24
New changeset abfaa4368263 by Chris Jerdonek in branch '3.2':
Issue #15533: Clarify docs and add tests for subprocess.Popen()'s cwd argument.

New changeset f66ff96f0030 by Chris Jerdonek in branch '3.3':
Issue #15533: Merge fix from 3.2.

New changeset 37f4aa15a1c6 by Chris Jerdonek in branch 'default':
Issue #15533: Merge fix from 3.3.
msg171621 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-30 09:26
I will commit to 2.7 separately.
msg171622 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-30 10:03
Two of the tests fail on at least some of the Windows bots.  I am investigating.

ERROR: test_cwd_with_relative_arg (test.test_subprocess.ProcessTestCase)
Traceback (most recent call last):
  File "D:\Buildslave\3.2.moore-windows\build\lib\test\", line 222, in test_cwd_with_relative_arg
    self._assert_cwd(python_dir, rel_python, cwd=python_dir)
  File "D:\Buildslave\3.2.moore-windows\build\lib\test\", line 195, in _assert_cwd
  File "D:\Buildslave\3.2.moore-windows\build\lib\", line 745, in __init__
    restore_signals, start_new_session)
  File "D:\Buildslave\3.2.moore-windows\build\lib\", line 964, in _execute_child
WindowsError: [Error 2] The system cannot find the file specified

ERROR: test_cwd_with_relative_executable (test.test_subprocess.ProcessTestCase)
Traceback (most recent call last):
  File "D:\Buildslave\3.2.moore-windows\build\lib\test\", line 240, in test_cwd_with_relative_executable
  File "D:\Buildslave\3.2.moore-windows\build\lib\test\", line 195, in _assert_cwd
  File "D:\Buildslave\3.2.moore-windows\build\lib\", line 745, in __init__
    restore_signals, start_new_session)
  File "D:\Buildslave\3.2.moore-windows\build\lib\", line 964, in _execute_child
WindowsError: [Error 2] The system cannot find the file specified
msg171649 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-30 16:33
So it seems the cwd argument to Popen() currently works differently on Windows from Mac OS X.  For example, the following doesn't work on Windows (but does on Mac).  Windows doesn't look for arg0 relative to arg_cwd:

def test_cwd(arg0, arg_cwd):
    os.chdir('foo')  # Make sure we're in a different directory from arg0.
    p = subprocess.Popen([arg0, "-c",
                          "import os, sys; "
                          "sys.stdout.write(os.getcwd()); "
    print("stdout: " +"utf-8"))
    print("return_code: %s" % p.returncode)

python_path = os.path.realpath(sys.executable)
python_dir, python_base = os.path.split(python_path)
rel_python = os.path.join(os.curdir, python_base)

# Raises: WindowsError: [Error 2] The system cannot find the file specified
test_cwd(rel_python, python_dir)

I'm going to mark the two tests as "skipped" on Windows pending a resolution.
msg171651 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-30 16:49
New changeset d8d52b5b4bc2 by Chris Jerdonek in branch '3.2':
Issue #15533: Skip test_cwd_with_relative_*() tests on Windows pending resolution of issue.

New changeset 17d709f0b69b by Chris Jerdonek in branch '3.3':
Issue #15533: Merge update from 3.2.

New changeset d10a7c1ac3a7 by Chris Jerdonek in branch 'default':
Issue #15533: Merge update from 3.3.
msg171692 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-01 05:28
I propose addressing the remainder of this issue by:

1) Documenting the difference in behavior between Windows and non-Windows, adjusting the tests to reflect this difference, and then closing this issue, and then

2) Creating a new issue to discuss whether and in what version to make the behavior of the cwd argument the same across Windows and non-Windows.
msg177694 - (view) Author: Jan Lachnitt (pepalogik) Date: 2012-12-18 16:16

I am using Python 3.2.3 on Windows XP. I encountered a problem with when using the cwd parameter. I used Google to look for a solution and I found this issue in Python tracker. But this issue seems absolutely reversed!

The subprocess documentation says this: "In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path." But this is NOT true. If I use the path relative to cwd, I get Windows Error 2 (system cannot find the file). If I change the executable's path to be relative to the current directory, where the script is running (i.e. NOT the cwd parameter passed to, it works fine.
msg177695 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-12-18 16:38
Hi Jan, yes, the documentation now describes the non-Windows behavior, which is different from the Windows behavior.  See the comment just before yours on what still needs to be done to resolve this issue.  I was in the middle of creating a bunch of test cases for the various combinations, as well as documenting the results.  But I got busy with other things.  I will see if I can revisit my work on this sometime soon and upload a diff.
msg177696 - (view) Author: Jan Lachnitt (pepalogik) Date: 2012-12-18 16:41

No, it doesn't work fine: the executable reports stack overflow. I thought this had nothing to do with Python, hence I didn't mention it. But If I run the executable without Python, THEN it works fine. But this may be another issue.

I'll update to 3.3.0 and then tell you if this has changed.
msg177697 - (view) Author: Jan Lachnitt (pepalogik) Date: 2012-12-18 16:53
Hi Chris, thank for your reply, I didn't see it while writing the edit. Does it mean that on Linux, it will use the cwd?
msg177747 - (view) Author: Jan Lachnitt (pepalogik) Date: 2012-12-19 15:28
Hi all,

I have solved the problem by using absolute path of the executable. The reason why the executable didn't work properly may be that the executable's relative path was inconsistent with current directory. See the following example (I have made an executable which shows its argv and cwd). If it is called normally, then:

argv[0] = phsh0.exe
cwd = D:\Jenda\AutoLEED\TESTING\default

But if it is called by Python's from "D:\Jenda\AutoLEED\TESTING" as I want, then:

argv[0] = default\phsh0.exe
cwd = D:\Jenda\AutoLEED\TESTING\default

The executable may be confused by this inconsistency. So it is not the documentation, but Python itself what should be changed. The executable should be searched in cwd on any platform to avoid the inconsistency.

I have not yet updated my Python installation, so my results apply to 3.2.3.
msg185549 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-03-30 07:19
Note, that test_executable_without_cwd now fails when the tests are run from an installed Python.  See Issue17046.
Date User Action Args
2013-03-30 07:19:51ned.deilysetmessages: + msg185549
2012-12-19 15:28:47pepalogiksetmessages: + msg177747
2012-12-18 16:53:15pepalogiksetmessages: + msg177697
2012-12-18 16:41:58pepalogiksetmessages: + msg177696
2012-12-18 16:38:48chris.jerdoneksetmessages: + msg177695
2012-12-18 16:16:19pepalogiksetnosy: + pepalogik
messages: + msg177694
2012-10-01 05:28:53chris.jerdoneksetmessages: + msg171692
stage: patch review -> test needed
2012-09-30 16:49:14python-devsetmessages: + msg171651
2012-09-30 16:33:08chris.jerdoneksetkeywords: - easy

messages: + msg171649
2012-09-30 10:03:04chris.jerdoneksetmessages: + msg171622
2012-09-30 09:26:32chris.jerdoneksetmessages: + msg171621
versions: + Python 3.2, Python 3.4
2012-09-30 09:24:38python-devsetnosy: + python-dev
messages: + msg171620
2012-09-30 05:55:33chris.jerdoneksetassignee: docs@python -> chris.jerdonek
2012-09-30 05:53:51asvetlovsetmessages: + msg171611
2012-09-11 11:35:50chris.jerdoneksetmessages: + msg170291
2012-09-05 14:25:51chris.jerdoneksetfiles: + issue-15533-8-default.patch

messages: + msg169871
2012-09-04 19:09:22chris.jerdoneksetfiles: + issue-15533-7-default.patch

messages: + msg169838
2012-09-04 13:12:32chris.jerdoneksetfiles: + issue-15533-6-default.patch

messages: + msg169829
2012-09-04 12:50:20chris.jerdoneksetfiles: + issue-15533-5-default.patch

messages: + msg169825
2012-09-04 12:21:34chris.jerdoneksetmessages: + msg169824
2012-09-04 10:23:20chris.jerdoneksetmessages: + msg169821
2012-09-04 09:27:06asvetlovsetmessages: + msg169820
2012-09-03 19:08:29chris.jerdoneksetmessages: + msg169796
2012-09-03 18:49:07asvetlovsetfiles: + issue-15533-4-default.patch

messages: + msg169795
2012-09-02 21:13:17asvetlovsetmessages: + msg169724
2012-08-28 00:28:48chris.jerdoneksetfiles: + issue-15533-3-default.patch

messages: + msg169240
2012-08-27 22:44:11ned.deilysetnosy: + ned.deily
messages: + msg169236
2012-08-27 20:37:09chris.jerdoneksetmessages: + msg169216
2012-08-27 20:27:59chris.jerdoneksetkeywords: + needs review
stage: patch review
2012-08-27 20:23:46chris.jerdoneksetfiles: + issue-15533-2-default.patch

messages: + msg169215
2012-08-27 16:16:55chris.jerdoneksetnosy: + asvetlov
2012-08-27 16:08:57chris.jerdoneksetfiles: + issue-15533-test-cases-1.patch
keywords: + patch
messages: + msg169206
2012-08-27 09:02:02cvrebertsetnosy: + cvrebert
2012-08-02 05:44:31chris.jerdonekcreate