classification
Title: test_subprocess and test_dtrace fails if the last entry of $PATH is a file
Type: compile error Stage: resolved
Components: Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: jayyin11043, lukasz.langa, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2018-01-25 15:12 by jayyin11043, last changed 2018-01-25 21:43 by jayyin11043. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5322 merged vstinner, 2018-01-25 17:29
PR 5323 merged python-dev, 2018-01-25 18:06
PR 5324 merged vstinner, 2018-01-25 18:18
Messages (8)
msg310678 - (view) Author: Jay Yin (jayyin11043) * Date: 2018-01-25 15:12
Hello everyone,


I've been trying to build the master branch on Ubuntu 16.04 and it currently fails 2 test, I was wondering if this was normal or if I'm missing dependencies, I also tried apt-get build-dev python3.6 and python3.7 to no avail, the build requirements install worked for python3.5 but I suspect 3.7 has different dependencies but I can't find where the documentation for the requirements are. 
2 tests failed:
    test_dtrace test_subprocess


running test_dtrace as verbose gave https://pastebin.com/ZGzzxwjk
	
[Bash] FAILED (errors=4) test test_dtrace failed 1 test failed: test_dtrace R - Pastebin.com
pastebin.com


and running test_subprocess gives


https://pastebin.com/DNjPzpgp
msg310691 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-25 17:28
I discussed with Jay Yin on IRC and we understood the issue on his setup: the last entry of his PATH environment variable is a path to an existing *file*, not a directory.

In this case, subprocess.Popen() fails with ENOTDIR if the program cannot be found in any other directory of the PATH.

Copy of _posixmodule.c:
---
    /* This loop matches the Lib/os.py _execvpe()'s PATH search when */
    /* given the executable_list generated by Lib/subprocess.py.     */
    saved_errno = 0;
    for (i = 0; exec_array[i] != NULL; ++i) {
        const char *executable = exec_array[i];
        if (envp) {
            execve(executable, argv, envp);
        } else {
            execv(executable, argv);
        }
        if (errno != ENOENT && errno != ENOTDIR && saved_errno == 0) {
            saved_errno = errno;
        }
    }
    /* Report the first exec error, not the last. */
    if (saved_errno)
        errno = saved_errno;
---

If the first execv() calls with ENOENT and the last one fails with ENOTDIR, the function fails with ENOTDIR.
msg310693 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-25 17:32
Attached PR 5322 fixes the issue.

Example to reproduce the bug:

$ touch file
$ PATH=$PATH:$PWD/file ./python -m test test_subprocess -m test_invalid_args -v
(...)
======================================================================
ERROR: test_invalid_args (test.test_subprocess.ContextManagerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/test/test_subprocess.py", line 3050, in test_invalid_args
    stderr=subprocess.PIPE) as proc:
  File "/home/vstinner/prog/python/master/Lib/subprocess.py", line 743, in __init__
    restore_signals, start_new_session)
  File "/home/vstinner/prog/python/master/Lib/subprocess.py", line 1431, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: 'nonexisting_i_hope'
(...)


With PR 5322 applied, the whole Python test suite pass with PATH=$PATH:$PWD/file.
msg310696 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-25 18:06
New changeset b31206a223955d614d7769f95fb979d60f77bf87 by Victor Stinner in branch 'master':
bpo-32667: Fix tests when $PATH contains a file (#5322)
https://github.com/python/cpython/commit/b31206a223955d614d7769f95fb979d60f77bf87
msg310708 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-25 21:39
New changeset 255dbd2102d5dec5ffbd0b94084377e98c3b56c4 by Victor Stinner (Miss Islington (bot)) in branch '3.6':
bpo-32667: Fix tests when $PATH contains a file (GH-5322) (#5323)
https://github.com/python/cpython/commit/255dbd2102d5dec5ffbd0b94084377e98c3b56c4
msg310709 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-25 21:41
New changeset 6996f284d4d90aa05c46d9fe6f38d1030454b224 by Victor Stinner in branch '2.7':
bpo-32667: Fix tests when $PATH contains a file (#5324)
https://github.com/python/cpython/commit/6996f284d4d90aa05c46d9fe6f38d1030454b224
msg310710 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-25 21:42
Thank you Jay Yin for your bug report. I fixed the bug in Python 2.7, 3.6 and master.
msg310711 - (view) Author: Jay Yin (jayyin11043) * Date: 2018-01-25 21:43
no problem, thanks for helping and fixing the issue, I can now help contribute to python =D
History
Date User Action Args
2018-01-25 21:43:15jayyin11043setmessages: + msg310711
2018-01-25 21:42:20vstinnersetmessages: + msg310710
2018-01-25 21:41:40vstinnersetmessages: + msg310709
2018-01-25 21:39:43vstinnersetmessages: + msg310708
2018-01-25 19:52:37jayyin11043setversions: + Python 2.7
2018-01-25 19:50:14jayyin11043setresolution: fixed
2018-01-25 19:50:06jayyin11043setversions: + Python 3.6
2018-01-25 19:49:49jayyin11043setstatus: open -> closed
stage: patch review -> resolved
2018-01-25 18:18:26vstinnersetpull_requests: + pull_request5169
2018-01-25 18:06:59python-devsetpull_requests: + pull_request5168
2018-01-25 18:06:07vstinnersetmessages: + msg310696
2018-01-25 17:32:17vstinnersetmessages: + msg310693
2018-01-25 17:29:43vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request5167
2018-01-25 17:28:51vstinnersetnosy: + vstinner

messages: + msg310691
title: Failing test_dtrace and test_subprocess on Ubuntu 16.04 on master -> test_subprocess and test_dtrace fails if the last entry of $PATH is a file
2018-01-25 16:37:37zach.waresetnosy: + lukasz.langa, zach.ware
2018-01-25 15:12:46jayyin11043create