When running subprocess.check_output(['doesnotexist']) as another user, I expect a FileNotFoundError, but a PermissionError is thrown.

How to reproduce:
============================================================
[user1 tmp]$ su --login user2
Password: 
[user2 ~]$ python
Python 3.5.0 (default, Sep 20 2015, 11:28:25) 
[GCC 5.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output(['asdf'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/subprocess.py", line 629, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 696, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.5/subprocess.py", line 950, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1540, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'asdf'
>>> quit()
[user2 ~]$ exit
logout
[user1 tmp]$ su user2
Password: 
[user2 tmp]$ python
Python 3.5.0 (default, Sep 20 2015, 11:28:25) 
[GCC 5.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output(['asdf'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/subprocess.py", line 629, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 696, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.5/subprocess.py", line 950, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1540, in _execute_child
    raise child_exception_type(errno_num, err_msg)
PermissionError: [Errno 13] Permission denied
>>> quit()
[user2 tmp]$

It is almost certainly the case that the error message is correct.  The most likely explanation given your pasted session is that user2 does not have permission in the CWD.  When you use --login, the CWD changes to user2's home, which it of course has permission in.

user2 in fact has permissions in the CWD.
The CWD has been set to /tmp intentionally, exactly for that reason that every user has "rwx" access there.

On some unix systems you cannot execute files in /tmp, even if x is set.  Is that true for yours?  Bascially, subprocess is just reporting the error that the OS is reporting to it, which is why I say the messages is almost certainly correct.

I just tried the same thing in /home/user2 which is the home of user2 and in /.
In both directories the result is the PermissionError. So the cause is not the CWD.

I just found that a "chmod o+x /home/user1" fixes the problem in the sense that now a FileNotFoundError is thrown.
So I am very certain that you can reproduce this by making your home directory non-executable.

Could it be that python tries to access the "original" home directory somehow? For example to write to a pipe?

Narrowing it down further:

In $PATH there is a subdirectory of /home/user1.
PATH="/home/user1/bin:/usr/local/sbin:/usr/local/bin:/usr/bin"

Doing an strace on the example above shows the line
stat("/home/user1/bin/python", 0x7fff9d365bb0) = -1 EACCES (Permission denied)
AFAICT this line is generated when the shell looks for the correct python executable, but it should not interfere with the (later) call to subprocess.check_output().

If I delete "/home/user1/bin" from $PATH, then the correct FileNotFoundError is thrown.

So it seems (just guessing) that subprocess.check_output() somehow throws the older, obsolete error code. Would this be possible?

What do you mean by older obsolete error code?  It sounds like the problem is that user2 doesn't have read (or execute?) permission for that directory in the path.  I'd think it would just skip it in that case, though.  I don't have time to run tests myself right now...maybe you can take a look at what subprocess is actually doing when that error is generated?  (It might be in the C code, though, I don't remember).

In subprocess.py there's the following code that builds a sequence of potential paths for the executable [1]:

    executable = os.fsencode(executable)
    if os.path.dirname(executable):
        executable_list = (executable,)
    else:
        # This matches the behavior of os._execvpe().
        executable_list = tuple(
            os.path.join(os.fsencode(dir), executable)
            for dir in os.get_exec_path(env))

In this case it tries to execute "/home/user1/bin/asdf", which fails with EACCES (to log this using strace, use -f to follow the fork). This occurs in child_exec in _posixsubprocess.c, in the following loop [2]:

    /* This loop matches the Lib/os.py _execvpe()'s PATH search when */
    /* given the executable_list generated by Lib/subprocess.py.     */
    saved_errno = 0;
    for (i = 0; exec_array[i] != NULL; ++i) {
        const char *executable = exec_array[i];
        if (envp) {
            execve(executable, argv, envp);
        } else {
            execv(executable, argv);
        }
        if (errno != ENOENT && errno != ENOTDIR && saved_errno == 0) {
            saved_errno = errno;
        }
    }
    /* Report the first exec error, not the last. */
    if (saved_errno)
        errno = saved_errno;

saved_errno will be set to EACCES and stored back to errno after all attempts to execute potential paths fail. This is then reported back to the parent process, which raises a PermissionError.

[1]: https://hg.python.org/cpython/file/3.5/Lib/subprocess.py#l1463
[2]: https://hg.python.org/cpython/file/3.5/Modules/_posixsubprocess.c#l487

So, two interesting questions: does this in fact match the behavior of os._execvpe, and does it match the behavior of the shell?  The latter would appear to be false, and could arguably be claimed to be a bug.  If we agree that it is, we need to learn what the behavior of the shell is in a bunch of corner cases (only inaccessible directories on path, only match is accessible but not executable, etc).

If this is a bug I'd guess it applies to all supported python versions.

Note that it should be possible to reproduce this using a single user, so I've changed the title accordingly.

Definitely a bug. The path search should silently skip directories it can't
access.

On Tue, Oct 27, 2015, 7:05 AM R. David Murray <report@bugs.python.org>
wrote:

>
> R. David Murray added the comment:
>
> So, two interesting questions: does this in fact match the behavior of
> os._execvpe, and does it match the behavior of the shell?  The latter would
> appear to be false, and could arguably be claimed to be a bug.  If we agree
> that it is, we need to learn what the behavior of the shell is in a bunch
> of corner cases (only inaccessible directories on path, only match is
> accessible but not executable, etc).
>
> If this is a bug I'd guess it applies to all supported python versions.
>
> Note that it should be possible to reproduce this using a single user, so
> I've changed the title accordingly.
>
> ----------
> nosy: +gps
> title: PermissionError in subprocess.check_output() when running as a
> different user (not login shell) -> PermissionError in
> subprocess.check_output() when an inaccessible directory on the path
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue25481>
> _______________________________________
>

> So, two interesting questions: does this in fact match the behavior of 
> os._execvpe, and does it match the behavior of the shell? 

I think it's fine. child_exec() tries all paths. It saves the first error that's not ENOENT or ENOTDIR. The saved error gets reported back, else ENOENT or ENOTDIR. This is the same as os._execvpe:

    for dir in path_list:
        fullname = path.join(dir, file)
        try:
            exec_func(fullname, *argrest)
        except (FileNotFoundError, NotADirectoryError) as e:
            last_exc = e
        except OSError as e:
            last_exc = e
            if saved_exc is None:
                saved_exc = e
    if saved_exc is not None:
        raise saved_exc
    raise last_exc

Perhaps the rule for PATH search errors should be documented for os.execvp[e] and subprocess.

I think it matches the shell, except, AFAIK, the shell doesn't distinguish ENOENT and ENOTDIR errors. For example, where "/noexec" is a directory that has no execute access:

    $ /bin/sh -c spam
    /bin/sh: 1: spam: not found

    $ PATH=/noexec:$PATH /bin/sh -c spam
    /bin/sh: 1: spam: Permission denied