Title: Spawned subprocesses don't respect environment
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.6
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Ofekmeister, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-06-27 02:23 by Ofekmeister, last changed 2017-06-27 03:40 by eryksun. This issue is now closed.

Messages (4)
msg296986 - (view) Author: Ofek Lev (Ofekmeister) * Date: 2017-06-27 02:23
The following example shows that we are indeed changing PATH, but the subprocess does not acknowledge it in Windows 7 x64. Also note this works in Linux (Ubuntu 16.04).


import os
import subprocess
from contextlib import contextmanager
from tempfile import TemporaryDirectory

def get_python_path():
    return subprocess.check_output(
        ['python', '-c', 'import sys;print(sys.executable)']

def temp_chdir(cwd=None):
    with TemporaryDirectory() as d:
        origin = cwd or os.getcwd()

            yield d

def create_venv(d, pypath=None):
    command = ['virtualenv', d]
    if pypath:
        command.extend(['-p', pypath])

def venv(d):
    if os.path.exists(os.path.join(d, 'bin')):  # no cov
        venv_exe_dir = os.path.join(d, 'bin')
    elif os.path.exists(os.path.join(d, 'Scripts')):
        venv_exe_dir = os.path.join(d, 'Scripts')
        raise OSError('Unable to locate executables directory.')

    old_path = os.environ['PATH']
    os.environ['PATH'] = '{}{}{}'.format(venv_exe_dir, os.pathsep, old_path)
    os.environ['PATH'] = old_path

def test_venv():
    with temp_chdir() as d:
        d = os.path.join(d, 'test_env')
        global_python = get_python_path()
        print('PATH', os.environ['PATH'][:140])

        with venv(d):
            print('PATH', os.environ['PATH'][:140])
            venv_python = get_python_path()

        assert global_python != venv_python
        assert global_python == get_python_path()
msg296989 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-06-27 03:01
subprocess.Popen calls CreateProcess on Windows, which searches for an unqualified executable in the command line as follows:

    1. The directory from which the application loaded.
    2. The current directory for the parent process. (Starting with
       Vista, the current directory is excluded from this search if
       the environment variable NoDefaultCurrentDirectoryInExePath is
    3. The 32-bit Windows system directory. Use the GetSystemDirectory
       function to get the path of this directory.
    4. The 16-bit Windows system directory. There is no function that
       obtains the path of this directory, but it is searched. The
       name of this directory is System.
    5. The Windows directory. Use the GetWindowsDirectory function to
       get the path of this directory.
    6. The directories that are listed in the PATH environment

Thus searching for "python" will always find "python.exe" from the application directory. 

To work around this, you can use shutil.which() to find python.exe on PATH and pass it as the `executable` argument. For example:

    os.environ['PATH'] = ';'.join([r'C:\Program Files\Python35', old_path])
    python35 = shutil.which('python')

    >>> print(python35)
    C:\Program Files\Python35\python.EXE

    >>> _ ='python -V')
    Python 3.6.1

    >>> _ ='python -V', executable=python35)
    Python 3.5.2
msg296993 - (view) Author: Ofek Lev (Ofekmeister) * Date: 2017-06-27 03:32
Fixed with shell=True
msg296994 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-06-27 03:40
cmd.exe implements its own search, like shutil.which, and uses the CreateProcess lpApplicationName parameter that corresponds to the Popen executable parameter. But in general (not always) it's better to use shutil.which because you don't have to worry about the security problems that come with using the shell.
Date User Action Args
2017-06-27 03:40:27eryksunsetmessages: + msg296994
2017-06-27 03:32:52Ofekmeistersetmessages: + msg296993
2017-06-27 03:01:19eryksunsetstatus: open -> closed

nosy: + eryksun
messages: + msg296989

resolution: not a bug
stage: resolved
2017-06-27 02:23:04Ofekmeistercreate