classification
Title: venv python reports wrong sys.executable in a subprocess on Windows
Type: Stage:
Components: Windows Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Benedek Rácz, Jurko.Gospodnetić, awaizman, eric.smith, eryksun, nirvana-msu, nsmcan, paul.moore, steve.dower, tim.golden, uranusjr, zach.ware
Priority: normal Keywords:

Created on 2019-11-24 08:17 by uranusjr, last changed 2021-03-29 22:53 by steve.dower.

Messages (30)
msg357392 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-11-24 08:17
To reproduce:

> py -m venv fooenv
> fooenv\Scripts\activate.bat
(fooenv) > python -c "import sys; print(sys.executable)"  % This is correct
C:\Users\uranusjr\Downloads\venvtest\Scripts\python.exe
(fooenv) > python -q
>>> import subprocess
>>> subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)'])
b'C:\\Users\\uranusjr\\AppData\\Local\\Programs\\Python\\Python37\\python.exe\r\n'

The output shows the base interpreter, not the interpreter in venv. Not sure whether this is a venv or subprocess problem.
msg357393 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-11-24 08:36
Linux works correctly (Ubuntu with self-compiled Python 3.7.5)

$ python3.7 -m venv fooenv
$ . fooenv/bin/activate
(fooenv) $ python -c "import sys; print(sys.executable)"
/home/uranusjr/fooenv/bin/python
(fooenv) $ python -q
>>> import subprocess
>>> subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)'])
b'/home/uranusjr/fooenv/bin/python\n'
msg357394 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-11-24 09:33
3.6 works correctly on Windows:

> py -3.6 -m venv test36
> test36\Scripts\activate.bat
>>> import subprocess
>>> print(subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)']))
b'C:\\Users\\uranusjr\\Downloads\\test36\\Scripts\\python.exe\r\n'

So it seems the problem is introduced sometime after.
msg357421 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-11-25 00:35
Your failing test case with 3.7 works for me.

If you don't use activate.bat, but just run the venv's python directly, what do you see? I get:

>py -m venv fooenv

>fooenv\Scripts\python -V
Python 3.7.0

>fooenv\Scripts\python -q
>>> import subprocess
>>> subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)'])
b'C:\\Users\\XXXXXXX\\fooenv\\Scripts\\python.exe\r\n'

What shell are you using? Above is with cmd.exe.

If you "echo %PATH%" after activate.bat, what do you see?

Before running activate.bat, do you have a python.exe in your path? If so, is it the one that subprocess is reporting?
msg357425 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-11-25 04:58
> If you don't use activate.bat, but just run the venv's python directly, what do you see? I get:
> What shell are you using? Above is with cmd.exe.

I get the same result as activating (i.e. shows the base interpeter). All results in cmd.exe as well.

> If you "echo %PATH%" after activate.bat, what do you see?
> Before running activate.bat, do you have a python.exe in your path? If so, is it the one that subprocess is reporting?

PATH is as expected, the venv’s Scripts directory at the front after activation. I (only) have a python.exe from Windows Store in PATH. The one reported by subprocess is not in PATH.

I’ll try to find a clean machine (maybe a VM) and try whether I can replicate this. BTW the problematic versions for me was 3.7.5 and 3.8.0.
msg357428 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2019-11-25 08:05
The behaviour in this area is different between 3.7.0, 3.7.2, and 3.7.3 (at least). I have reproduced the issue with 3.7.3. Steve Dower made changes to the way the python executable works in venvs in the point releases of 3.7 - see https://github.com/pypa/virtualenv/issues/1380 and https://github.com/pypa/virtualenv/issues/1339 for some discussion of how this affected virtualenv.

I suspect this issue is related - from 3.7.2 onwards, the python.exe in a venv is a redirector which runs the "base" python.exe, but with sys.executable showing the redirector rather than the actual running exe.
msg357429 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-11-25 08:09
I tested the following in various versions (all 64-bit) in a VM. All installations are 64-bit per-user.

> py -m venv testenv
> testenv\Scripts\python.exe -c "import subprocess; print(subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)']))"

3.8.0: Incorrect
3.7.5: Incorrect
3.7.4: Incorrect
3.7.3: Incorrect
3.7.2: Correct
3.6.8: Correct
3.7.1: Correct
3.7.0: Correct

So the change seems to have happened somewhere between 3.7.2 and 3.7.3. Does this timeline line up with the venv redirector change?
msg357430 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2019-11-25 08:21
Yes, it does.

I think we'd need input from Steve Dower here, as these changes were made (I believe) in support of the Windows Store build of Python, so any changes would need to be considered in the light of how they would affect that. I do, however, consider this to be a regression that should be fixed.

BTW, just for completeness,

>>> subprocess.check_output([sys.executable, '-c', 'import sys; print(sys.executable)'])

works as I'd expect, and that's the idiom that is often used. So relying on a path search to find the correct Python can be considered an unusual case (but nevertheless one I'd expect to be fixed).

I assume that the issue here is that the code is being run by the python.dll in the base environment, and whens searching for executables, Windows gives "exes that are in the same directory as the currently executing code" priority over PATH. See https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw, specifically

"""
If the file name does not contain a directory path, the system searches for the executable file in the following sequence:

1. The directory from which the application loaded.
2. The current directory for the parent process.
3. The 32-bit Windows system directory. Use the GetSystemDirectory function to get the path of this directory.
4. The 16-bit Windows system directory. There is no function that obtains the path of this directory, but it is searched. The name of this directory is System.
5. The Windows directory. Use the GetWindowsDirectory function to get the path of this directory.
6. The directories that are listed in the PATH environment variable. Note that this function does not search the per-application path specified by the App Paths registry key. To include this per-application path in the search sequence, use the ShellExecute function.
"""
msg357437 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-11-25 11:00
> whens searching for executables, Windows gives "exes that are in the
> same directory as the currently executing code" priority over PATH.

That subprocess lets CreateProcessW use a platform-dependent search that prioritizes the application directory has come up in previous issues. To avoid this, we'd have to implement our own search for the given or parsed executable name. Then pass the fully-qualified executable path as the lpApplicationName name of CreateProcessW. This is how CMD works, since it has its own search routine that incorporates the PATHEXT environment variable.

Because the application directory is searched before the working directory (if the working directory is searched at all, depending on context), this issue also affects searching for executable paths that contain a path separator. In Unix a relative path that contains a path separator is always relative to the working directory, but Windows CreateProcessW uses a normal search for a relative name unless it explicitly references the working directory as "." (e.g. ".\Scripts\pip.exe" instead of "Scripts\pip.exe").
msg357439 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2019-11-25 11:51
I presume there's also the option of setting up the environment (or however it's done now - I know the details changed as the feature was developed) so that the "base" python.exe pretends to be the venv one, exactly as the wrapper does.

However, that may well have other difficult-to-fix implications, not least that calling the base Python using an explicit full path should act like the base Python, and *not* like the venv one.

IMO, the key thing here is that either the various limitations/quirks of redirecting to the base Python should either be treated as bugs, or they should be documented (even if only in the form of explicitly saying not to rely on any specific behaviour - e.g. "running an unqualified python and expecting a PATH search to pick up the same executable as the parent shell would is not supported and may produce unexpected results").

Virtual environments are a key part of most Python development workflows, and virtualenv is in the process of switching to use the core venv module internally. When that happens, there will be a lot more visibility for unexpected behaviours like this one.
msg357440 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-11-25 12:36
> "running an unqualified python and expecting a PATH search to pick up
> the same executable as the parent shell would is not supported and may
> produce unexpected results"

CreateProcessW finds "python.exe" in __APPDIR__ before it even searches PATH. I expect that some scripts depend on this when python.exe isn't in PATH, or when a different version is in PATH. If subprocess implements its own search, it can continue to prioritize the *effective* application directory, from dirname(sys.executable).
msg357453 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-11-25 18:58
Yeah, this definitely relates to how Windows handles unqualified argv[0] in CreateProcess. Though I thought we checked that out in another issue and decided that "most" people are correctly using sys.executable here? (Or decided that it was documented well enough that they should, and using a "python" literal was relying on OS behaviour.)

I'm not a fan of trying to override the OS handling in subprocess, though that would be the fix. Possibly we could handle "python[.exe]" literals by substituting sys.executable transparently?
msg357456 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-11-25 19:20
To provide concrete context, the problem I’m facing is with how Flit resolves `flit install --python`:

https://github.com/takluyver/flit/blob/7e65ffc7a540d76b96de0df473d3edff6f97c26c/flit/__init__.py#L18-L28

Generally the setup is to install Flit into a globally available location (let’s name it env A), so it’s usable for every project and environment. For a project foo you’d have a virtual environment (env X) that’s created from a base interpreter (env B, which may or may not be the same as env A).

So the comment workflow would look like this:

> pythonB -m venv project-env
> project-env\Scripts\activate.bat
(project-env) > pythonA -m flit install --python=pythonX

This results in the following subprocess call:

subprocess.check_output(
    ["pythonX", "-c", "import sys; print(sys.executable)"],
    universal_newlines=True,
).strip()

And ideally (pre-3.7.2 Windows, or current POSIX behaviour) this would give you the absolute path to pythonX. But right now on Windows the result is pythonB.

So if this is to be determined as acceptable behaviour, we’d need to come up with a suggestion how this code can be rewritten. `shutil.which` could be a direction, but still not enough since it’d break `flit install --python=py` because that’s give you the location of py.exe, not the actual interperter. What else?
msg357457 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-11-25 19:37
> `shutil.which` could be a direction, but still not enough since it’d break `flit install --python=py` because that’s give you the location of py.exe, not the actual interperter.

This would be fine if you still run the process to get its sys.executable.

Your specific example would never have worked, FWIW, as it always would have picked up pythonA rather than the application one or the base one, unless you were relying on python3/python3.7 not being available on Windows (which is no longer true - they are included in the Store package now).
msg357459 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-11-25 20:27
> Possibly we could handle "python[.exe]" literals by substituting 
> sys.executable transparently?

Perhaps generalize this as splitext(basename(sys.executable))[0] in order to support names other than "python" and "pythonw". It would have to handle quoting enough to ignore an initial double quote when parsing the executable name out of an args string, or _winapi could wrap the shell's CommandLineToArgvW function to handle this.
msg357460 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-11-25 21:06
> Perhaps generalize this as splitext(basename(sys.executable))[0] in
> order to support names other than "python" and "pythonw". It would have
> to handle quoting enough to ignore an initial double quote when parsing
> the executable name out of an args string, or _winapi could wrap the
> shell's CommandLineToArgvW function to handle this.

This is where we've hit the point of complexity that it would have to be added as a new API. And since it's now opt-in, we may as well document that shutil.which() is the recommended way to resolve a filename against PATH (and make that work properly... ISTR it has some inconsistencies with the OS).
msg357479 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-11-26 04:28
>> not enough since it’d break `flit install --python=py` because that’s give you the location of py.exe, not the actual interperter.
> This would be fine if you still run the process to get its sys.executable.

But then I need two separate workflows based on what is passed in. For py.exe I need to run it and get sys.executable. But for python.exe I *cannot* use sys.executable because that’s the base interepeter, not the venv path I want. And `if Path(arg).stem == "py"` just seems like a bug waiting to happen.


> Your specific example would never have worked, FWIW, as it always would have picked up pythonA rather than the application one or the base one, unless you were relying on python3/python3.7 not being available on Windows (which is no longer true - they are included in the Store package now).

It is an illustration. I am fully aware of Windows not having version-named python executables. Thanks for the reminder anyway.
msg357519 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-11-26 17:31
> But then I need two separate workflows based on what is passed in. For py.exe I need to run it and get sys.executable. But for python.exe I *cannot* use sys.executable because that’s the base interepeter, not the venv path I want. And `if Path(arg).stem == "py"` just seems like a bug waiting to happen.

If you use shutil.which() to resolve "python" or "py" against PATH (which doesn't include the application directory), then you'll get the full path to the correct python.exe and will get the expected sys.executable. So it's only one flow that works for both once you add the shutil.which step.
msg366755 - (view) Author: Benedek Rácz (Benedek Rácz) Date: 2020-04-19 05:39
Is there any update/solution for this issue? This issue is the root cause of this SO post: https://stackoverflow.com/q/61290972/2506522
msg366824 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-04-20 13:20
I posted a workaround right above your post for when you want to resolve executables against PATH instead of using Windows's normal rules.

I'm not sure we reached any good approach for launching sys.executable in a venv and automatically bypassing the redirector.
msg370656 - (view) Author: Jurko Gospodnetić (Jurko.Gospodnetić) * Date: 2020-06-03 11:09
encountered what I believe is the same problem, so here are some more details I noticed, after checking what exactly that process name lookup does using ProcessMonitor:

- when running my external process in a `venv` generated virtual environment, POpen() started subprocess executables are first looked up in the base Python executable's folder (the one used to create our virtual environment) as opposed to the one running the external Python process, if POpen() is called with `shell=False`

- when doing the same and passing `shell=True` to the POpen() call, there is no such initial folder in the lookup

- when doing the same on a `virtualenv` generated virtual environment, the file lookup is done in the correct folder first - the one containing the current Python executable and not in the base one
msg371357 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-06-12 12:01
Thanks for confirming that reality aligns with the documentation.

> when doing the same and passing `shell=True` to the POpen() call, there is no such initial folder in the lookup

Correct, because a fully qualified path to cmd.exe is used, and cmd.exe's application folder is not Python's.

Ultimately, the above behaviour could have occurred anyway, so all that has really happened is that it is now more obvious. This is what always happens when you include support for other platforms - you find out which of your previous assumptions were incorrect (same principle applies to all of life if you think about it).

If someone wants to update subprocess to manually search PATH when just provided with a filename, I'm okay with that. It would be a new enhancement though.

For earlier versions, updating the docs to clarify that subprocess does not do a PATH search but relies on operating system behaviour for relative paths is probably the most helpful way to raise awareness.
msg378284 - (view) Author: Sergey Nudnou (nsmcan) Date: 2020-10-08 19:34
Hello,

I've just run into a related issue. I have a python script, which starts an another python script using subprocess.Popen(). The parent script gets pid of the child and monitors up its activity through a database by this pid. The child script updates its activity in the database using the pid it gotten from os.getpid()

Both scripts live in a virtual environment.

It worked fine in Python 3.5 and stopped working after migration to Python 3.8.

My base Python location: D:\Python\Python38\pythonw.exe
Virtual Environment: D:\test\venv\Scripts\pythonw.exe

I have realized, that when I run the following from a command prompt:
D:\test\venv\Scripts\pythonw.exe test.py

2 processes with the different PIDs are created:
PID:                   97040
Parent PID:            12004  (cmd.exe)
Command Line:          D:\test\venv\Scripts\pythonw.exe test.py

PID:                   85548
Parent PID:            97040  (pythonw.exe)
Command Line:          D:\Python\Python38\pythonw.exe test.py

It is definitely a regression, and will potentially break a lot of applications expecting a child Python process to be a direct descendant of its parent.

Also it is a waste of system resources
msg385923 - (view) Author: Alexander Stepanov (nirvana-msu) Date: 2021-01-29 17:38
Another victim of this change in `venv` behavior is Ray, which hangs forever because the workers fail to register as parent does not recognize their PIDs.
https://github.com/ray-project/ray/issues/13794
msg389321 - (view) Author: Assaf (awaizman) Date: 2021-03-22 14:17
The fact that now there is a redirector process seems to me like a regression issue.
I have an application which Popen several processes and uses the Popen object in order to get the subprocesses pids and act upon this pids (i.e. looks for log files with associated pid).

Now this breaks because the subprocess pid is the one of the redirector and not the actual one.

This is similar to 'Ray' issue mentioned earlier.
msg389345 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-22 22:14
In some cases, the problem can be worked around by setting the __PYVENV_LAUNCHER__ environment variable and executing the base executable. For example:

    import os
    import sys
    import subprocess

    def get_python_exe_env():
        if sys.executable == getattr(sys, '_base_executable', sys.executable):
            return sys.executable, None
        environ = os.environ.copy()
        environ['__PYVENV_LAUNCHER__'] = sys.executable
        return sys._base_executable, environ

    executable, environ = get_python_exe_env()
    p = subprocess.Popen([executable], env=environ)

---

As is, I don't know how to solve the problem in which an arbitrary application runs a script using the "python" launcher. Half of the problem could be solved. When the launcher creates the base Python process, it could set the parent via PROC_THREAD_ATTRIBUTE_PARENT_PROCESS [1], presuming it can open a handle for the parent application with PROCESS_CREATE_PROCESS access. However, this doesn't solve the problem from the application's perspective. It still gets the handle and process ID of the launcher.

If the purpose of using a launcher was only to allow an in-place upgrade of the base "python3x.dll", then an alternative design would be to set the __PYVENV_LAUNCHER__ environment variable, load the Python DLL from the pyvenv.cfg "home" directory, and call Py_Main(argc, argv). This would eliminate the parent<->child problem and the application directory (__APPDIR__) problem. 

However, naively implementing something like that cannot work for the store app distribution. "python3x.dll" in the app installation directory under "%ProgramFiles%\WindowsApps" only grants execute access to users that have the app's WIN://SYSAPPID identifier in their access token, which gets added by CreateProcessW() when the "python[3[.x]].exe" appexec link is executed. I'd have to experiment to see what works. Maybe "python.exe" in the virtual environment could be created as an appexec link to "venvlauncher.exe" in the app, which loads the DLL, etc.

---

[1] https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-updateprocthreadattribute
msg389741 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-03-29 20:35
The main motivator for the redirector was to support the Store package, which can only be *executed* from outside its container (we can't load the DLL directly). Given we want the two distributions to be interchangeable from a code POV, it made the most sense to switch both over. That said, there were other issues with the previous system that were worth fixing, it just happens that a new one was introduced (though it definitely already existed for some users - IPC is a complex pattern to get right even when you fully control the processes).

I still haven't been able to come up with a viable workaround that doesn't break more cases than it helps. Passing the internal environment variable around might help, but I don't want to fully support it because then that would prevent us fixing this properly!

PEP 582 (or otherwise getting away from virtual environments in favour of some other way of having project-specific package installation) may be our best hope.
msg389760 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-29 22:13
> the Store package, which can only be *executed* from outside its 
> container (we can't load the DLL directly).

The idea I was pondering was to create "python.exe" in the virtual environment as an appexec link to "C:\Program Files\WindowsApps\venvlauncher.exe" and reverse engineer whatever is needed to make CreateProcessW() build the required access token. For example, if it simply checks for the existence of "venvlauncher.exe" in the user's "WindowsApps" directory, then set that as an appexec alias as well.

However, it can't work for a simple reason. For an appexec link, CreateProcessW() reads and executes the target of the link. The original appexec link path is not retained, so there's no way for the "venvlauncher.exe" process to know about the virtual environment.

> PEP 582

"__pypackages__" doesn't seem like a drop-in replacement for virtual environments. The current directory (in the REPL) and script directory have always had precedence over the standard library, except in isolated mode, so "__pypackages__" seems more about providing a directory for Python packages that's neatly separated from the script directory. I presume also that "__pypackages__" would be added to sys.path even in isolated mode, though the PEP makes no mention of it.
msg389761 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-29 22:21
> "C:\Program Files\WindowsApps\venvlauncher.exe"

The above was supposed to be "C:\Program Files\WindowsApps\Python...\venvlauncher.exe", where "Python..." is the elided name of the package directory.
msg389768 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-03-29 22:53
> "__pypackages__" doesn't seem like a drop-in replacement for virtual environments.
Right, it's not. But if enough people adapt their workflows to use 
[something like] it rather than relying on having a local copy of 
python.exe in order to launch, apps that currently assume that their 
subprocesses don't have their own children would be correct again.

If it was a drop-in replacement, we'd have dropped it in already. All 
the drop in replacements come with similar issues, which is why we need 
to change things more broadly in order to handle all the use cases that 
exist.
History
Date User Action Args
2021-03-29 22:53:56steve.dowersetmessages: + msg389768
2021-03-29 22:21:31eryksunsetmessages: + msg389761
2021-03-29 22:13:28eryksunsetmessages: + msg389760
2021-03-29 20:35:44steve.dowersetmessages: + msg389741
2021-03-22 22:14:35eryksunsetmessages: + msg389345
2021-03-22 14:17:28awaizmansetnosy: + awaizman
messages: + msg389321
2021-01-29 17:38:55nirvana-msusetnosy: + nirvana-msu
messages: + msg385923
2020-10-08 19:34:11nsmcansetnosy: + nsmcan
messages: + msg378284
2020-06-12 12:01:44steve.dowersetmessages: + msg371357
components: - Library (Lib)
versions: + Python 3.10, - Python 3.7, Python 3.8
2020-06-03 11:09:05Jurko.Gospodnetićsetnosy: + Jurko.Gospodnetić
messages: + msg370656
2020-04-20 13:20:49steve.dowersetmessages: + msg366824
2020-04-19 05:39:01Benedek Ráczsetnosy: + Benedek Rácz
messages: + msg366755
2019-11-26 17:31:37steve.dowersetmessages: + msg357519
2019-11-26 04:28:01uranusjrsetmessages: + msg357479
2019-11-25 21:06:06steve.dowersetmessages: + msg357460
2019-11-25 20:27:05eryksunsetmessages: + msg357459
2019-11-25 19:37:07steve.dowersetmessages: + msg357457
2019-11-25 19:20:31uranusjrsetmessages: + msg357456
2019-11-25 18:58:07steve.dowersetmessages: + msg357453
2019-11-25 12:36:18eryksunsetmessages: + msg357440
2019-11-25 11:51:12paul.mooresetmessages: + msg357439
2019-11-25 11:00:39eryksunsetmessages: + msg357437
2019-11-25 08:21:15paul.mooresetmessages: + msg357430
2019-11-25 08:09:36uranusjrsetmessages: + msg357429
2019-11-25 08:05:25paul.mooresetmessages: + msg357428
2019-11-25 04:58:28uranusjrsetmessages: + msg357425
2019-11-25 00:35:55eric.smithsetnosy: + eric.smith
messages: + msg357421
2019-11-24 10:42:55xtreaksetnosy: + eryksun
2019-11-24 09:33:18uranusjrsetmessages: + msg357394
2019-11-24 08:36:19uranusjrsetmessages: + msg357393
2019-11-24 08:17:40uranusjrcreate