classification
Title: Windows Multiprocessing in Virtualenv: sys.prefix is incorrect
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder: environment variables not passed correctly using new virtualenv launching in windows and python3.7+
View: 38092
Assigned To: Nosy List: eryksun, meilyadam, miss-islington, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2020-01-23 21:04 by meilyadam, last changed 2020-01-28 10:52 by miss-islington. This issue is now closed.

Files
File name Uploaded Description Edit
multiproc_venv_prefix.py meilyadam, 2020-01-23 21:04 Example script that demonstrates the behavior
Pull Requests
URL Status Linked Edit
PR 18157 merged meilyadam, 2020-01-24 03:06
PR 18158 merged meilyadam, 2020-01-24 03:14
PR 18159 closed meilyadam, 2020-01-24 03:17
PR 18237 closed miss-islington, 2020-01-28 10:34
PR 18238 merged miss-islington, 2020-01-28 10:35
Messages (6)
msg360581 - (view) Author: Adam Meily (meilyadam) * Date: 2020-01-23 21:04
I upgraded from Python 3.7.1 to 3.7.6 and began noticing a behavior that was breaking my code. My code detects if it's running in a virtualenv. This check worked in 3.7.1 but is broken in 3.7.6.

From the documentation, sys.prefix and sys.exec_prefix should point to the virtualenv when one is active. However, I'm seeing that both of these constants are pointing to the system installation directory and not my virtualenv when I am in a multiprocessing child. Here is an example output of a test application running in 3.7.6 (I've attached the test script to this ticket):

Parent process
=============================================
sys.prefix:           C:\Users\user\project\venv
sys.exec_prefix:      C:\Users\user\project\venv
sys.base_prefix:      C:\Program Files\Python37
sys.base_exec_prefix: C:\Program Files\Python37
=============================================

Subprocess
=============================================
sys.prefix:           C:\Program Files\Python37
sys.exec_prefix:      C:\Program Files\Python37
sys.base_prefix:      C:\Program Files\Python37
sys.base_exec_prefix: C:\Program Files\Python37
=============================================


I would expect that sys.prefix and sys.exec_prefix to be identical in the parent and child process.

I verified that this behavior is present in 3.7.5, 3.7.6, and 3.8.1. I am on a Windows 10 x64 system.

Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)] on win32
msg360582 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-01-23 22:39
This is a duplicate of issue 38092. The required change to fix this is very simple, if you want to work on a PR.
msg360589 - (view) Author: Adam Meily (meilyadam) * Date: 2020-01-24 01:25
OK, that makes sense.

For 3.7, I can create a PR for that corrects the order of arguments passed into _winapi.CreateProcess

For 3.8 / master, the problem appears to be that the check in popen_spawn_win32.py to set the subprocess env is failing because sys.executable != spawn.get_executable() -- spawn.get_executable() is returning sys._base_executable. So, can you confirm that the fix is to just change spawn.get_executable() to return sys.executable, like it was prior to the PR mentioned in the other ticket?
msg360590 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-01-24 01:54
> can you confirm that the fix is to just change spawn.get_executable() 
> to return sys.executable, like it was prior to the PR mentioned in 
> the other ticket?

Yes, in spawn.py, change `_python_exe = sys._base_executable` to `_python_exe = sys.executable`. This is the value that gets returned by spawn.get_executable().

This will restore the operation of the following code in Popen.__init__ in popen_spawn_win32.py:

        python_exe = spawn.get_executable()

        # bpo-35797: When running in a venv, we bypass the redirect
        # executor and launch our base Python.
        if WINENV and _path_eq(python_exe, sys.executable):
            python_exe = sys._base_executable
            env = os.environ.copy()
            env["__PYVENV_LAUNCHER__"] = sys.executable

This code switches to running the base executable in a virtual environment, which is required in order to support duplicating handles to the child or stealing handles from the parent. And the startup code in the child process is made aware of virtual environment via the "__PYVENV_LAUNCHER__" environment variable.
msg360862 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-01-28 10:34
New changeset 0be3246d4f9c8eddcd55491901d95b09fe163f15 by Steve Dower (Adam Meily) in branch 'master':
bpo-39439: Fix multiprocessing spawn path in a venv on Windows (GH-18158)
https://github.com/python/cpython/commit/0be3246d4f9c8eddcd55491901d95b09fe163f15
msg360864 - (view) Author: miss-islington (miss-islington) Date: 2020-01-28 10:52
New changeset dbb37aac1428ddcba784d9ddc77c1708c391aa80 by Miss Islington (bot) in branch '3.8':
bpo-39439: Fix multiprocessing spawn path in a venv on Windows (GH-18158)
https://github.com/python/cpython/commit/dbb37aac1428ddcba784d9ddc77c1708c391aa80
History
Date User Action Args
2020-01-28 10:52:50miss-islingtonsetnosy: + miss-islington
messages: + msg360864
2020-01-28 10:37:17steve.dowersetresolution: duplicate -> fixed
versions: + Python 3.9
2020-01-28 10:35:05miss-islingtonsetpull_requests: + pull_request17618
2020-01-28 10:34:50miss-islingtonsetpull_requests: + pull_request17617
2020-01-28 10:34:26steve.dowersetmessages: + msg360862
2020-01-24 03:17:00meilyadamsetpull_requests: + pull_request17546
2020-01-24 03:14:37meilyadamsetpull_requests: + pull_request17545
2020-01-24 03:06:24meilyadamsetpull_requests: + pull_request17544
2020-01-24 01:54:21eryksunsetmessages: + msg360590
2020-01-24 01:25:06meilyadamsetmessages: + msg360589
2020-01-23 22:39:03eryksunsetstatus: open -> closed

superseder: environment variables not passed correctly using new virtualenv launching in windows and python3.7+
nosy: + eryksun

messages: + msg360582
type: behavior
resolution: duplicate
stage: resolved
2020-01-23 21:04:50meilyadamcreate