classification
Title: venv subprocess call to python resolves to wrong interpreter
Type: Stage:
Components: Windows Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, eryksun, gaborjbernat, keller00, paul.moore, steve.dower, tim.golden, vinay.sajip, zach.ware
Priority: normal Keywords: patch

Created on 2020-10-15 05:50 by keller00, last changed 2020-10-20 20:02 by paul.moore.

Files
File name Uploaded Description Edit
repro.tar.gz keller00, 2020-10-15 05:50 A gunzip tar file with files needed to reproduce the issue
Pull Requests
URL Status Linked Edit
PR 22715 merged paul.moore, 2020-10-15 18:54
Messages (33)
msg378665 - (view) Author: Mark Keller (keller00) Date: 2020-10-15 05:50
Hey,

Wanted to report this weird behavior I saw recently. Let me first explain how to reproduce and then talk about where I think the issue comes from.

I attached a tar.gz file with 2 Python files in it.

Here's how to reproduce:
0. Make sure that pytest is unavailable in your command line, or please edit to another library that you don't have available from another Python install (make sure to update the import statement in `script1.py`).
1. Untar the 2 scripts into current working directory. Note: I reproduced on a Windows VM (Version 10.0.18363 Build 18363).
2. Make a Python 3.7, or 3.8 environment venv. Note: I used Python 3.8.2.
3. Activate venv and install pytest in it.
4. Run `python run_scripts.py` and observe the error.

Note: The same thing works on Python 3.6 as expected.

My team observed this issue when virtualenv released this Monday. It changed how it works with Windows and Python 3.7+, before it was not using the Windows redirect script generated by venv and just worked like how older versions on Windows do.
But what we are seeing is that if you start a subprocess call to `python` then that gets resolved to the system wide Python binary without the venv site-packages being in PYTHONPATH.
So the subprocess will fail when importing pytest in the subprocess.

If this is not clear please don't hesitate to ask me for clarification, or refer to https://github.com/pypa/virtualenv/issues/1981 for more debugging info.

Another interesting thing that I noticed is that if you make a subprocess call to `pytest` instead then that gets resolved correctly.
msg378666 - (view) Author: Bernat Gabor (gaborjbernat) * Date: 2020-10-15 07:20
I wonder if this is an interaction issue with the builtin Windows provided python executable?
msg378676 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-10-15 11:51
This is due to how Windows resolves relative paths when creating a new process. It *always* looks in the current application directory first, which with this setup will be the original Python executable rather than the venv redirector.

The best fix is to pass sys.executable instead of "python", which is more reliable on every OS and under every configuration. I've heard multiple people say that they prefer to launch venvs directly rather than modifying their environment in order to put it first on PATH.

If you are deliberately trying to use PATH to resolve the user's default "python" (which may not be in any way related to the one that's currently running), then use shutil.which() first and pass the full path to subprocess.

There's no bug to fix here, but if someone wants to add a section to the docs (and ideally, Mark or someone who has hit this problem can tell us which docs they read where they might have seen it) explaining that passing "python" to subprocess is a bad idea, feel free.
msg378681 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-15 12:57
Agreed, this is just "normal Windows behaviour". It's definitely surprising and non-obvious (I saw this issue and it never even occurred to me that this was what was happening) so I'd support documenting it somewhere. My instinct is that the subprocess documentation is the right place, but I suspect that in practice, people hitting this issue with a virtual environment won't spot anything we write there, so it'll end up just being a "well, we told you so" measure rather than helping users in a practical sense.

So I'd agree with Steve's comment - Mark, if you can suggest anywhere we could document this which you *would* have spotted when you encountered this issue, that would be great.

Hmm, maybe we could document in venv that the Python executable in a venv redirects to the base Python, so that the "application directory" for a script run from a venv is the base executable's directory, not the venv - with a link to the MS documentation that explains how this affects path searches? (IIRC, we've been hesitant to document the redirection as it's an "implementation detail", but this is clearly a case where it affects user-visible behaviour, so we can't really use that argument any more...)
msg378682 - (view) Author: Bernat Gabor (gaborjbernat) * Date: 2020-10-15 13:20
Taking a look at the existing documentation this should probably be mentioned under the args part of https://docs.python.org/3/library/subprocess.html#popen-constructor, not? (that's where other similar gotchas are explained too)
msg378689 - (view) Author: Mark Keller (keller00) Date: 2020-10-15 17:14
Thank you so much for the prompt responses.

I would say that this should be documented on the venv page (https://docs.python.org/3/library/venv.html) as I find the subprocess page too dense of information already, but I do agree that venv page is most likely not where developers would look first.

I'm a little embarrassed to say this, but I didn't even know that sys.executable existed.

So maybe the better way to document this would be to mention sys.executable on the subprocess page, just as a sure way find where the current interpreter lives and then go into detail why this behavior difference is on the venv page. Maybe have them linked to each other?

What do you think about this?
msg378692 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-15 18:55
I've created https://github.com/python/cpython/pull/22715 as a suggested documentation fix. Mark, does this cover what you would have needed to know? Steve, is what I've written technically accurate?
msg378694 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-15 19:19
PR 22715 is accurate for the current implementation of subprocess. There has been discussion in past issues about implementing a PATH search that's similar to what the CMD shell does -- as in calling os.system() or Popen() with shell=True -- instead of relying on the search that WinAPI CreateProcessW implements. The resolved path would be passed as the `executable` argument, which becomes the lpApplicationName argument of CreateProcessW.
msg378697 - (view) Author: Mark Keller (keller00) Date: 2020-10-15 20:15
This sounds good to me Paul, thank you again!
msg378698 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-15 20:41
Eryk - I suggest that we modify (or remove) the note in venv that my PR adds as part of such a change, rather than try to qualify the text now (which would probably only make it harder for people to understand the issue).
msg378719 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-10-16 16:29
Thanks for doing the PR, Paul.

Honestly, I'd rather document in subprocess that Popen("python") is not a good idea, and should use either sys.executable or shutil.which depending on what you mean. This is a platform-independent gotcha, as it's very easy to reproduce the failure (e.g. run on Linux where "python"==Python 2.x).

Describing the current state of how venv works in the venv docs is fine, and what you've got there is accurate (might be easier to follow on first reading if it leads with Python picking up the venv based on the launched executable, rather than anything in the terminal's environment).

My concern is that we don't make this level of detail a documented guarantee, and can still change it between major versions without the deprecation period. I'm happy to guarantee intended uses, and it might be worth figuring out what those are (e.g. does double-clicking in Explorer matter? what about the Run window? Desktop shortcuts? With/without activation? etc.) and likely we'd have to clarify which ones don't work the same for the symlink vs. redirector options.

IOW, if someone writes an actual spec (and we all agree on it), we can guarantee that. Without it, I'd rather implementation internals remain internal.
msg378727 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-16 17:54
Fair enough. I think we have to be careful here - people do rely on how the "internal" aspects of virtual environments work, like it or not, and we've definitely broken people's code in the past as a result of taking a hardline "it's not documented, so it's not guaranteed" approach. (Disclaimer - we broke some of my code, so I'm not unbiased when I say the above ;-))

I take your point that this is a cross-platform issue, and as such would be worth putting in the `subprocess` documentation. When I looked, though, I couldn't really find anywhere that felt appropriate. I'll take a second look, but if you have any suggestions I'd appreciate it.

The bit that I *do* think is a venv gotcha is that it's entirely reasonable for a user to expect that if they run path\to\venv\Scripts\python.exe, then their Python script will be run by that executable. The redirector breaks that assumption, and while I'm fine with the idea that we don't want to document all of the details of the redirector, I *do* think we should document that you must not make that assumption.

How about this:

   The mechanism by which virtual enviroments are implemented means that
   you should not assume that the executable in the virtual environment
   is the one that will ultimately run your script. As a result, you
   should always use `sys.executable` to identify "this script's Python
   interpreter", as that is guaranteed to be set as you would expect.

I'm not completely happy with the above, but that's mostly because I'd rather just document how the process works (even if it's with a disclaimer "this is implementation detail and may change without notice"). But that's not my call to make, so I'll defer to you on that.
msg378729 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-10-16 18:17
> The bit that I *do* think is a venv gotcha is that it's entirely reasonable for a user to expect that if they run path\to\venv\Scripts\python.exe, then their Python script will be run by that executable.

This is still true, though, as much as it's ever been. The gotcha is that launching "python" may not launch "path\to\venv\Scripts\python.exe" even if that's the first entry on PATH.

For the documentation, we could phrase it more positively as "If you need the path to the Python executable, use sys.executable. Relying on explicit or implicit PATH resolution may result in the wrong version of Python being used, especially when the user has launched from a virtual environment."
msg378741 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-16 19:45
> This is still true, though, as much as it's ever been.

But it's *not*, if I understand the problem here. Windows resolves a bare "python" command by searching the directory containing the running executable first. I would reasonably assume that if I type

    path/to/venv/scripts/python.exe foo.py

then the "running executable" is "path/to/venv/scripts/python.exe", and so the path containing the running executable is "path/to/venv/scripts", so a search for "python" will locate "path/to/venv/scripts/python.exe", because that's how Windows path search rules work. The problem is that that exe is a redirector, and the script is *actually* being run by the system Python.

Of course, the assumption I made is flawed, because there's never been anything saying that a program called "python.exe" can't do anything it likes in the process of running a script, including redirecting. But it's been the case for a long time, and IMO the introduction of the redirector counts as a user-visible behaviour change, and we should therefore explicitly point it out.

I really don't see why you are so reluctant to include this. I'm not asking that we guarantee any behaviour, or that we commit ourselves to anything. Just that we note that people ought not to be making a specific assumption (which it appears from this issue, people *have* been making).

> we could phrase it more positively

Cool, I'll update the PR to add this statement or something similar to the subprocess docs in the next couple of days.

I'm not going to fight to keep the comments in the venv documentation. I'm disappointed, and I feel like we'll end up with people relying on *more* of the implementation details because we're keeping things vague, but I have more important things to debate, so I'll drop this :-)
msg378760 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-10-16 22:00
> I'm not asking that we guarantee any behaviour, or that we commit ourselves to anything. Just that we note that people ought not to be making a specific assumption

Which is a fine intent, it's just the running assumption is that if it's in the documentation, it's supported (e.g. the distutils discussion). So no matter how carefully we document something as unsupported, people will still argue that it's supported because it's in the docs 🤷‍♂️

> which it appears from this issue, people *have* been making

We'd need Mark to weigh in (or possibly track down and ask a colleague), but I expect the assumption is that subprocess.Popen("python") searches PATH first. I've honestly never encountered anyone who deliberately depended on the application directory search (at least the first time, some who have discovered it have used it, though it tends to surprise colleagues when they do).

> I'll update the PR to add this statement or something similar to the subprocess docs in the next couple of days.

No rush. We've got the sprints next week, and I don't have any major time consuming projects, so will probably get a few easier PRs done. Looks like I can edit your PR, so I can finish it up if you want to just move on :)
msg378772 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-16 22:30
> We'd need Mark to weigh in (or possibly track down and ask a colleague), but I expect the assumption is that subprocess.Popen("python") searches PATH first.

I'm sure it was, TBH. The executable directory side of things is really just on digging a bit deeper - "well, Windows actually uses the executable directory, but that's not relevant because I ran the venv Python". That's when the mistaken assumption happens.
msg378876 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-18 16:05
OK, PR updated as per discussion.
msg378881 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-18 18:07
I don't know how much should be documented for subprocess.Popen, but here are the details for how searching works with shell=False (default) and shell=True.

For shell=False, the search path used by WinAPI CreateProcessW checks %__APPDIR__%; %__CD__% (unless %NoDefaultCurrentDirectoryInExePath% is defined and the name to search contains no backslashes); %SystemRoot%\System32; %SystemRoot%\System; %SystemRoot%; and then the %PATH% directories. The search path is passed to WinAPI SearchPathW, with a default ".exe" extension. SearchPathW tries to resolve a relative path (but not a drive-relative or rooted-relative path) against every directory in the search path, unless it explicitly begins with a "." or ".." component. For the relative case, it appends the default ".exe" extension to a name if and only if it has no extension (a trailing "." counts as an extension). For the non-relative case, it first tries to find the name as given and then with the default extension appended, even if the filename already has an extension (no exception is made for a trailing dot, i.e. searching for "C:\Temp\spam." will check for "spam." and then "spam..exe").

From the POSIX perspective, the implicit inclusion of the application directory, working directory, and system directories is strange and, regarding the inclusion of the working directory, troubling. The fact that searching for a relative name with one or more slashes, such as "spam\\python", is not resolved against *only* the working directory is strange and not documented. The rules governing when ".exe" will be appended are complicated and incorrectly documented (e.g. the claim "if the file name contains a path, .exe is not appended").

With shell=True, the CMD shell simply checks %__CD__% (unless %NoDefaultCurrentDirectoryInExePath% is defined and the name to search contains no slashes) and %PATH%. Support for forward slash in the name to search is wonky; it works only for quoted paths. But at least a relative path that contains slashes is only resolved against the current working directory instead of every directory in the search path. CMD's rules for appending default extensions are simpler than SearchPathW in some ways, but also more complicated because it's generalized as the PATHEXT list of extensions. In each directory, CMD always looks first for the searched name as given and then the name plus each extension in PATHEXT, regardless of the filepath type or whether the searched name already has an extension. It will not find a name that has no extension unless PATHEXT includes a "." entry for the empty extension. (This is consistent with the desktop shell API, which supports defining an association for the "." filetype.)
msg378886 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-18 19:02
I don't think we should document this level of detail, both because it's basically Windows standard behaviour and therefore not up to us to document, and because it's *way* too overwhelming for the average user.
msg378889 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-18 19:19
> I don't think we should document this level of detail

But a lot of it -- most of it -- is also strange behavior that no one would expect without reading about it somewhere. Most users of subprocess.Popen() will never wade through the documentation of CreateProcessW, SearchPathW, ShellExecuteExW, and cmd.exe -- to the extent that the behavior is usefully and correctly documented by Microsoft.

One of the reasons I bother writing it out in detail here is to make the case for always using shutil.which(), regardless of the value of `shell` (unless shell=True is being used beyond the filesystem scope). We have complete control of the implementation and documentation of shutil.which(). I don't want that advice to be narrowed down to just talking about running "python".
msg378890 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-18 19:50
Well, I'm not convinced that using `shutil.which` is the right answer for anything other than "python". And even there, I'd recommend using `sys.executable` over `shutil.which` in pretty much every case.

My view is that if we ignore the "weirdness" introduced by the redirector[1], for everything else `subprocess` is fine - it searches for the executable using the normal platform semantics. That's nearly always the same across platforms, and when it isn't, users may well prefer consistency with the platform over identical cross-platform behaviour. And if they *do* want cross-platform consistency, at the cost of occasional differences between their program's behaviour and (for example) the shell, then they have `shutil.which` available.

Saying that `subprocess` follows platform semantics is easy, and clear. People who need to know platform details can find them out. People who want Python-defined cross-platform semantics have `shutil.which`.

That's the basis on which I feel that documenting the Windows rules is unnecessary. Also, why would we document the Windows rules, but not the POSIX rules? They are arguably just as strange to someone who doesn't know them.

[1] I would characterise that weirdness as being that the redirector impacts the behaviour of how the platform locates the "python" command under the platform search rules of the platform the redirector is used on - even though we don't document the behaviour of the redirector so the user cannot infer the actual rules without knowing implementation details.
msg378891 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-18 20:06
> Also, why would we document the Windows rules, but not the POSIX 
> rules? They are arguably just as strange to someone who doesn't 
> know them.

POSIX rules are simply to search PATH for the filename as passed, and if it has a slash in it, the path is resolved against the working directory. There is no implicit search of the application directory, current directory, and system directories. There is no search for "dir/file" in *every* search path directory instead of just resolving against the working directory. There is no figuring out when ".exe" or PATHEXT extensions will be appended or how to search for a filename that has no extension by appending a trailing "." to the name. And there is no massive inconsistency between the search semantics of shell=True and shell=False. What happens with "platform semantics" in Windows is complicated compared to POSIX. I'm more comfortable telling people to search via shutil.which() than relying on the platform search. I'd be much more comfortable if that's what subprocess.Popen() just did on its own. But we're locked into platform semantics, and I don't see that changing.
msg378894 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-18 20:30
Exactly. Why isn't the current directory searched? Why isn't foo/bar searched for on PATH? Why is it possible for the user to accidentally remove system commands from PATH and lose access to them? Any system is confusing and surprising to users only familiar with another system.

And I've had very bad experiences in the past with languages/applications that presume to define a "cross-platform" abstraction that ends up just "not behaving how I expect a native app to". You can build a cross-platform wrapper on top of native behaviour (witness `shutil.which`) but you can't do the opposite.

Maybe there's scope for a section in the documentation that discusses how to use `subprocess` in a platform-agnostic manner. I'd be OK with that, although I'd want it to read along the lines of "these are places where Windows and POSIX behave differently" and not "here's some weird stuff Windows does that POSIX users need to be aware of" which was how your comment read to me.

Never mind, I think we can just agree to differ on this. It's not likely to impact this issue or the PR for it.
msg378902 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-18 21:17
> You can build a cross-platform wrapper on top of native behaviour 
> (witness `shutil.which`) but you can't do the opposite.

Nothing would prevent adding a parameter to use the platform semantics, if that had been in the design from the outset. But it's not something we should change at this stage. It's too fundamental.

> I'm not convinced that using `shutil.which` is the right 
> answer for anything other than "python".

Actually, I think shutil.which() is the right answer generally (within the filesystem scope) except for "python", for which sys.executable is usually what people want, i.e. the current interpreter. A PATH search via shutil.which('python') doesn't necessarily find the current interpreter, if anything at all. So I'm not arguing against advice to use sys.executable, but rather that recommmending shutil.which() should be done in a more generic way that's separate from the "python" problem. 

For platform semantics, I'd prefer a link to the CreateProcessW docs, with advice to read about the lpApplicationName parameter with respect to `executable` and lpCommandLine with respect to `args` and the platform search semantics. The CreateProcessW docs are rather long and off-putting, so I think it helps to narrow it down for people, and explicitly map between Popen() and CreateProcessW parameters.

Over the years, I've come across many forum questions in which novice users waste days on problems with respect to Windows quirks that can be resolved in a minute or so by someone with knowledge and experience. They get lots of advice with good intentions from programmers who only have POSIX experience (the Python ecosystem online is still heavily dominated by POSIX, despite the number of Windows users), and it's mostly wrong advice and a waste of their time (not even a learning experience). Documenting platform inconsistencies helps experienced programmers to use the docs in order to help novice programmers. It's not necessarily about helping novices directly. Often just a nudge in the right direction is enough.
msg378909 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-18 22:46
> For platform semantics, I'd prefer a link to the CreateProcessW docs, 
> with advice to read about the lpApplicationName parameter with 
> respect to `executable` and lpCommandLine with respect to `args` 
> and the platform search semantics. 

For example, let's help someone figure out that because "python3.8" has a ".8" filename 'extension', ".exe" may or may not be appended in a search.

    >>> print(shutil.which('python3.8'))
    C:\Users\someone\AppData\Local\Microsoft\WindowsApps\python3.8.EXE

    >>> subprocess.call(['python3.8.exe', '-V'])
    Python 3.8.6
    0

SearchPathW (called internally by CreateProcessW) won't append the ".exe" default extension to a name that already has a ".8" extension:

    >>> try: subprocess.call(['python3.8', '-V'])
    ... except OSError as e: print(e)
    ...
    [WinError 2] The system cannot find the file specified

But with shell=True it works because CMD always appends the PATHEXT extensions (thankfully there isn't a "python3.8.com" file to get in the way, since .COM is usually listed before .EXE in PATHEXT):

    >>> subprocess.call('python3.8 -V', shell=True)
    Python 3.8.6
    0

SearchPathW does append the default ".exe" extension for a qualified path:

    >>> subprocess.call([r'C:\Users\someone\AppData\Local\Microsoft\WindowsApps\python3.8', '-V'])
    Python 3.8.6
    0
msg378927 - (view) Author: Mark Keller (keller00) Date: 2020-10-19 06:06
Paul explained precisely of what I was going through when I reached out to you with this:

> then the "running executable" is "path/to/venv/scripts/python.exe", and so the path containing the running executable is "path/to/venv/scripts", so a search for "python" will locate "path/to/venv/scripts/python.exe", because that's how Windows path search rules work. The problem is that that exe is a redirector, and the script is *actually* being run by the system Python.

However, after reading through your messages and thinking about this for a few days I think it'd be sufficient to point out sys.executable's existence early on in the subprocess docs and then say something like how python is not reliable. Similarly to what Paul said here: https://bugs.python.org/issue42041#msg378727

I used python in my subprocess call because I thought it would always resolve to the the current interpreter. Clearly I don't work a whole lot with Windows...

Eryk - I really appreciate your explanation and I think that level of detail should be documented. I'd read it because I'm interested, but that is probably too much detail for someone who is trying to get something done quickly.

I think both the quick explanation (and pointing out sys.executable's existence) and the detailed why would have it's place in the docs.
Maybe the details could be documented for educational purposes and not as a guarantee of implementation?
msg378940 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-19 11:10
> I'd read it because I'm interested, but that is probably too much detail
> for someone who is trying to get something done quickly.

I did say that I don't know how much should be documented. I tend to dump a lot of information in issues so that people can make informed decisions. Personally I would keep it limited. I do want more help, and generic help, in the warning, but I don't want to spam the already exhausting and MEGO-prone (i.e. my eyes glaze over) subprocess docs.

With regard to the use of a platform search, note that subprocess actually implements its own search in POSIX. (subprocess supports posix_spawn -- but not posix_spawnp -- on macOS and on Linux with glibc 2.24+, but it's limited to a narrow subset of Popen arguments, excluding `cwd`, and it requires a qualified `executable` path.) A name containing a slash is resolved against `cwd` or the process current working directory. An unqualified name is searched for in the os.get_exec_path(env) search path, based on PATH from `env` or os.environ, else os.defpath. This allows `env` to override the POSIX search path, just as `cwd` in POSIX overrides the current working directory when resolving a qualified args[0] path and `executable`. This is sort of documented by saying that POSIX Popen() works "os.execvp()-like", but to be more correct it should say "os.execvpe()-like". The final "e" in "execvpe" is fundamentally important since PATH can be sourced from `env`.

In Windows, the search path and the working directory used for resolving paths can't be overridden without passing shell=True. subprocess incorrectly documents this by claiming in general that "the function looks for `executable` (or for the first item in `args`) relative to `cwd` if the executable path is a relative path". That wording isn't even correct for POSIX, for which the working directory only applies to args[0] if it contains a slash.

In light of this, how about inserting the following warning:

      Resolving the path of *executable* or the first item of *args* is
      platform dependent even with ``shell=False``. For POSIX, see
      :meth:`os.execvpe`, and note that when resolving or searching for the
      executable path, *cwd* overrides the current working directory and *env*
      can override the ``PATH`` environment variable. For Windows, see the
      documentation of the ``lpApplicationName`` and ``lpCommandLine``
      parameters of WinAPI ``CreateProcess``, and note that when resolving or
      searching for the executable path with ``shell=False``, *cwd* does not
      override the current working directory and *env* cannot override the
      ``PATH`` environment variable. Cross-platform usage can improve
      reliability by using a fully-qualified path for the executable, such as
      from :meth:`shutil.which`.

      On all platforms, using :data:`sys.executable` as the path of the
      executable is the recommended and reliable way to run the current Python
      interpreter.
msg378941 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-19 11:33
My biggest concern with the suggested wording (which in broad terms looks OK to me) is that it reintroduces the issue with the redirector.

By specifically saying that the behaviour in `CreateProcess` applies, we lead the user to the statement that the search path includes "the directory containing the running application", which the user will interpret as the directory of `sys.executable`. So we need to qualify this somewhere by clarifying that "the running application" may not actually be `sys.executable` (without saying what it actually *is*, as that would mean documentin what the redirector does, which Steve wants to avoid).

I don't have a good answer here - we have two conflicting preferences (document the search behaviour but don't document the way the redirector works) and something needs to give. (To be fair, there's a third conflicting priority here, which is me wanting everything to be explicit and clear. I'm willing to give that up if you and Steve can agree on something, though).
msg378950 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-19 14:03
> My biggest concern with the suggested wording ... is that it 
> reintroduces the issue with the redirector.

I thought I addressed that in second paragraph by recommending sys.executable. It could be emphasized that running "python[x][.y]" is unreliable, without going into detailed examples of where it's unreliable on various platforms (i.e. avoid going into detail about issues with naming of binaries, embedding, the search path, redirectors, etc). A separate cross-platform note in venv could advise scripts to use sys.executable, with a link to the subprocess.Popen docs and a reference to the note about the platform-dependent search behavior and unreliability of running "python[x][.y]".

If the redirector issue is mentioned anywhere, I think it should be in the venv docs. It can include a note about the Windows implementation detail to use a redirector for non-symlink virtual environments. Of concern to me here is that the process handle and PID returned by CreateProcess is for the redirector. One can't use the returned process handle or PID with DuplicateHandle or WSADuplicateSocket (i.e. socket.socket.share) to share handles and sockets with a script that's running as a child process. It might seem to still be working, purely by accident, because the parent script is executing the base "python" instead of the redirector, but in that case the child script isn't using the virtual environment. There isn't an official way to support running sys._base_executable with the __PYVENV_LAUNCHER__ environment variable, as multiprocessing implements internally. Maybe the workaround should be incorporated implicitly in subprocess.Popen if `executable`, `args`, or args[0] is equal to sys.executable and sys._base_executable is different. If the latter is implemented, and using sys.executable is strongly advised, it strengthens the case to avoid discussing the redirector entirely.
msg378992 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-10-19 19:16
Proposed alternative based on Eryk's:

      For maximum reliability, use a fully-qualified path for the executable.
      To search for an unqualified name on :envvar:`PATH`, use
      :meth:`shutil.which`. On all platforms, passing :data:`sys.executable`
      is the recommended way to launch the current Python interpreter again,
      and use the ``-m`` command-line format to launch an installed module.

      Resolving the path of *executable* (or the first item of *args*) is
      platform dependent. For POSIX, see :meth:`os.execvpe`, and note that
      when resolving or searching for the executable path, *cwd* overrides the
      current working directory and *env* can override the ``PATH``
      environment variable. For Windows, see the documentation of the
      ``lpApplicationName`` and ``lpCommandLine`` parameters of WinAPI
      ``CreateProcess``, and note that when resolving or searching for the
      executable path with ``shell=False``, *cwd* does not override the
      current working directory and *env* cannot override the ``PATH``
      environment variable. Using a full path avoids all of these variations.
msg379000 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-19 19:39
I like that, I've updated the PR accordingly.
msg379158 - (view) Author: Mark Keller (keller00) Date: 2020-10-20 19:48
+1 to what Steve said
msg379159 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-20 20:02
New changeset 5ab27cc518f614a0b954ff3eb125290f264242d5 by Paul Moore in branch 'master':
bpo-42041: Clarify how subprocess searches for the executable (GH-22715)
https://github.com/python/cpython/commit/5ab27cc518f614a0b954ff3eb125290f264242d5
History
Date User Action Args
2020-10-20 20:02:27paul.mooresetmessages: + msg379159
2020-10-20 19:48:53keller00setmessages: + msg379158
2020-10-19 19:39:53paul.mooresetmessages: + msg379000
2020-10-19 19:16:59steve.dowersetmessages: + msg378992
2020-10-19 14:03:41eryksunsetmessages: + msg378950
2020-10-19 11:33:18paul.mooresetmessages: + msg378941
2020-10-19 11:10:38eryksunsetmessages: + msg378940
2020-10-19 06:06:03keller00setmessages: + msg378927
2020-10-18 22:46:05eryksunsetmessages: + msg378909
2020-10-18 21:17:30eryksunsetmessages: + msg378902
2020-10-18 20:30:17paul.mooresetmessages: + msg378894
2020-10-18 20:06:19eryksunsetmessages: + msg378891
2020-10-18 19:50:24paul.mooresetmessages: + msg378890
2020-10-18 19:19:44eryksunsetmessages: + msg378889
2020-10-18 19:02:24paul.mooresetmessages: + msg378886
2020-10-18 18:07:11eryksunsetmessages: + msg378881
2020-10-18 16:05:33paul.mooresetmessages: + msg378876
2020-10-16 22:30:13paul.mooresetmessages: + msg378772
2020-10-16 22:00:24steve.dowersetmessages: + msg378760
2020-10-16 19:45:54paul.mooresetmessages: + msg378741
2020-10-16 18:17:42steve.dowersetmessages: + msg378729
2020-10-16 17:54:13paul.mooresetmessages: + msg378727
2020-10-16 16:29:33steve.dowersetmessages: + msg378719
2020-10-15 20:41:17paul.mooresetmessages: + msg378698
2020-10-15 20:15:09keller00setmessages: + msg378697
2020-10-15 19:19:12eryksunsetmessages: + msg378694
2020-10-15 18:55:46paul.mooresetmessages: + msg378692
stage: patch review ->
2020-10-15 18:54:00paul.mooresetkeywords: + patch
stage: patch review
pull_requests: + pull_request21683
2020-10-15 17:14:02keller00setmessages: + msg378689
components: - Documentation
versions: + Python 3.7, Python 3.8, - Python 3.9, Python 3.10
2020-10-15 13:20:37gaborjbernatsetmessages: + msg378682
2020-10-15 12:57:58paul.mooresetmessages: + msg378681
2020-10-15 11:51:37steve.dowersetversions: + Python 3.9, Python 3.10, - Python 3.7, Python 3.8
nosy: + docs@python

messages: + msg378676

assignee: docs@python
components: + Documentation
2020-10-15 07:20:20gaborjbernatsetnosy: + eryksun, gaborjbernat
messages: + msg378666
2020-10-15 05:50:57keller00create