classification
Title: [Windows] Add global python and python3 commands
Type: enhancement Stage: resolved
Components: Windows Versions: Python 3.9
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, lukasz.langa, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2020-05-18 09:06 by steve.dower, last changed 2020-05-27 17:08 by steve.dower. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 20226 closed steve.dower, 2020-05-19 13:37
Messages (9)
msg369196 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-05-18 09:06
So I'm totally sick of being abused on the internet about the whole PATH issue, and want to put this up as an alternative. I'm willing to do the work, but I'm worried that my emotional state about the whole thing is blinding me to obvious issues.

In short, add a new option to the installer to "Add global python3.exe and python.exe commands". Also replace the current PATH option on the first page (not under Customize) with this. I'll also embed a "Learn More" link to the (new) doc page that explains the impact.

The option will pass an extra flag to the *launcher* installer to include more copies of py.exe, renamed to python3.exe and python.exe (we already pass a similar option to add file associations). These additional copies will be just as global as py.exe (depending on whether the user is admin or not) and obviously work identically.

Importantly, this does *not* drop an arbitrary/writable directory on PATH, and it keeps the DLLs out of the search path. 

There are some risks:
* these may overtake people's existing installs and replace them with something more clever (this annoys people)
* these add an extra process in the chain, which can break callers who need the process ID/handle (this annoys people)
* this won't put Scripts on PATH (this annoys people)
* changing anything at all annoys people
* would remove python3.dll and python39.dll from PATH (probably for the best), and also vcruntime*.dll (which is a *good* thing) and anything installed by packages/tools into sys.prefix (which is a *massive* security vulnerability)
* the help text for the launcher is not obvious above the help for the default Python interpreter
* shebang lines will start being noticed
* things should work in a venv because of both PATH and VIRTUALENV variables, but I'm not going to stake my life on this
* "python3.exe" could launch Python 2, but I'm not too concerned by this

I feel like this option is the best of a bad lot. But if people weigh in and say it's worse than doing nothing, then I'll do nothing.

(Including Ɓukasz in case he wants to veto adding this to 3.9 after beta 1. It's not a language change, and barely even an installer change - just a few optional copies of the existing launcher plus some docs. But as I mentioned above it *will* break some people.)
msg369238 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-05-18 15:00
> these may overtake people's existing installs and replace them 
> with something more clever (this annoys people)

In particular, installing the launchers to %SystemRoot% means that in some contexts, such as the search path that CreateProcessW and SearchPathW use, the launcher commands will take precedence over %PATH%.

> shebang lines will start being noticed
> "python3.exe" could launch Python 2, but I'm not too concerned by this

The launcher can get the name of the current process to restrict its behavior. For "python3.exe", run as "py.exe -3", which is subject to PY_PYTHON3. For "python2.exe", run as "py.exe -2", which is subject to PY_PYTHON2. For "python.exe", run as "py.exe", which is subject to shebangs, VIRTUAL_ENV, and PY_PYTHON.

Since the loader doesn't resolve symbolic links, these extra names can actually just be symlinks if the filesystem supports them and the installer is allowed to create them. If installing the launchers to %SystemRoot%, the filesystem must be NTFS and we must have admin access, so symlinks should be supported.

> these add an extra process in the chain, which can break callers 
> who need the process ID/handle

The launcher could facilitate finding the image path when programs need to spawn the interpreter directly as a child process. Maybe when running as "python2.exe" and "python3.exe", the launcher's -0p option could print just the image path that it would execute instead of the list of installed versions.
msg369241 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-05-18 15:15
> For "python.exe", run as "py.exe", which is subject to shebangs,
> VIRTUAL_ENV, and PY_PYTHON.

Maybe in the "python[w].exe" case, it should skip shebang processing. That way only py[w].exe would handle shebangs, such as for the template command that gets installed as the default .py[w] file association: C:\Windows\py[w].exe "%1" %*.
msg369255 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2020-05-18 15:49
I'm in favor of this.

I especially want this with the self-name-checking that Eryk mentioned, and even more if each installed Python adds its own symlink to `py.exe` with its own version (i.e. the Python 3.9 installer adds `python3.9.exe -> py.exe`).  I'm good with the base change even without these enhancements, though.

(Maybe at some point in the future we can do something crazy like have the launcher just load the appropriate python3X.dll and run as the python process itself, but that is its own can of worms and should not stand in the way of this improvement :))
msg369257 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-05-18 15:54
The self-name checking is not something I'm volunteering to write, btw, nor any new options. But feel free to put those up as separate issues.

I'm also not getting into symbolic links here - WiX doesn't support them natively, and custom actions are the source of 99% of issues, so I don't want to add one. It'll be multiple complete copies of py.exe (which the installer will compress).

And I wasn't even going to bother with pythonw.exe. Literally nobody has ever complained about those.
msg369259 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-05-18 15:55
Oh, and there certainly won't be a python2.exe :)
msg369385 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-05-19 20:09
If .py files are associated with py.exe, handling the shebang "#!/usr/bin/env python[3]" might spawn the python[3].exe launcher recursively if it handles shebangs (see bpo-40687). To avoid this, process() can set a flag based on the image name that prevents calling maybe_handle_shebang(). 

I still think it would be for the best if the python3.exe launcher ran as "py -3", and the python.exe launcher ran as "py", based simply on checking the image name. Also, I think these two launchers should drop all command-line option support, i.e. -h, --help, --list, --list-paths, -0[p], and -X[.Y][-32|-64]. This keeps the command-line interface for the "python" command consistent across the board. PY_PYTHON3 would set the default for the "python3.exe" launcher, and PY_PYTHON would set the default for the "python.exe" launcher. For whatever reason, PY_PYTHON could be set to 2 to make the python.exe launcher run 2.x. (Even in Ubuntu I install the python-is-python3 package, but sadly some people are stuck maintaining 2.x code.)
msg369396 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-05-19 21:48
> To avoid this, process() can set a flag based on the image name that prevents calling maybe_handle_shebang(). 

Or the other solution, which is stop treating "usr/bin/env" as "search PATH" - I'm sure there's another issue open somewhere saying this is the best approach.

At the same time, one other gap is that this would leave the global Scripts folder off of PATH. On one hand, good (for all the same reasons). But on the other hand, people legitimately complain about it and we don't yet have a workaround.

Since the whole point of this exercise is to minimise complaining, it's looking better to just leave people with the devil they know...
msg370111 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-05-27 17:08
After thinking this through some more, I don't think this solves the problem, but only changes it. And we're better off leaving things as they are than just moving them around.
History
Date User Action Args
2020-05-27 17:08:07steve.dowersetstatus: open -> closed
resolution: rejected
messages: + msg370111

stage: patch review -> resolved
2020-05-19 21:48:57steve.dowersetmessages: + msg369396
2020-05-19 20:09:11eryksunsetmessages: + msg369385
2020-05-19 13:37:42steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request19517
2020-05-18 15:55:35steve.dowersetmessages: + msg369259
2020-05-18 15:54:45steve.dowersetmessages: + msg369257
2020-05-18 15:49:59zach.waresetmessages: + msg369255
2020-05-18 15:15:31eryksunsetmessages: + msg369241
2020-05-18 15:00:19eryksunsetnosy: + eryksun
messages: + msg369238
2020-05-18 09:06:49steve.dowercreate