This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Windows installer: Append Python to PATH instead of prepending it
Type: enhancement Stage: resolved
Components: Windows Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Saxomania, bn_append, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2021-08-17 10:35 by bn_append, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
buildout.txt bn_append, 2021-08-23 20:08
orca.png bn_append, 2021-08-24 00:28 Orca output for appendpath.msi environment table
Pull Requests
URL Status Linked Edit
PR 27889 merged bn_append, 2021-08-22 15:09
Messages (20)
msg399737 - (view) Author: Bastian Neuburger (bn_append) * Date: 2021-08-17 10:35
Hi there,

in our organization Python 3.9 is installed on Windows with the PrependPath option; as expected the Install and Scripts directories are prepended to path.

However if there are Python scripts with the same name as a system command (e.g. a script named ping.py vs. the included ping.exe), the Python script gets preferred if I run ping in cmd.exe or Powershell, which makes sense, since the Python script path is considered before e.g. C:\Windows\System32\. 

Is it possible to either change the option to append the Install and scripts directories to the systems PATH instead of prepending it or add an AppendPath option?

I searched for a discussion why prepending was chosen instead of appending but I didn't find anything.
msg399759 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-08-17 14:56
Prepending is used because it makes the most recently installed version of Python the "active" one. It also ensures that a deliberate install is going to override any applications that may have put their own copy on PATH (deliberately or otherwise). There's no real mechanism for managing multiple Python entries on PATH, so it just works out better to prepend and let the latest install win.

However, we also disable this option by default because it can lead to confusion like what you've experienced. The best option here is to leave it disabled and update PATH yourself (or use "py.exe" which is always on PATH), since you've got the best awareness of your environment.
msg399935 - (view) Author: Sascha (Saxomania) Date: 2021-08-19 21:04
hi,

if you know that it will lead to confusion why is this confusion not written here:

https://docs.python.org/3/using/windows.html

If you do software deployment in a company you will only have one version so having more than one version is not the point. And if you make append path optional available like prepend everybody can choose. Anyway.

What i have seen now is if you uninstall Python it is not cleaning up the path variable. Is there any parameter i missed?
msg399936 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-08-19 21:31
> why is this confusion not written here

From https://docs.python.org/3/using/windows.html#configuring-python

> While the installer provides an option to configure the PATH and PATHEXT variables for you, this is only reliable for a single, system-wide installation. If you regularly use multiple versions of Python, consider using the Python Launcher for Windows.

Now, that isn't *exactly* the issue in the original post, but the warning is present.

> If you do software deployment in a company you will only have one version so having more than one version is not the point.

Unfortunately not true. It's been a while since I had actual data on this, but last time I did the *majority* had 2 versions installed, and 3-5 versions were all more likely than only a single one. So handling multiple versions matters.

> What i have seen now is if you uninstall Python it is not cleaning up the path variable.

We do the best we can by using the native Windows support for installing environment variables. It sometimes doesn't work if you (or another installer) has modified the variable since you installed Python, or if you've updated it yourself. That's out of our control - we can't do a better job than the OS.

> if you make append path optional available like prepend everybody can choose

Contributions are welcome. We aren't paid to implement features - we volunteer to manage contributions, so if you'd really like this to be in the installer, feel free to go for it. (Last time the installer was modified we had most trouble making the UI not break, but it's probably not too hard if this was a command-line only.)
msg399944 - (view) Author: Bastian Neuburger (bn_append) * Date: 2021-08-20 09:20
Hi Steve,

Of course there are various reasons for having multiple Python versions on a developer workstation but I would assume that the majority of Windows installer users simple want to run Python projects, at least this holds true for the majority of Python on Windows users in my org.

> if you'd really like this to be in the installer, feel free to go for it.

could you point me to the relevant files and lines for the PrependPath CLI option and implementation logic on Github? Not sure if I got it right, but I assume the following file is in charge of modifying the env:path:

https://github.com/python/cpython/blob/main/Tools/msi/path/path.wxs

Since I have never worked within the MSI framework, I could not figure out how the "call flow" from enabling the definition of the PrependPath in
https://github.com/python/cpython/blob/main/Tools/msi/bundle/bundle.wxs 
to actually modifying the path

Regarging path.wxs: Would you prefer that I add a pathappend/pathappend.wxs or should I simply add a new Component ID like so:

<Component Id="AppendPath_LM" Directory="InstallDirectory" Guid="*">
    <Condition>ALLUSERS=1</Condition>
    ...

Or would I introduce a new condition ALLUSERS_APPEND or something?
msg399994 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-08-20 20:41
The logic is probably simplest if you clone the Tools/msi/path project entirely as pathappend. That will technically allow people to both prepend and append if they set both flags, but I think it's okay to say "don't do that", especially if AppendPath is only a command line option.

Per-component or optional features are complicated and best avoided if we can. For a simple on-off like this, a whole MSI is actually the simplest way to manage it.
msg399999 - (view) Author: Bastian Neuburger (bn_append) * Date: 2021-08-20 22:11
Thanks for the rundown, after digging a little bit into the abyss that is the WIX toolset if you've never dealt with it before I already started going down that route. 

I pushed it to my fork of cpython:
https://github.com/bneuburg/cpython/commits/issue44934

Since I'm pretty sure that my commits are not correct yet (following the instructions in Tools/msi/README.txt regarding fetching dependencies building the MSI resulted in a failed build before I even touched a single line of code) I abstained from handing in a pull request.

Feedback would be very much appreciated. Of course if you'd want to use that stuff and refine it I would be totally fine with it ;)

In case I messed up any formal requirements (i.e. wording of commit messages etc.): I will get myself up to speed and fix that, but since I'll be offline for a week starting Monday I wanted to get my initial draft out there, so in case you have any feedback until Monday I can incorporate it before my brain pushes that stuff out...
msg400000 - (view) Author: Bastian Neuburger (bn_append) * Date: 2021-08-20 22:30
Maybe this view better shows what I've actually done:

issue44934?expand=1">https://github.com/python/cpython/compare/16b9be4861e007ad483611ba0479feb2b90ea783...bneuburg:issue44934?expand=1
msg400001 - (view) Author: Bastian Neuburger (bn_append) * Date: 2021-08-20 22:35
Previous URL got "fixed" by roundup, hopefully this one works:

issue44934">https://github.com/python/cpython/compare/16b9be4861e007ad483611ba0479feb2b90ea783...bneuburg:issue44934
msg400003 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-08-20 22:48
That branch looks like you're on the right track. The build scripts in 
Tools/msi should Just Work, so I'd be interested to see the error you get.

You can also submit it as a PR and the MSI will be built automatically.

One thing I noticed is updating the name of the "Add Python to PATH" 
option, which I'd rather keep as "Add" since we won't have both options 
in the UI. The word "Prepend" is quite obscure jargon, and for the range 
of users we have it isn't fair to expect them to know it (or to break 
all of the guidance out there explaining that checkbox already).
msg400077 - (view) Author: Bastian Neuburger (bn_append) * Date: 2021-08-22 15:18
Steve, I signed the CLA and created the PR, I don't know how long it takes until the info that the CLA is signed is propagated to bpo or github.

Regarding failed builds: I will try to build again tomorrow when back at the office and can copy the errors, how would you like to receive them?

For building I spun up a MS Windows 10 Edge Test VM, installed Visual Studio 2019 and then tried to run get_externals.bat; that already failed since the nuget config was empty and thus had no upstreams configured. Maybe this problem stems from the free test VM. After editing the nuget config, that script worked out fine and then I run build.bat, which seemed to be quite busy for at least 10 minutes, then I left my office. Upon return I saw plenty of errors.
msg400148 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-08-23 16:37
I kicked off the build on the PR so we can see what errors it finds. Nothing jumps out as obviously wrong in the changes, but it only takes one missing reference to a file to break it all.

If you want to share the errors from your test machine, copy-pasting all the output into a text file and attaching it here is best.
msg400172 - (view) Author: Bastian Neuburger (bn_append) * Date: 2021-08-23 20:08
Attached is the full output of:

- msbuild.exe -version
- get_externals.bat
- build.bat

For the first few build tasks there are git related errors, which make sense since I ran the build from a downloaded zip release of the source instead of a git clone. I assume the first "real" error occurs when building the wix targets, just grep for MSB4062, this is when things go downhill.
msg400177 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-08-23 20:57
Oh I remember this one. The Wix toolset still requires .NET 3.5, which you have to install manually these days. Because it's being launched as a console app, the UI never gets triggered.

I thought we had this in the devguide already... but maybe not (or not prominently enough). Might be worth trying to add some detection for it - hopefully a new version of Wix will move to newer .NET too.

See https://docs.microsoft.com/en-us/dotnet/framework/install/dotnet-35-windows-10 for install instructions. It should work after that. The CI build succeeded, but it'll need manual testing.
msg400178 - (view) Author: Bastian Neuburger (bn_append) * Date: 2021-08-23 21:13
Ah ok. Do you know if VS 2019 will do the job at all? I encountered a few (unrelated) projects on github that state they will only build with VS 2017.

I will install .NET 3.5 via the VS 2019 installer (should do the trick, right?) and see how it turns out. 

Could you attach the msi installer build artifact from the build you kicked off so I can test if the appendpath stuff works as expected? Or did you already have a look at it?

P.S.: bpo now recognizes that I signed the CLA, github not yet.
msg400179 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-08-23 22:01
I don't think we upload the artifact from the MSI builds, sorry.

If Visual Studio has an option for the older version of .NET, then yeah, that'll do it. But because it was an OS component, you may have to do it through the OS dialog in the link I posted.

GitHub will get the CLA update soon enough. It's always a little slower - bpo is the canonical source.
msg400187 - (view) Author: Bastian Neuburger (bn_append) * Date: 2021-08-24 00:28
Update: .NET 3.5 installation worked out fine via VS 2019 installer, I also added the VS2017 build tools as a precaution since the VM builds take some time.

Afterwards build.bat still failed during wix steps, complaining about missing python.exe in PCBuild/win32, not sure if the 32bit interpreter should be fetched/built automagically. Then I tried build.bat -x64, which ran succesfully and provided attached installer.

To test it (in administrative shell):
1. git clone https://github.com/bneuburg/python_msi_appendpath
2. cd python_appendpath_installer
3. .\python-3.11.0.7539-amd64.exe /quiet InstallAllUsers=1 AppendPath=1 Include_test=0

Running this in an administrative shell appended InstallDir and Installdir/Scripts to the sytem PATH and created the PATHEXT entries for PY and PYC as well. So far so good!

I only noticed the following problem:
The first uninstall also succesfully removed all environment variables, however after reinstalling and re-uninstalling, only the PATHEXT were removed, the Python entries in PATH were not removed. This was also independent of the order of software removal (Python Launcher first or second). 

Thus I had a look at it with orca.exe [1]:
1. Install with appendpath=1
2. Get list of installed msis with: wmic product get /format:csv > software.csv
3. Search for appendpath in software.csv which will return the location of the msi somewhere in c:\windows\installer
4. Right click that file and select 'Edit with Orca'

In the environment table it has entries in the following form for a Local Machine install (c.f. for attached screenshot):

Environment  | Name    | Value                  | Component 
PATH_LM      | =-*PATH | [~]:[InstallDirectory] | AppendPath_LM

which according to Microsoft documentation [2] seems to be as intended (add on install, remove on uninstall). But it just doesn't, also see Sascha's comment above. When looking at Orca I also noticed that I forgot to change the position of InstallDirectory in PATH for the CU install from first to last, I updated the PR accordingly.

Since you have a Microsoft e-mail address: Any chances you could ask the MSI pros there what is going on?
I'll be busy with offline stuff for a few weeks, but if you have any ideas how to troubleshoot this, let me know. I found a discussion where the order of the prefixes for names was slightly changed [3], i.e. *=- instead of =-* but I don't know if this matters at all or how to test it.

Sorry for the text book, things like that really keep me digging.

[1] https://docs.microsoft.com/de-de/windows/win32/msi/orca-exe
[2] https://docs.microsoft.com/en-us/windows/win32/msi/environment-table
[3] https://www.itninja.com/question/add-to-path-envronment-variable
msg400230 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-08-24 19:22
Removing environment variables is "known" to be unreliable - another reason I don't like modifying them from an installer. It's meant to remove the item from the list based on value, so if anything has changed then it's not a huge surprise it will fail.

I'm more inclined to think that the "[~]:" prefix might be up to something. I forget exactly what it means, but if it's not all resolving to an empty string, it might concatenate fine but then mess up the removal?
msg410836 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2022-01-18 00:19
New changeset c47c9e6589eb7a272cfe4d352eb87389eb20ec2f by bneuburg in branch 'main':
bpo-44934: Add optional feature AppendPath to Windows MSI installer (GH-27889)
https://github.com/python/cpython/commit/c47c9e6589eb7a272cfe4d352eb87389eb20ec2f
msg410837 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2022-01-18 00:21
Thanks for the PR! This is a good contribution, that wasn't trivial to do. It should be in Python 3.11 alpha 5, so please test it out and make sure it's behaving as you expect.
History
Date User Action Args
2022-04-11 14:59:48adminsetgithub: 89097
2022-01-18 00:21:05steve.dowersetstatus: open -> closed
resolution: fixed
messages: + msg410837

stage: patch review -> resolved
2022-01-18 00:19:05steve.dowersetmessages: + msg410836
2021-08-24 19:22:38steve.dowersetmessages: + msg400230
2021-08-24 00:28:14bn_appendsetfiles: + orca.png

messages: + msg400187
2021-08-23 22:01:46steve.dowersetmessages: + msg400179
2021-08-23 21:13:37bn_appendsetmessages: + msg400178
2021-08-23 20:57:21steve.dowersetmessages: + msg400177
2021-08-23 20:08:06bn_appendsetfiles: + buildout.txt

messages: + msg400172
2021-08-23 16:37:51steve.dowersetmessages: + msg400148
2021-08-22 15:18:31bn_appendsetmessages: + msg400077
2021-08-22 15:09:48bn_appendsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request26343
2021-08-20 22:48:49steve.dowersetmessages: + msg400003
2021-08-20 22:35:27bn_appendsetmessages: + msg400001
2021-08-20 22:30:20bn_appendsetmessages: + msg400000
2021-08-20 22:11:58bn_appendsetmessages: + msg399999
2021-08-20 20:41:49steve.dowersetmessages: + msg399994
2021-08-20 09:20:20bn_appendsetmessages: + msg399944
2021-08-19 21:31:35steve.dowersettype: behavior -> enhancement
stage: needs patch
messages: + msg399936
versions: + Python 3.11, - Python 3.9
2021-08-19 21:04:23Saxomaniasetnosy: + Saxomania
messages: + msg399935
2021-08-17 14:56:05steve.dowersetmessages: + msg399759
2021-08-17 10:35:52bn_appendcreate