classification
Title: Python installation fails when run under system account if there is already an existing installation on the system already
Type: Stage:
Components: Installation Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Jurko.Gospodnetić, steve.dower
Priority: normal Keywords:

Created on 2020-09-17 07:49 by Jurko.Gospodnetić, last changed 2020-09-21 16:28 by steve.dower.

Files
File name Uploaded Description Edit
Python 3.8.3 (32-bit)_20200723053832.7z Jurko.Gospodnetić, 2020-09-17 08:15 a bunch of Python installation log files collected from a debugging session researching this issue
Messages (6)
msg377039 - (view) Author: Jurko Gospodnetić (Jurko.Gospodnetić) * Date: 2020-09-17 07:49
This issue seems to affect all Python 3 Windows installers.

Reproduced on:
- various Windows 10 systems
- one Windows 8 system
- one Windows 7 system
- all tested Python 3+ installers up to and including 3.8.5

Scenario:
1. there is already an existing Python 3+ installation on a Windows system, installed for all users
2. you run a Python 3+ installer (any version, even the one matching the already installed Python version) under the `systemprofile` Windows user account, targeting all users and the same target folder that the currently existing installation is located in
- example how to do this manually:
    - using the Microsoft/SysInternals `psexec` utility to open a command prompt running under the `systemprofile` Windows user account:
      `psexec -i -s cmd`
    - run the Python installer from that command-prompt, in our case it was the following command:
      `python-3.8.5.exe /passive TargetDir=C:\Python38 InstallAllUsers=1 PrependPath=1`
3. result will be any of the following at random:
- failed installation with different Windows error codes, including totally opaque 0x80070643 `Fatal Error during installation`
- successful installation
- seemingly successful installation that does not really install all the Python components, e.g. the whole `Lib` folder may end up being empty
- seemingly successful installation that then can not be uninstalled as the uninstaller reports that it can not find the existing installation on the system
  - this effect can often, but not always, be fixed by rerunning the installer and then running the uninstaller
    - you can run the uninstaller from the command-line using something like:
      `python-3.8.5.exe /uninstall /passive`

Additional notes on produced results:
- each installation seems to generate differently named log files in the `C:\Windows\Temp` folder, i.e. the `systemprofile` user account's `%TEMP%` folder
- the log file names indicate whether each specific component is being installed for the current-user or all-user variants, and those seem to be chosen at random in our problematic scenario
- it seems like the components whose log files are marked as being installed for the current user are not getting installed at all, or may possibly be getting installed to some other undisclosed location

The same does not occur if any of the following is true:
- if the installation is run from an regular end-user account
- if there is no Python currently installed on the system

A workaround we found in our case (Python gets installed as a step in one of our installers) is to basically do (uninstall, install) in a loop with some hand-crafted checking to see if the existing installation is ok (includes running some Python code + checking for some specific Python distribution files we've found to be missing in some scenarios).
msg377040 - (view) Author: Jurko Gospodnetić (Jurko.Gospodnetić) * Date: 2020-09-17 07:50
This also seems related to https://bugs.python.org/issue29033.
msg377044 - (view) Author: Jurko Gospodnetić (Jurko.Gospodnetić) * Date: 2020-09-17 08:15
Some other potentially related issues:
- https://bugs.python.org/issue25164
- https://bugs.python.org/issue32966
- https://bugs.python.org/issue37756

And some other related links:
- https://docs.microsoft.com/en-us/troubleshoot/windows-server/application-management/msi-installation-error-1603
- https://superuser.com/questions/1238473/cant-uninstall-python-3-6-on-windows-10
- https://www.programmersought.com/article/41342430755/
- https://www.programmersought.com/article/41342430755/

And I'm attaching some of the Python installer logs I found from one of the debugging sessions researching this problem. It contains logs from a  few installation runs and you can see there from the first log file in each batch and from the names of the other log files that the initial two failed while the last one was presumably done after an uninstall and worked.
msg377069 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-09-17 22:01
Yeah, this whole issue is basically because Burn (our installer tool) won't do per-machine bundle registration even when it's only installing per-machine packages. So the install block that it would normally throw up  fails if you use a different user account to install it the second time.

> (Python gets installed as a step in one of our installers)

I'm intrigued by this - if Python is a dependency of your application, it's better to repackage it yourself and install to your own directory. The installer is basically there for end-users, but if you're not doing an end-user kind of install there's really nothing gained.

This whole scenario is fairly under-developed though. It comes up quite infrequently, so we don't have any really convenient instructions on how to go about it. Doesn't mean we can't help, but because we don't have a clear idea of what you may be trying to achieve, we'd like to hear about it before telling you how to approach this kind of thing.
msg377087 - (view) Author: Jurko Gospodnetić (Jurko.Gospodnetić) * Date: 2020-09-18 06:42
Hi Steve,

Thanks for a quick reply.

> Yeah, this whole issue is basically because Burn (our installer tool) won't do
> per-machine bundle registration even when it's only installing per-machine
> packages. So the install block that it would normally throw up fails if you
> use a different user account to install it the second time.

Hmmm, I'm not sure this is exactly the case as this seems to happen even when we
install Python using the `systemuser` user account initially and then rerun the
installer using that same user account.

Might I suggest adding this information to some 'known defects/limitations'
Python documentation though? Together with more details on what it implies and
how to deal with it? The current behaviour definitely does not seem like
something one would expect as a user.

> > (Python gets installed as a step in one of our installers)
>
> I'm intrigued by this - if Python is a dependency of your application, it's
> better to repackage it yourself and install to your own directory. The
> installer is basically there for end-users, but if you're not doing an
> end-user kind of install there's really nothing gained.
>
> This whole scenario is fairly under-developed though. It comes up quite
> infrequently, so we don't have any really convenient instructions on how to go
> about it. Doesn't mean we can't help, but because we don't have a clear idea
> of what you may be trying to achieve, we'd like to hear about it before
> telling you how to approach this kind of thing.

Our thinking was that if Python already wrapped up all of its installation logic
to 'work correctly' (clean install, upgrade, uninstall, detecting files already
in use, updating any relevant registry settings or globally installed files,
etc.), we might as well use that, so we run the installer in silent mode and
just process its exit code for stuff like errors, reboot required, etc.
Logically the same process that Microsoft gives as the official way to install
its Visual Studio run-time libraries (not saying that process is not filled with
its own caveats :-D).

We can of course take the installed Python distribution folder content and
package that up in our installer directly, and that would avoid the official
Python installer problems. We were hoping to avoid the problem of manually
updating such a repackaging, e.g. Python deciding to add some more data to the
Windows registry, some add-on installation copes with that and now requires that
information to work correctly as well so now we have to debug/detect that also
and tweak our installer to do the same, deal with any related version updating
logic, etc.

By embedding your installer, our 'Python version upgrade' process basically
becomes:
- pop the new Python installer into our 3rdparty library archive
- update the used Python version configuration
- run the installer build

So if this bad installer behaviour is not going to get fixed, I guess we'll go
and play with getting this working using Python's Windows embeddable zip file
distribution instead.

All suggestions welcome. :-)

Thanks again!

Best regards,
  Jurko Gospodnetić
msg377261 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-09-21 16:28
Python's installer doesn't change that often ;)

Without some information about your scenario, I can't make a recommendation.

However, I will say that if you are including any of your own Python code with your application, and especially any compiled modules, you probably don't want to give users the ability to independently remove or modify your Python install. Much better to include the runtime you have tested with and only update your users to a newer version once you have tested with that one.
History
Date User Action Args
2020-09-21 16:28:40steve.dowersetmessages: + msg377261
2020-09-18 06:42:26Jurko.Gospodnetićsetmessages: + msg377087
2020-09-17 22:01:45steve.dowersetnosy: + steve.dower

messages: + msg377069
versions: + Python 3.9, Python 3.10, - Python 3.5, Python 3.6, Python 3.7
2020-09-17 08:21:57Jurko.Gospodnetićsetversions: - Python 3.9, Python 3.10
2020-09-17 08:21:39Jurko.Gospodnetićsetversions: + Python 3.6, Python 3.7, Python 3.8, Python 3.9, Python 3.10
2020-09-17 08:15:51Jurko.Gospodnetićsetfiles: + Python 3.8.3 (32-bit)_20200723053832.7z

messages: + msg377044
2020-09-17 07:50:31Jurko.Gospodnetićsetmessages: + msg377040
2020-09-17 07:49:13Jurko.Gospodnetićcreate