Title: Broken MSIs in Python 3.5+
Type: behavior Stage: resolved
Components: Installation, Windows Versions: Python 3.6, Python 3.5
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: KeyWeeUsr, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-01-10 21:14 by KeyWeeUsr, last changed 2017-01-11 19:50 by KeyWeeUsr. This issue is now closed.

Messages (8)
msg285157 - (view) Author: KeyWeeUsr (KeyWeeUsr) Date: 2017-01-10 21:14
In versions _lower_ than 3.5.0 there was a `.MSI` installer for Windows, which had a really nice hidden option. That option looked like this:

    msiexec.exe /a "file.msi" /qb /L*V "file.log" ALLUSERS=0 TARGETDIR="target" CURRENTDIRECTORY="%cd%" <additional options>

which basically allows me to ignore admin rights, because the MSI turns to some kind of installer for whole network, thus the permission workaround for older Python versions.

With Python 3.5.0 was introduced a new `.EXE` installer, which has `.MSI` files packed in itself, you can get them out with:

    python-3.5.0.exe /layout [optional target directory]

yet there's a really annoying thing going around with this solution. When I do this, the `.MSI` files have a `-d.msi` suffix and when I manually unpack it with the `msiexec` command above, **every file** has that suffix too, which makes it a _completely damaged_ installation. Renaming the files isn't really an option as each file has `-d.<file ext>`, not `-d.msi.<file ext>` suffix, so it's too hard to rename it in a simple way with tools such as Batch, unless I want to check for multiple cases (e.g. folders).

Is there any way how to extract the content of the installer to a separate folder, like it was possible before **without** any additional stuff put into `Programs and features`, such as `Python 3.5.0 (64bit)` or similar?

Or other question - is there any way how to forbid the installer to access `Programs and features`, shut it from asking admin privileges and registry at all?

It's quite useful if I want to have multiple python fresh installations not bound to anything at all with testing as its main purpose. Note, that I have no intention to use python launcher (that `py.exe` thing), virtualenv or any other alternative "solution" as every of them allows me to install only a one **global** Python installation of the same version and/or is too big for quick usage and at the same time I need to have an option for testing on a fresh interpreter not modified by anything and completely cut off from the OS stuff.

Any ideas how can I achieve the same behavior even on newer Python versions with the EXE installer?
msg285240 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-01-11 16:45
I discussed installers on my blog at a while back. There are some other options linked from there, and also rationale for why things are how they are.
msg285247 - (view) Author: KeyWeeUsr (KeyWeeUsr) Date: 2017-01-11 17:32
Re blog post "Why are there so many Python installers?"

> As a result, the old installer always requires administrative privileges just in case you choose to install for all users. This prevents installation of Python on machines where you do not have full control over the system.

So not true. Read the first command I used, that doesn't require any permission at all. I was able to install Python without any registry or P&F stuff at "Guest" account in an internet cafe where you couldn't even throw stuff to the Trash with the same MSI installer. Notice the "/a" switch, not "/i" (install).

Secondly, I'm glad for the change, but the installer is still in an imperfect shape - leaves traces. That makes you able to use only a single version (as mentioned before). Unless, of course, you don't want to copy&paste the whole folder, which although works is also stupid and still requires at least one successful installation with traces in the system.

> Always requires administrator privileges <-- no
Only allows installation for all users <-- no
Only allows configuration at the command line (via msiexec) <-- not sure what the "configuration" means in this context
Prevents the executable installer from installing for all users <-- I didn't really use that, but... what? Seems weird. Have you tried with "/a ALLUSERS=1"? It should purge the elevation.

Anyway, this has nothing to do with the issue, because what I'm talking about are MSIs that are _extracted_ from the current EXE installer via "/layout" switch. There comes the problem - you can't install from those MSIs (not directly, but probably works with EXE as a casual python installation - I don't want that), because the MSIs are ***corrupted*** in some weird way, probably some debug command is called together with the "/layout" judging from the "-d" string in the filenames - useful only for debugging the inside of the MSIs, otherwise completely unusable. Please, just try to do what I wrote to see what's actually going on.

If there is any way how to fix the "/layout" to unpack correctly, ++ for the new installer. Otherwise, although the EXE has a lot of features I use for system-wide interpreter, I hate the change.
msg285256 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-01-11 18:26
"msiexec /a" does not install anything - it converts the MSI into a source layout for lazy network installs. It is, and always has been, completely unsupported as an approach to installing Python, and we are under no obligation to maintain it.

Copy-pasting the install folder after installation is also unsupported. We obviously can't stop you doing it, and at times it is the best way to achieve certain things, but it is not supported and we do not have to make it work better.

The /layout command is intended for full offline installs: it downloads all optional packages without evaluating any conditions and puts them in a location where they will be used later without redownloading.

You've always been able to use only a single micro version of Python at a time when installing it. That has not changed (and will not, as it's by design). We have added new package formats that do not install the full development kit, and it sounds like the nuget packages (a.k.a. zip files by a different name) are perfect for you.

I have no idea what the "-d" suffix is, unless you mean the "_d.msi" modules which contain different files from the non-"_d" versions. The internals of the installer are just that - internals - but FYI there are typically 3 installers for each component. For example, "exe.msi" contains the files related to launching an interactive shell (including licenses, shortcuts, and icons); "exe_pdb.msi" contains the PDB files associated with files in exe.msi; "exe_d.msi" contains a debug build for those who are debugging native extensions and want consistent build settings between Python and their extension.

Your criticism of the *hypothetical* single MSI installer doesn't make any sense, since that installer does not exist. I included those points to show what restrictions would be necessary to avoid any such single MSI installer breaking users who would prefer to use the EXE installer.

There is no bug here, so I'm closing the issue.
msg285262 - (view) Author: KeyWeeUsr (KeyWeeUsr) Date: 2017-01-11 18:59
python-3.5.2.exe /layout <folder>

11.01.2017  19:30    <DIR>          .
11.01.2017  19:30    <DIR>          ..
11.01.2017  19:30         3 035 136 core_d.msi
11.01.2017  19:30         2 240 512 core_pdb.msi
11.01.2017  19:30            98 304 dev_d.msi
11.01.2017  19:30           110 592 exe_d.msi
11.01.2017  19:30           102 400 exe_pdb.msi
11.01.2017  19:30         5 378 048 lib_d.msi
11.01.2017  19:30         4 820 992 lib_pdb.msi
29.10.2016  11:13        29 269 656 python-3.5.2.exe
11.01.2017  19:30         1 179 648 tcltk_d.msi
11.01.2017  19:30           139 264 tcltk_pdb.msi
11.01.2017  19:30           331 776 test_d.msi
11.01.2017  19:30           286 720 test_pdb.msi
              12 File(s)     46 993 048 bytes
               2 Dir(s)   1 245 736 960 bytes free
msiexec /a exe_d.msi targetdir=%cd%\<folder>

Msiexec window has title "Python 3.5.2 Executables (32-bit debug)" and the targetdir contains:

11.01.2017  19:36    <DIR>          .
11.01.2017  19:36    <DIR>          ..
25.06.2016  21:57            51 712 pythonw_d.exe
25.06.2016  21:57           364 544 pythonw_d.pdb
25.06.2016  21:57            51 712 python_d.exe
25.06.2016  21:57           364 544 python_d.pdb
               4 File(s)        832 512 bytes
               2 Dir(s)   1 264 963 584 bytes free

Notice, that everywhere is "_d" I talk about so much.

> ...and puts them in a location where they will be used later without redownloading...

Yes, in a state that only the EXE installer is able to use them.

> *hypothetical*

msiexec /a python-2.7.13.msi targetdir=%cd%\<folder>

11.01.2017  19:45    <DIR>          .
11.01.2017  19:45    <DIR>          ..
11.01.2017  19:45    <DIR>          DLLs
11.01.2017  19:45    <DIR>          Doc
11.01.2017  19:45    <DIR>          include
11.01.2017  19:45    <DIR>          Lib
11.01.2017  19:45    <DIR>          libs
17.12.2016  20:49            38 591 LICENSE.txt
17.12.2016  20:34           474 595 NEWS.txt
11.01.2017  19:45           835 584 python-2.7.13.msi
17.12.2016  20:44            27 136 python.exe
17.12.2016  20:43         2 639 872 python27.dll
17.12.2016  20:44            27 648 pythonw.exe
03.12.2016  21:01            56 938 README.txt
11.01.2017  19:45    <DIR>          tcl
11.01.2017  19:45    <DIR>          Tools
17.12.2016  20:44           111 104 w9xpopen.exe
               8 File(s)      4 211 468 bytes
               9 Dir(s)   1 162 199 040 bytes free

Fully usable, portable and working Python interpreter for Python installers <3.5.0 without a single beep about admin rights (kivy, numpy, scipy, cython, mingwpy, etc.). With some workaround probably even usable for ALLUSERS=1, but I'm not going to try that on my system.

How is Debug installer not an issue/bug in a Python Release? Am I missing something or is it ok to unpack Debug installers from an official installer and not the actually used ones, the ones that produce Release that works?
msg285263 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-01-11 19:26
Ah, you're using the full installer, which has the non-debug versions embedded and therefore does not ever need to download them.

If you start from the web installer, it will download the MSIs for the non-debug components as well. But as I said, you're in totally unsupported territory by not running the proper installer. See if the nuget packages are more helpful.
msg285264 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-01-11 19:27
> Fully usable, portable

*and unsupported* - you apparently ignored that point from Steve's comment

> and working Python interpreter

That's (essentially) pure luck. I don't think I've ever seen any
suggestion that the /a flag for *any* MSI installer (Python or
otherwise) will produce a working install - it's designed (and
documented by Microsoft) as being to allow system admins to unpack the
MSI onto a network share from where it can be installed onto user
machines (for details see
msg285267 - (view) Author: KeyWeeUsr (KeyWeeUsr) Date: 2017-01-11 19:50
> Ah, you're using the full installer, which has the non-debug versions embedded and therefore does not ever need to download them.

Hm, makes sense now, however, isn't that a bug? I mean, even if it's embeded... I'm just curious ^^

> If you start from the web installer...

Aha, there's my problem! After using the web installer everything seems ok for me. Python now runs happily!

RE the unsupported: yeah, I kind of ignored that intentionally. Don't get me wrong, but the functionality you so badly want to push away is a wonderful thing. Ok, there's venv, I'm mostly satisfied with that or system-wide interpreter, but there are cases when you have to debug some annoying problems and you might broke the installation before thus a hypotetical issue arises. Maybe you deleted a file unintentionally, put a header for compilation here or there, renamed something...

What is the first thing someone asks you on for example IRC? "Have you tried a fresh Python installation?" Uhm, sure, because it's so easy if the installer throws things everywhere it can (registry, p&f, ...).

That's mostly not an issue, simple "Repair" is enough unless... you have a large build environment and reinstalling the environment from scratch could take a **long** while depending on the complexity of the environment. How do you test a fresh installation then? <-- Really, that's a serious question. I could come only with this before, though nuget coule be usable too ^^

Also, the network installer is almost the same thing as common MSI, but without elevation. It's like ~500kB or something and does basically this - adds keys to registry, to p&f, etc with a folder that you choose for installing via the network installer afaik. I just skip the part that makes Python unportable or burned into the system. What I do is basically what you do after compilation. Take binaries and put them somewhere, so that they run, nothing less, nothing more.

Thanks for your help, guys :)
Date User Action Args
2017-01-11 19:50:40KeyWeeUsrsetresolution: not a bug -> works for me
messages: + msg285267
2017-01-11 19:27:10paul.mooresetmessages: + msg285264
2017-01-11 19:26:59steve.dowersetmessages: + msg285263
2017-01-11 18:59:13KeyWeeUsrsetmessages: + msg285262
2017-01-11 18:26:57steve.dowersetstatus: open -> closed
resolution: not a bug
messages: + msg285256

stage: resolved
2017-01-11 17:32:42KeyWeeUsrsetmessages: + msg285247
2017-01-11 16:45:43steve.dowersetmessages: + msg285240
2017-01-10 21:31:18KeyWeeUsrsetversions: + Python 3.6
2017-01-10 21:14:07KeyWeeUsrcreate