classification
Title: Windows Python installers not installing DLL to System32/SysWOW64
Type: behavior Stage: needs patch
Components: Documentation, Installation, Windows Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Decorater, TBBle, docs@python, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: easy

Created on 2017-03-18 10:03 by TBBle, last changed 2017-03-20 10:11 by eryksun.

Messages (7)
msg289801 - (view) Author: Paul "TBBle" Hampson (TBBle) Date: 2017-03-18 10:03
As noted in https://github.com/python/cpython/tree/master/Tools/msi
===
When installed for all users, the following files are installed to
either "%SystemRoot%\System32" or "%SystemRoot%\SysWOW64" as
appropriate. For the current user, they are installed in the Python
install directory.

.\python3x.dll      The core interpreter
.\python3.dll       The stable ABI reference
===

However, at least with the Python 3.5.3 and Python 3.6.0 installers from the official download page, even an all-users install puts the relevant DLLs in the installation directory instead.

This is the both with the command-line option and checking the relevant box during installation.

I've also confirmed that it happens whether you add Python to the path or not. The latter is my use-case as I have multiple versions of Python installed and use the Python Launcher for Windows to select a version to run or virtualenv to build.

Looking at the source, I suspect this feature was completely lost when the MSI build system was rewritten in commit https://github.com/python/cpython/commit/bb24087a2cbfb186b540cc71a74ec8c39c1ebe3a (formerly https://hg.python.org/cpython/rev/e7dbef447157) for issue #23260 which removed all references to SystemFolder or System64Folder
msg289806 - (view) Author: Eryk Sun (eryksun) * Date: 2017-03-18 11:06
3.5+ doesn't install files to system directories -- except the py launcher in %SystemRoot%. This was an intentional change, so I'm flagging this as a documentation issue.
msg289816 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-18 14:37
I don't even think there's documentation to fix, the install page should say where things are installed.

The system directory is for the system to install files - the Python installer is not a system component, it's a developer kit. For most cases, the install directory was "fixed" rather than broken.

If you have particular needs to install a system-wide python3x.dll, I'd suggest crafting your own install script from the files in a normal install (though putting the install directory on PATH should also enable dependency resolution for most processes).
msg289838 - (view) Author: Paul "TBBle" Hampson (TBBle) Date: 2017-03-19 05:19
If this is just a documentation fix, then there's two places that need it:

* https://github.com/python/cpython/blob/master/Tools/msi/README.txt contains the text I quoted in the original report.

* Some kind of release note (https://docs.python.org/3/whatsnew/3.5.html) so that things that have relied on this behaviour in the past (e.g. gvim with the Python3/dynamic build option) can have all their documentation updated to tell people to... I don't know? Grab the embedded version into the gvim install directory? What _is_ the recommended approach here, to give system-wide-installed applications access to a system-wide-installed Python environment?

==============================================

I don't agree that this change is correct. It's inconsistent with the Linux experience of an all-users installation (i.e. anything on the system can link against libpython2.7.so and get the expected behaviour of using the system-installed Python) and I would also consider an All-Users install to *be* a system component, since the intent is clearly to make it available to all users and applications without further effort.

Unlike putting python.exe in the path, the DLLs are version-named, so you don't suffer the conflict of *which Python* you get. The Python Launcher for Windows has taken care of that nicely, and if I have to add all the Python install directories to my path to ensure the DLLs are visible to applications that link against them, that seems to be a regression in the behaviour that launcher was trying to fix.

An issue I see with the embedded installation approach is that if I want to make modules available to something like gvim's Python environment, then I need to maintain those modules distinctly from the modules I maintain on the system level. It also means I need to be modifying gvim's install directory to add that embedded distribution, and that leaves a whole bunch of manual tracking of installed things I need to worry about.

I have the same concerns (manual tracking of things) if I have to add extra infrastructure to manage a copy of python3.X.dll in System32. Searching Google for this issue suggests that many people are just grabbing a random DLL off the internet named python3.5.dll and putting it in System32 when they encounter this problem, because "it used to work with Python 3.2".

Alternatively... Does it make sense to have a "System Component" installer version of Python for use by other applications that want to offer (optional) Python interpreter support? It would be pretty-much identical to the installer we have now, except putting the DLL entry point (python3.x.dll) into the system path, but not having python.exe in the path to confuse command-line usage.
msg289858 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-19 18:41
Maybe I need to look through the history to see whether I actually intended the all-users install to be in System, but I don't think I did (despite writing the documentation linked - it may have been adapted from the old documentation).

Applications that need Python should look in the registry or embed the embeddable distro, depending on whether they are providing access to the user's own Python (e.g. an IDE) or it's an internal implementation detail (e.g. a tool that just happens to be implemented using Python). The former should really offer options for the user to specify a path to the install as well.

Reliably finding the required standard library for a copy of Python that is installed somewhere else is every bit as problematic as locating the install itself. When everyone follows PEP 514 properly it may be safe for a System32-DLL to use the registry again for finding its library, but as long as various distros want to overwrite the core registry keys, everyone is better off without it being a system component.
msg289859 - (view) Author: Decorater (Decorater) * Date: 2017-03-19 18:56
tbh, I would rather have it default for all python files installed for all users to go in ``%SystemDrive%\Python{major}{minor}`` on windows and then work with then the environment variables like currently so that way none of the python dll's are outside of such folder in case someone installs for example 3.6.0 in ``%SystemDrive%\Python360`` and then installs 3.6.1 in ``%SystemDrive%\Python361`` without uninstalling 3.6.0.
msg289882 - (view) Author: Eryk Sun (eryksun) * Date: 2017-03-20 10:11
> It's inconsistent with the Linux experience of an all-users 
> installation

Yes, if you build with --enable-shared on Linux, then the shared libraries libpython3.X.so.1.0 and libpython3.so are installed in /usr/local/lib. 

Currently there's no direct equivalent for 3.5+ on Windows. However, delay-loading the DLL is an alternative to a static import. At program startup, get the install path from the registry and load python3x.dll manually via LoadLibraryEx with the flag LOAD_WITH_ALTERED_SEARCH_PATH. Delayed loading automates calling GetProcAddress, so you get the flexibility of a dynamic import without losing the convenience of a static import.

> add all the Python install directories to my path to ensure the DLLs 
> are visible to applications that link against them

If the 32-bit DLL were distributed as, for example, python36-32.dll, then this would at least be reliable, albeit tedious. Using System32 and SysWOW64 handles this problem reliably via file-system redirection.

> all users to go in ``%SystemDrive%\Python{major}{minor}``

The change to use %ProgramFiles% for a machine installation and %LocalAppData% for a user installation locks down the discretionary file security. In contrast, the file security inherited from C:\ is permissive. It allows any authenticated user the right to modify the directory, subdirectories, and files. The only rights not granted are delete-child (meaningless since the user has delete access for all files) and the right to modify the file security.
History
Date User Action Args
2017-03-20 10:11:44eryksunsetmessages: + msg289882
2017-03-19 18:56:05Decoratersetnosy: + Decorater
messages: + msg289859
2017-03-19 18:41:09steve.dowersetmessages: + msg289858
2017-03-19 05:19:12TBBlesetmessages: + msg289838
2017-03-18 14:37:15steve.dowersetmessages: + msg289816
2017-03-18 11:06:32eryksunsetassignee: docs@python
components: + Documentation, Windows
versions: + Python 3.7
keywords: + easy
nosy: + paul.moore, tim.golden, eryksun, docs@python, zach.ware, steve.dower

messages: + msg289806
stage: needs patch
2017-03-18 10:04:31TBBlesettype: behavior
components: + Installation
2017-03-18 10:03:53TBBlecreate