classification
Title: "pip install --user numpy" fails on Python from the Windows Store
Type: Stage:
Components: Windows Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, mattip, paul.moore, stephtr, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2019-01-08 22:45 by mattip, last changed 2019-01-15 22:37 by steve.dower.

Messages (5)
msg333262 - (view) Author: mattip (mattip) * Date: 2019-01-08 22:45
After enabling Insider and installing Python3.7 from the Windows Store, I open a cmd window and do `pip install --user numpy` which runs to completion. But I cannot `import numpy`. 

The NumPy `mutiarray` c-extension module in the `numpy/core` directory depends on an `OpenBLAS` DLL that is installed into the `numpy/.libs` directory. But even after adding that directory to the `PATH` before running python (and checking with `depends.exe` that the `multiarray` c-extension module is now not missing any dependencies) I still cannot `import numpy`.

See also NumPy issue https://github.com/numpy/numpy/issues/12667
msg333552 - (view) Author: mattip (mattip) * Date: 2019-01-13 10:50
The difference in search order between apps from the app store and desktop applications may be relevant

https://docs.microsoft.com/en-us/windows/desktop/Dlls/dynamic-link-library-search-order#alternate-search-order-for-windows-store-apps
msg333723 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-15 18:18
I posted on the numpy thread: Most likely the DLL is failing to load, which the importer returns as "not found" (as it falls back on other search mechanisms and doesn't retain the error). I suggested loading it directly with ctypes to see if there's a better error indicator.
msg333732 - (view) Author: mattip (mattip) * Date: 2019-01-15 21:44
It seems changing os.environ['PATH'] is a security risk and is not allowed for Windows Store apps. The suggestion in the NumPy issue is to:

- use AddDllDirectory, (which is as accessable as os.environ['PATH'] but is not considered a security risk so far), but this requires using SetDefaultDllDirectories which breaks other things

- put any dlls required for the c-extension pyd in the same directory which means scipy and numpy will be using duplicate and potentially different OpenBLAS dlls, and whoever imports first wins

- load all the required dlls via LoadLibrary, meaning NumPy will have to export a windows-only API to SciPy so the latter can know where the DLL is.

I am glad NumPy only has one DLL, and not a dozen like QT or wxPython. 

Is there a PEP that describes the overall design of windows directory layout or a design guide for package authors with best practices for additional dll dependencies?
msg333735 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-15 22:37
> use AddDllDirectory, (which is as accessable as os.environ['PATH'] but is not considered a security risk so far)

The parenthical is incorrect. The user-specified DLL search directory is separate from PATH, and both appear in the default DLL search order when resolving relative paths. In more secure configurations, PATH is not used for DLL resolution.

> but this requires using SetDefaultDllDirectories which breaks other things

Specifically, it breaks any extension relying on the implicit default search order by enabling one of the more secure configurations.

> put any dlls required for the c-extension pyd in the same directory which means scipy and numpy will be using duplicate and potentially different OpenBLAS dlls, and whoever imports first wins

Doesn't scipy import numpy? Which means numpy wins every time. Or alternatively, put "-numpy" in the name of numpy's one and "-scipy" in the name of scipy's one, and you can have both.

> load all the required dlls via LoadLibrary, meaning NumPy will have to export a windows-only API to SciPy so the latter can know where the DLL is.

Perhaps that API could be exported via normal module import as is currently is? That way scipy can just "import numpy" to locate numpy?

Alternatively, if you do indeed need to have shared state with scipy, then you should come up with an API that they can depend on. This is how shared state normally works.

> Is there a PEP that describes the overall design of windows directory layout or a design guide for package authors with best practices for additional dll dependencies?

No, but there is a doc page that deserves an update: https://docs.python.org/3/extending/windows.html

If we make a dramatic change to CPython here, then there may be a PEP, but it should still defer to the documentation as that is what gets updated over time.

Currently, the best info comes from https://docs.microsoft.com/windows/desktop/Dlls/dynamic-link-library-search-order and awareness that only the LOAD_WITH_ALTERED_SEARCH_PATH flag is used when loading extension modules (see https://github.com/python/cpython/blob/master/Python/dynload_win.c#L221)


Since I just saw the confirmation at https://docs.microsoft.com/en-us/windows/desktop/Dlls/dynamic-link-library-search-order#search-order-using-load_library_search-flags, I don't think we can safely change the LoadLibraryEx option in CPython until we drop support for Windows 7 completely, as the update containing the new flags may not be installed. If/when we do that, it will break any extension relying on unsafe DLL search semantics (that is, anything appearing in the earlier section but not in this section).
History
Date User Action Args
2019-01-15 22:37:35steve.dowersetmessages: + msg333735
2019-01-15 21:44:42mattipsetmessages: + msg333732
2019-01-15 18:18:41steve.dowersetmessages: + msg333723
2019-01-13 10:50:06mattipsetmessages: + msg333552
2019-01-10 13:14:28stephtrsetnosy: + stephtr
2019-01-09 04:46:51xtreaksettitle: "pip install --user numpy" fails on Python from the Windos Store -> "pip install --user numpy" fails on Python from the Windows Store
2019-01-08 22:45:20mattipcreate