classification
Title: Unable to import "site" in an embedded application
Type: behavior Stage:
Components: Installation, Windows Versions: Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: 32-bit 2.7.11 installer creates registry keys that are incompatible with the installed python27.dll
View: 25824
Assigned To: Nosy List: AlexWMF, paul.moore, r.david.murray, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2016-05-11 11:00 by AlexWMF, last changed 2016-05-13 15:46 by steve.dower. This issue is now closed.

Files
File name Uploaded Description Edit
Clipboard29.png AlexWMF, 2016-05-11 13:18 output of both x86 and x86_64 python interpreters
Messages (7)
msg265303 - (view) Author: AlexWMF (AlexWMF) Date: 2016-05-11 11:00
Unable to import "site" error occurs in an embedded application.
I have made small research and found why this module isn't visible for python core.

I have run the python 2.7.11 x86 (for all users) at my win 7x64.
The installer writes the registry key "Software\Wow6432Node\Python\PythonCore\%s\", where "%s" is equal to "32" on my machine.
Difference between x86 and x86_64 registry keys is only in "Wow6432Node" subkey in that key path. 
Tracing the application, I have got to getpythonregpath(...) function. This function reads the registry key to determine "PythonPath".
The python core tries to open the following key "Software\Wow6432Node\Python\PythonCore\2.7-32\PythonPath" and fails due to absence of that key.

In this case, the DLL was compiled with PyWin_DLLVersionString = "2.7-32".
Because of this, the python core can't open this key and the "sys.path" stays not fully filled with paths from registry.

The PyWin_DLLVersionString is compiles from MS_DLL_ID="$(SysWinVer)", which is equal to "2.7-32" on my system.
It's defined in .\PCbuild\python.props file as 
<SysWinVer>$(MajorVersionNumber).$(MinorVersionNumber)</SysWinVer>
<SysWinVer Condition="$(Platform) == 'Win32' or $(Platform) == 'x86'">$(SysWinVer)-32</SysWinVer>

So, the simplest solution of this bug is to fix installer to store proper key
msg265308 - (view) Author: AlexWMF (AlexWMF) Date: 2016-05-11 12:10
I've checked the x86_64 version of python27.dll.
It contains "2.7-32" as well as x86 python dll. So strange, seems like "-32" is always there, independently of target bitness.
msg265311 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-05-11 12:57
As far as I know the registry key is a supplament and not a requirement for a correctly functioning python.  There is an entire PEP about the registry keys and what to do with them...I'm not sure what the current status is on 2.7, but like I said, I believe that lack of registry keys should not keep python from working if it is installed correctly.  Python uses other methods to find its library.  Since you are in an embedded scenario, this probably means that there's something about the environment and/or disk layout that you haven't set up in the way that python expects it to be set up.  You migth try asking on python-list or the #python irc channel on freenode for help.  

I'm guessing this means there's something missing from the ebedding docs with respect to Windows, so I'm going to leave this open until that is confirmed or disproven :)
msg265313 - (view) Author: AlexWMF (AlexWMF) Date: 2016-05-11 13:13
Could you provide link to that PEP?
As I posted in the previous message, both x86 and x86_64 python27.dll have version string "2.7-32" (shown in sys.winver). Is this right? Any of installers (x86 nor x86_64) don't create "2.7-32" subkey.

I understand that python should work without set up of sys.path from the registry, but after start of embedded python instance, sys.path don't have any valid path where 'c:\Python27x64\lib' folder resides. sys.path have ['c:\windows\system32\python27.zip', '.\Dlls', '.\Lib\'] and '.\\' expands to current application directory.
But directory is always different, not points to the python home.
When start the application which embeds x86_64 python, it just exits because can't find the 'site' module when tried to perform Py_InitializeEx(0).
msg265314 - (view) Author: AlexWMF (AlexWMF) Date: 2016-05-11 13:18
The output of both x86 and x86_64 python interpreters is attached
msg265317 - (view) Author: AlexWMF (AlexWMF) Date: 2016-05-11 14:31
I found the following PEP https://www.python.org/dev/peps/pep-0514/

According to this PEP, starting from python 2.7.11 the backward compatibility was broken, because .dll can't read from the following registry keys anymore:
HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag>
HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag>
HKEY_CURRENT_USER\Software\Python\PythonCore\<Tag>
HKEY_CURRENT_USER\Software\Wow6432Node\Python\PythonCore\<Tag>

That means, the python27.dll (x86/x86_64) tries to reads '2.7-32' and never see old keys like '2.7'
msg265478 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-05-13 15:46
The PEP doesn't really apply to 2.7, apart from explicitly expecting tools to still detect 32-bit and 64-bit versions despite sys.winver being the same.

As for sys.winver being incorrect in 2.7.11, that's been fixed (issue25824), but wasn't deemed critical enough to release 2.7.12 immediately. The next update will be released soon with the fix.
History
Date User Action Args
2016-05-13 15:46:12steve.dowersetstatus: open -> closed
superseder: 32-bit 2.7.11 installer creates registry keys that are incompatible with the installed python27.dll
resolution: duplicate
messages: + msg265478
2016-05-11 14:31:20AlexWMFsetmessages: + msg265317
2016-05-11 13:18:29AlexWMFsetfiles: + Clipboard29.png

messages: + msg265314
2016-05-11 13:13:10AlexWMFsetmessages: + msg265313
2016-05-11 12:57:32r.david.murraysetnosy: + paul.moore, tim.golden, r.david.murray, zach.ware, steve.dower
messages: + msg265311
components: + Windows
2016-05-11 12:10:46AlexWMFsetmessages: + msg265308
2016-05-11 11:00:28AlexWMFcreate