This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: python3.dll export forwarders not resolved in all situations due to missing python3?.dll dependency and DLL search path behavior
Type: behavior Stage:
Components: Windows Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Elli Pirelli, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-01-31 17:14 by Elli Pirelli, last changed 2022-04-11 14:58 by admin.

Messages (11)
msg286544 - (view) Author: Elli Pirelli (Elli Pirelli) Date: 2017-01-31 17:14
Let me first apologize if this issue has already been reported and discussed before. I would be surprised if there are not a few developers that had to deal with this issue before, but i was unable to find an existing bug report or discussion about this. So, here it goes...


Importing/using Py_* symbols from python3.dll instead directly from python3?.dll can be a problem when python3.dll and python3?.dll reside in a directory that is not part of the standard DLL search path. An example would be an application that tries loading Python from a sub directory of the application.

The functions exported in python3.dll are export forwarders referencing the respective functions in python3?.dll.
python3.dll itself has no import dependency on python3?.dll.

Without an import dependency on python3?.dll, this DLL will be loaded when one of the export forwarders in python3.dll will be resolved, for example when calling GetProcAddress("Py_Main").

However, in such a case an altered DLL search path will be completely ignored, and python3?.dll will only be looked for in the standard DLL search paths. Since the sub directory with the python DLLs is not part of the standard DLL search paths, the resolution of the export fowarder(s) fails.

Trying to alter the DLL search path has no effect. Loading python3.dll with LoadLibraryEx and the flag LOAD_WITH_ALTERED_SEARCH_PATH does not help. Neither does calling SetDllDirectory or AddDllDirectory before loading python3.dll and/or before invoking GetProcAddress.

Note that loading python3.dll itself works fine. However, resolving one of the exports with GetProcAddress will fail if python3?.dll cannot be found in the standard DLL search path.

I don't know, if there is a way to make it work properly...

As of now the only two options out of this dilemma are either to put the python DLLs (and assorted files) in the same directory as the application exe (ugly mess) or to load python3?.dll instead of python3.dll (which sort of defeats the purpose of having python3.dll, i guess).

My suggestion for future Python 3 versions is to let python3.dll have a DLL dependency on python3?.dll. Linking python3.dll to python3?.dll (i.e., loading python3.dll will cause python3?.dll being loaded) allows using altered DLL search paths.
msg286553 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-01-31 21:56
LoadLibraryEx with LOAD_WITH_ALTERED_SEARCH_PATH worked for me in Windows 10. I tested loading 64-bit python3.dll, for both 3.5 and 3.6, and getting the address of Py_Main. The DLLs were loaded from DLL35 and DLL36 subdirectories. It worked using both a relative and a fully-qualified path, but, per the docs, to be safe you should use the fully-qualified path to python3.dll.
msg286560 - (view) Author: Elli Pirelli (Elli Pirelli) Date: 2017-01-31 22:36
It does not work on Windows 7 Pro x64.

And i am not the only one. The maintainer of the "python-embedded-launcher" also encountered the issue (see discussion here: https://github.com/zsquareplusc/python-embedded-launcher/issues/3)

Are you sure it really works on Windows 10? Perhaps you have the python3?.dll in the standard DLL search path already, making you mistakenly believe that it works properly? I would suggest to use a tool like SysInternal's Process Monitor to see and verify where the python3?.dll is really being loaded from.


The issue is not with incorrectly using LoadLibraryEx, SetDllDirectory and AddDllDirectory. The problem can be reduced to the python3.dll not having a dependcy on python3?.dll.

I did verify this with a little project, emulating how python3.dll uses export forwarders. I made a DLL (real.dll) with a dummy function Func  and another DLL (forwarder.dll) containing just an export forwarder Func=real.Func. 

As described, as long as forwarder.dll does not contain a import dependency on real.dll, loading forwarder.dll and doing GetProcAddress("Func") will fail, if real.dll is not in the standard DLL search path.

As soon as forwarder.dll includes a dependency to real.dll, the export forwarder can be resolved by GetProcAddress("Func") properly.

If i were using LoadLibraryEx/SetDllDirectory/AddDllDirectory incorrectly, attempting to load the forwarder.dll with the dependency on real.dll would just fail -- which it doesn't.

Unfortunately, the whole business with export forwarders is very poorly documented. It could perhaps be that MS did change something in Win8/2K12 or Win10 to fix this issue. Since i currently don't have access to either of these OS, i myself cannot confirm whether the problem would also occur or not on Win 10...
msg286562 - (view) Author: Elli Pirelli (Elli Pirelli) Date: 2017-01-31 22:48
I should perhaps clarify my sentence "As described, as long as forwarder.dll does not contain a import dependency on real.dll, loading forwarder.dll and doing GetProcAddress("Func") will fail, if real.dll is not in the standard DLL search path."

What i tried to say was:

As described, as long as forwarder.dll does not contain a import dependency on real.dll, loading forwarder.dll succeeds, but doing GetProcAddress("Func") will fail if real.dll is not in the standard DLL search path. If this forwarder.dll is put in one of the standard DLL search paths, GetProcAddress("Func") successfully resolves the address of the dummy function Func in real.dll.
msg286564 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-01-31 22:53
> Are you sure it really works on Windows 10? 

Neither the DLL35 nor DLL36 directory was set in PATH, nor any directory containing python3x.dll. Each of the latter two directories contained only the minimal set python3.dll, python3x.dll, and vcruntime140.dll. I ran under a debugger to confirm the delayed loading of python3x.dll when GetProcAddress is called, and that the returned address of Py_Main is within the mapped range of python3x.dll. I'll try on Windows 7 in a little while.
msg286568 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-02-01 00:07
I can confirm that LoadLibraryEx w/ LOAD_WITH_ALTERED_SEARCH_PATH unfortunately does not work in Windows 7. Actually I almost looked into this before on python-list:

https://mail.python.org/pipermail/python-list/2016-September/thread.html#714622

but I got sidetracked in the manifest discussion and forgot to fully investigate the issue in different versions of Windows. Using the py3embed assembly that I described in that thread will work in Windows 7, if you want to give that a try. In some ways it's simpler.
msg387682 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-25 17:21
This issue affects Windows 7 and earlier, so I'm changing the affected version to Python 3.8, the last version to support Windows 7. I don't have access to Windows 7 currently. If someone has access to an updated Windows 7 installation (all required and optional updates) and has the free time to check this, please confirm whether resolving forwarded functions still fails when "python3.dll" is loaded with LOAD_WITH_ALTERED_SEARCH_PATH. Also, check the newer flags added by KB2533623: LOAD_LIBRARY_SEARCH_DEFAULT_DIRS | LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR. The update to support these flags may align the behavior with Windows 8.1 and 10.
msg387684 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-25 18:07
I'm not sure we ever meant for LoadLibrary("python3.dll") to actively load the concrete python3X.dll. The APIs are the same, so you can (should) LoadLibrary the one that you want.

It's when you use static imports in extensions that it matters, but in that case it's ensured that both python3X.dll and python3.dll are already loaded.

I guess somewhere we just need to specify that python3.dll is for python3.lib, and not for LoadLibrary? I'm not even sure where the existing documentation is that we would change.
msg387687 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-25 19:59
> I'm not sure we ever meant for LoadLibrary("python3.dll") to 
> actively load the concrete python3X.dll.

IIRC, Paul Moore was doing something like this to create a script runner that loads "python3.dll", which runs as a regular application, not as a launcher for "python.exe". He didn't want to tie the executable to a particular "python3x.dll" or include the DLLs in the application directory beside the executable. I think he had the embedded distribution(s) in a subdirectory. That's solvable by defining an assembly in the subdirectory, which gets declared in the application manifest. But I think he wanted to keep it simple. So he was just manually loading "python3.dll" and calling GetProcAddress() to look up Py_Main(), which works in Windows 8+. Alternatively, for this kind of a script runner, the script itself can declare the version of Python it needs in a shebang (assuming a single architecture for the executable and Python), and the executable can then manually load the required Python DLL from a subdirectory, or other known location.
msg387689 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-25 22:39
Yeah, but in that scenario, it is just as good to LoadLibrary("python39.dll") and use it as if it was LoadLibrary("python3.dll") because the interaction model is identical.

The only reason to load python3.dll explicitly is if you are not keeping it adjacent to python39.dll, and so you need to pre-load it before the interpreter tries to import a native module. It doesn't provide any benefit for the host app other than not having to know what DLL you're loading, and most of us consider a critical security vulnerability rather than a feature ;)
msg387691 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2021-02-25 22:48
No, because I want to work with whatever version of Python the user puts there. Yes, I could search for "python3*.dll" and load the one I find, but I'm writing this in C, and I get a migraine whenever I have to write more than about 15 lines of C code these days :-)

It's not a big deal either way, though. That project turned out to be too much effort to be worth it, so it's now mostly just a proof-of-concept experiment.

> most of us consider a critical security vulnerability rather than a feature

:-) Given that my execution model is "run a user-supplied Python script with a user-supplied interpreter" I think any attacker has far easier ways of compromising things than hijacking python3.dll...
History
Date User Action Args
2022-04-11 14:58:42adminsetgithub: 73585
2021-02-25 22:48:22paul.mooresetmessages: + msg387691
2021-02-25 22:39:36steve.dowersetmessages: + msg387689
2021-02-25 19:59:34eryksunsetmessages: + msg387687
2021-02-25 19:32:22eryksunsetmessages: - msg387686
2021-02-25 18:37:02eryksunsetmessages: + msg387686
2021-02-25 18:07:06steve.dowersetmessages: + msg387684
2021-02-25 17:21:34eryksunsetmessages: + msg387682
versions: + Python 3.8, - Python 3.5, Python 3.6, Python 3.7
2017-02-01 00:07:43eryksunsetmessages: + msg286568
2017-01-31 22:53:53eryksunsetmessages: + msg286564
2017-01-31 22:48:10Elli Pirellisetmessages: + msg286562
2017-01-31 22:36:41Elli Pirellisetmessages: + msg286560
2017-01-31 21:56:41eryksunsetnosy: + eryksun
messages: + msg286553
2017-01-31 17:14:44Elli Pirellisetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2017-01-31 17:14:06Elli Pirellicreate