classification
Title: Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders
Type: enhancement Stage: resolved
Components: C API Versions: Python 3.10
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: indygreg, lukasz.langa, pablogsal, petr.viktorin, serhiy.storchaka, vstinner
Priority: Keywords: patch

Created on 2021-09-28 01:01 by indygreg, last changed 2021-10-04 19:18 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 28594 merged serhiy.storchaka, 2021-09-28 09:04
Messages (6)
msg402754 - (view) Author: Gregory Szorc (indygreg) * Date: 2021-09-28 01:01
https://bugs.python.org/issue41994 / commit 4db8988420e0a122d617df741381b0c385af032c removed _PyImport_FindExtensionObject() from the C API and it is no longer available in CPython 3.10 after alpha 5.

At least py2exe and PyOxidizer rely on this API for implementing a custom module loader for extension modules. Essentially, both want to implement a custom Loader.create_module() so they can use a custom mechanism for injecting a shared library into the process. While the details shouldn't be important beyond "they can't use imp.create_dynamic()," both use a similar technique that hooks LoadLibrary() on Windows to enable them to load a DLL from memory (as opposed to a file).

While I don't have the extension module loading mechanism fully paged in to my head at the moment, I believe the reason that _PyImport_FindExtensionObject() (now effectively import_find_extension()) is important for py2exe and PyOxidizer is because they need to support at most once initialization, including honoring the multi-phase initialization semantics. Since the state of extension modules is stored in `static PyObject *extensions` and the thread state (which are opaque to the C API), the loss of _PyImport_FindExtensionObject() means there is no way to check for and use an existing extension module module object from the bowels of the importer machinery. And I think this means it isn't possible to implement well-behaved alternate extension module loaders any more.

I'm aware the deleted API was "private" and probably shouldn't have been used in the first place. And what py2exe and PyOxidizer are doing here is rather unorthodox.

In my mind the simplest path forward is restoring _PyImport_FindExtensionObject(). But a properly designed public API is probably a better solution.

Until 3.10 makes equivalent functionality available or another workaround is supported, PyOxidizer won't be able to support loading extension modules from memory on Windows on Python 3.10. This is unfortunate. But probably not a deal breaker and I can probably go several months shipping PyOxidizer with this regression without too many complaints.
msg402761 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-28 09:15
Gregory, could you please build Python 3.10 with PR 28594 applied and test whether py2exe and PyOxidizer work well with it? The restored function no longer used in the CPython code, so it is not tested now.
msg402776 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-28 12:39
> Until 3.10 makes equivalent functionality available or another workaround is supported,

I don't understand which long term solution do you propose.

> a properly designed public API is probably a better solution

Can you propose a public API?
msg402777 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-28 12:48
> I don't understand which long term solution do you propose.


A kind reminder that for 3.10 we cannot add new APIs so this proposal still *makes sense* for the short term (3.10).
msg402807 - (view) Author: Ɓukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-28 21:12
New changeset ec4e2ec241acb3bf4e04a351c06c0b05a1e4b7d2 by Serhiy Storchaka in branch '3.10':
[3.10] bpo-45307: Restore private C API function _PyImport_FindExtensionObject() (GH-28594)
https://github.com/python/cpython/commit/ec4e2ec241acb3bf4e04a351c06c0b05a1e4b7d2
msg403168 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-10-04 19:18
New changeset 2ca4ab8031107f9e474e2ce26561ab9ad51faf9c by Pablo Galindo (Serhiy Storchaka) in branch '3.10':
[3.10] bpo-45307: Restore private C API function _PyImport_FindExtensionObject() (GH-28594)
https://github.com/python/cpython/commit/2ca4ab8031107f9e474e2ce26561ab9ad51faf9c
History
Date User Action Args
2021-10-04 19:18:43pablogsalsetmessages: + msg403168
2021-09-29 12:11:01pablogsalsetpriority: release blocker ->
status: open -> closed
stage: patch review -> resolved
2021-09-28 21:12:57lukasz.langasetnosy: + lukasz.langa
messages: + msg402807
2021-09-28 12:48:34pablogsalsetmessages: + msg402777
2021-09-28 12:39:38vstinnersetmessages: + msg402776
2021-09-28 09:15:26serhiy.storchakasetpriority: normal -> release blocker
nosy: + pablogsal
2021-09-28 09:15:01serhiy.storchakasetmessages: + msg402761
2021-09-28 09:04:37serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request26974
2021-09-28 01:01:15indygregcreate