This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Daniel Shaulov
Recipients Daniel Shaulov, brett.cannon, eric.snow, ncoghlan
Date 2016-03-26.20:20:25
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1459023628.39.0.77878937212.issue26646@psf.upfronthosting.co.za>
In-reply-to
Content
Hi,

I was trying to build a python interpreter that has the cpp part of msgpack as a built-in module. I thought that I could just copy the msgpack folder to the Modules folder and add this 2 lines to Modules/Setup.local:

msgpack._packer msgpack/_packer.cpp
msgpack._unpacker msgpack/_unpacker.cpp

I had a few obstacles, the attached patch fixes them all.

The first - makesetup has a list of regexes to match and it has *.* after going through all known extensions to throw a "bad word" error.
I removed the check. All those things will now be assumed to be a module with a package.
Now the actual init function name is PyInit__packer and doesn't have msgpack in it, so I also added code in makesetup to use the full name as the name and only the module name for the PyInit function.


The second is that in Lib/importlib/_bootsrap.py in BuiltinImporter.find_spec there is a specific case to ignore modules that are part of a package.
Is there a reason to forbid it? I removed that check.
There were also unit tests that checked this behavior which I deleted.
I added tests that check module in package instead of them.
Changing _bootsrap.py also changes Python/importlib.h (it is the frozen importlib), I added a separate patch with that change, to not clutter the main patch.

The third - the __name__ didn't have the package prefix.
Digging around I found a comment in PyModule_Create2 that says that the shared library loader stores the full name _Py_PackageContext before loading the module, so I did the same in _imp_create_builtin that is done in _PyImport_LoadDynamicModuleWithSpec to set _Py_PackageContext and the __name__ was fixed too.

(If anyone tries to do this with msgpack and wants to see that it works - you also need to copy the msgpack directory to the Lib directory, and in __init__.py, where it catches the import error and goes to the fallback, just reraise the exception instead of letting it go to the fallback)

Do note that this does not allow for built-in packages, only build-it module in package. If we want to allow built-in packages, we will need to decide on some syntax to distinguish it in the Setup files and some way to distinguish them in PyImport_Inittab (for example - an asterix before the name of the package?)

Thanks, Daniel.
History
Date User Action Args
2016-03-26 20:20:28Daniel Shaulovsetrecipients: + Daniel Shaulov, brett.cannon, ncoghlan, eric.snow
2016-03-26 20:20:28Daniel Shaulovsetmessageid: <1459023628.39.0.77878937212.issue26646@psf.upfronthosting.co.za>
2016-03-26 20:20:28Daniel Shaulovlinkissue26646 messages
2016-03-26 20:20:27Daniel Shaulovcreate