classification
Title: Allow built-in packages and submodules as well as top-level modules
Type: enhancement Stage:
Components: Interpreter Core Versions: Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, brett.cannon, collinwinter, encukou, haypo, jd, loewis, mlobo, ncoghlan, terry.reedy
Priority: normal Keywords: patch

Created on 2007-01-25 22:12 by mlobo, last changed 2015-10-06 03:45 by ncoghlan.

Files
File name Uploaded Description Edit
import.diff mlobo, 2007-03-06 19:48 Allow importing built-in submodules (updated) review
test_submodule.py mlobo, 2007-03-06 19:49 Test case; to be placed in Lib/test
xxsubmoduletest.c mlobo, 2007-03-06 21:30 Built-in submodule for regression test; to be placed in Modules
test_builtin_submodule.py ncoghlan, 2010-04-29 13:43 More robust tests of expected package invariants
Messages (14)
msg51802 - (view) Author: Miguel Lobo (mlobo) Date: 2007-01-25 22:12
At the moment importing built-in submodules (in my case PyQt4.QtCore and PyQt4.QtGui) does not work.  This seems to be because find_module in import.c checks only the module name (e.g. QtCore) against the built-in list, which should contain the full name (e.g. Python.QtCore) instead.

Also, the above check is performed after the code to check if the parent module is frozen, which would have already exited in that case.

By moving the is_builtin() check to earlier in find_module and using fullname instead of name, I can build PyQt4.QtCore and PyQt4.QtGui into the interpreter and import and use them with no problem whatsoever, even if their parent module (PyQt4) is frozen.

I have run the regression tests and everything seems Ok.

I am completely new to CPython development so it is quite possible that my solution is undesirable or that I have done something incorrectly.  Please let me know if that is the case.

Finally, the attached patch is for Python-2.5, but I have checked it also applies to current svn trunk with only a one-line offset.
msg51803 - (view) Author: Collin Winter (collinwinter) * (Python committer) Date: 2007-03-06 00:23
This is the kind of thing you should bring up on the python-dev list (http://mail.python.org/mailman/listinfo/python-dev). Please subscribe to the list and send in a post with your problem and proposed solution. Thanks for your effort!
msg51804 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2007-03-06 15:43
Can you come up with a test case? Put the module in Modules/Setup, then have a test case that tries importing it. The test case should check whether the module is in builtin_module_names, and, if it is, try importing it.

(Ideally, it should only make the module builtin if debug is enabled, but I can't see how that can be achieved. Perhaps some trickery in config.c could do that, and #ifdefing out the body of the module if debug is disabled).
msg51805 - (view) Author: Miguel Lobo (mlobo) Date: 2007-03-06 19:48
Ok, I have added a test case and it has actually allowed me to fix a small bug: I had tested the case where the parent of the submodule is a normal Python module and the case where it is a frozen module, but another small fix was needed to make it work if the parent is built-in itself.

Please be aware that I have only run the regression tests under Windows, as at the moment I don't have a suitable Unix box available.

Also, I have uploaded the test case file (test_submodule.py) separately, as I don't know how to convince SVN to put this file in the patch.

The new patch is against current trunk.
File Added: import.diff
msg51806 - (view) Author: Miguel Lobo (mlobo) Date: 2007-03-06 19:49
File Added: test_submodule.py
msg51807 - (view) Author: Miguel Lobo (mlobo) Date: 2007-03-06 21:30
Argh, I've just realized I forgot to upload the built-in submodule itself.  This obviously goes in Modules.
File Added: xxsubmoduletest.c
msg96588 - (view) Author: Julien Danjou (jd) Date: 2009-12-18 23:07
Is there to chance to see this *bug* fixed someday?
msg96590 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-12-18 23:22
> Is there to chance to see this *bug* fixed someday?

Please ask on python-dev. I may be willing to revive my five-for-one offer.
msg104436 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-04-28 15:05
I'll have a closer look at this tomorrow with the aim of getting it into 2.7b2.

(I'm inclined to agree with jd that this is just a bug in the existing implementation, hence the change in the issue type)
msg104506 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-04-29 13:43
OK, reassessing with brain fully engaged this time: the current patch is incorrect, and this request is more complicated than one might initially think :)

It appears that since the patch was originally tried out only on Windows, a Modules/Setup based system based system like mine can't even build a patched tree. The current incarnation of the Modules/makesetup script won't allow the use of a dotted name for a module named in Modules/Setup.

Anyway, I'm uploading a more fleshed out test case which explicitly details some of the module namespace invariants that built-in packages would need to support (and changing the issue type and title accordingly).

Even beyond these stricter tests, pkgutil and importlib would need to be checked to make sure they also support the new behaviour. 

Since I can't build the patch as it currently stands, I don't know how well it actually fairs against the stronger set of invariants. However, just looking at the patch I'm pretty confident that it doesn't include the necessary work to make sure that the parent package actually looks like a package from the interpreter's point of view.
msg104507 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-04-29 13:48
Unassigning - this idea needs a lot more specification work to be done before actually implementing it becomes a good idea.

(e.g. IronPython and Jython should be looked at to see how they handle the naming schemes in the standard libraries for their respective runtime environments)
msg113445 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-08-09 18:40
I believe this is covered by the PEP3003 3.2 change moratorium.
msg113471 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-08-09 21:02
Import is explicitly exempt from the moratorium. But since the moratorium expires starting with Python 3.3 it really doesn't matter since this change will not go into 3.2.
msg252374 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-10-06 03:45
Petr, this could be an interesting issue to take a look at from the point of view of the builtin and extension module import changes in Python 3.5.

Given the extent of the import system changes across 3.3/4/5, it's even conceivable we may have made this work somewhere along the line. In that case we'd still need a new regression test to ensure it keeps working - we didn't even have a regression test to ensure extension module imports from inside a package worked, so we definitely don't have one for builtin submodules.
History
Date User Action Args
2015-10-06 03:45:50ncoghlansetnosy: + encukou
messages: + msg252374
2013-10-17 16:35:25hayposetnosy: + haypo
2012-07-20 17:54:23brett.cannonsetversions: + Python 3.4, - Python 3.3
2012-06-12 18:48:06Arfreversetnosy: + Arfrever
2010-09-13 13:33:59ncoghlanlinkissue9716 dependencies
2010-08-09 21:02:20brett.cannonsetmessages: + msg113471
2010-08-09 18:40:11terry.reedysetnosy: + terry.reedy

messages: + msg113445
versions: + Python 3.3, - Python 3.2
2010-05-20 20:39:17skip.montanarosetnosy: - skip.montanaro
2010-04-29 13:48:31ncoghlansetassignee: ncoghlan ->
messages: + msg104507
2010-04-29 13:44:02ncoghlansetstage: needs patch ->
2010-04-29 13:43:46ncoghlansetfiles: + test_builtin_submodule.py

type: behavior -> enhancement
components: - Build
title: Allow importing built-in submodules -> Allow built-in packages and submodules as well as top-level modules
nosy: loewis, skip.montanaro, brett.cannon, collinwinter, ncoghlan, mlobo, jd
versions: - Python 2.6, Python 3.1, Python 2.7
messages: + msg104506
stage: needs patch
2010-04-28 15:05:28ncoghlansettype: enhancement -> behavior
messages: + msg104436
versions: + Python 3.1, Python 2.7, Python 3.2
2009-12-19 18:03:46skip.montanarosetnosy: + skip.montanaro
2009-12-18 23:50:49ncoghlansetassignee: ncoghlan

nosy: + ncoghlan
2009-12-18 23:22:28loewissetmessages: + msg96590
2009-12-18 23:07:50jdsetnosy: + jd
messages: + msg96588
2009-04-01 18:46:00brett.cannonsetassignee: brett.cannon -> (no value)
2009-02-11 03:11:02ajaksu2setassignee: brett.cannon
nosy: + brett.cannon
2008-01-12 05:12:41christian.heimessettype: enhancement
components: + Build
versions: + Python 2.6, - Python 2.5
2007-01-25 22:12:30mlobocreate