classification
Title: import silently prefers package over module when both available
Type: behavior Stage:
Components: Documentation Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Arfrever, docs@python, eric.araujo, eric.snow, r.david.murray, shai, terry.reedy
Priority: normal Keywords:

Created on 2013-02-02 22:41 by shai, last changed 2013-02-09 00:46 by terry.reedy.

Messages (14)
msg181225 - (view) Author: Shai Berger (shai) Date: 2013-02-02 22:41
Consider the following directory structure:

a-\
  __init__.py
  b.py
  b-|
    __init__.py

Now, in Python (I checked 2.7.3 and 3.2.3, haven't seen the issue mentioned anywhere so I suspect it is also in later Pythons), if you import a.b, you always get the package (that is, the b folder), and the module (b.py) is silently ignored. I tested by putting the line """print("I'm a package")""" in a/b/__init__.py and """print("I'm a module")""" in a/b.py.

This becomes a real problem with tools which find modules dynamically, like test harnesses.

I'd expect that in such cases, Python should "avoid the temptation to guess", and raise an ImportError.

Thanks, Shai.
msg181232 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-02-02 23:39
This is behavior has been true since packages were introduced, and is not going to change.  However, I agree that it could be better documented.

In Python3 something that needs to do dynamic module discovery can use importlib to be sure of using the same logic that Python itself uses.  (Well, that's completely true only for 3.3+, but it is 99+% true for 3.1 and 3.2 as well).
msg181233 - (view) Author: Shai Berger (shai) Date: 2013-02-02 23:51
Thanks for the quick response.

If this isn't changing, I'd definitely want better documentation. In particular, the rationale behind this should be explained.

I submitted the bug because a co-worker unintentionally caused a whole suite of tests to be ignored.

Thanks again,
Shai.
msg181234 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-02-03 00:02
Other than the language reference (with its updated info on imports--thanks Barry!), what other documentation would benefit from a note on this?  Somwhere in http://docs.python.org/dev/tutorial/modules.html?
msg181236 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-02-03 01:31
It was mostly the language reference I was thinking of, since that's where I think one would naturally go to find out about unexpected behavior of the import statement, but a note in the tutorial is probably not a bad idea.  I'm not sure if this rises to the level of a FAQ or not, but perhaps it does.

Technically I suppose the importlib documentation should also mention it where appropriate if it does not already, but I wouldn't expect that to be a very discoverable location for it.

By the way, I seem to remember there being some detail about namespace packages that potentially affected this, but I don't remember if that was a proposal or something that wound up in the final implementation.  Do you?
msg181242 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-02-03 05:07
> By the way,...

Yeah, PEP 420 (implemented in 3.3) introduced namespace packages.  The new behavior you're thinking of is where a package doesn't need a __init__.py.  So path-based lookup for modules, the order goes like this (for "import spam.eggs"):

1. look for a directory named "spam" with a __init__.py,
2. look for a file named spam.py,
3. look for a directory named "spam" (becomes an namespace package),
4. raise ImportError (used to be step 3).

Once spam gets loaded, spam.eggs gets imported...
msg181268 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-02-03 14:04
Ah, yes.

To clarify for Shai, by the way, the reason this stuff can't change (and the reason there is a new step 3 instead of changing the whole algorithm to be something more sensible) is because of the requirement of maintaining backward compatibility.
msg181269 - (view) Author: Shai Berger (shai) Date: 2013-02-03 14:16
Hi,

> the reason this stuff can't change [... is] backward compatibility.

Thanks, but this is still unclear to me. The required fix for code that would break because of the change I propose, is removal of dead code which looks misleadingly alive. 

Is the backward-compatibility requirement really that strict?
msg181272 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-02-03 14:48
Yes.  Imagine you have a deployed application, and there happens to be an xx.py file that is masked by a package in it.  You upgrade from pythonX.Y.Z to X.Y.Z+1, and your application is suddenly throwing an error.  Yes it is easy to fix, but we prefer not to break things that way.

Now, could we throw an error in Python 3.4?  It could be discussed, at least.
msg181273 - (view) Author: Shai Berger (shai) Date: 2013-02-03 14:54
Oh, sure, this was unclear of me. I thought you were talking about Python 3.4. I wasn't really expecting this to be fixed in the stable branches.

Thanks,
Shai.
msg181687 - (view) Author: √Čric Araujo (eric.araujo) * (Python committer) Date: 2013-02-08 17:40
I knew that a package would win over a module, but an initless package does not?  Yuck :(
msg181688 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-02-08 17:44
It has to be that way to preserve backward compatibility, since IIUC before the PEP there was no such thing as "an initless package", it was just a directory that was ignored by import.
msg181692 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-02-08 18:54
Deprecating pkg/__init__.py and having pkg.py coexist with pkg/ was on the table in an earlier proposal (PEP 402).  In that case pkg/__init__.py would have been tried first for backward compatbility (until eliminated in Python 4 or whenever).  PEP 420 (namespace packages) took a more conservative approach, leaving the question of pkg.py coexisting with pkg/ on the table.

I still find the idea appealing of replacing pkg/__init__.py with simply pkg.py + pkg/.  PEP 402 outlines the rationale pretty well.  Considering that PEP 420 made __init__.py-less packages legal, deprecating __init__.py isn't a huge leap.  The challenge of deciding if a directory is a package is tricky when there is not marker (like __init__.py is), but PEP 420 already tackled that for the most part.

Regardless, it would definitely require a new PEP (likely derived from 402) and some caution, especially since you could argue that people may be relying on the current precedence policy.  It would also take a little bit of work for the implementation, and a bunch of work to make sure the stdlib is happy.
msg181710 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-02-09 00:46
I looked through both the old 2.7 import statement doc and the new 3.3 import statement doc and import system chapter (5. The import system) and could not find anything about what a Path Based Finder path entry finder does when a particular path entry has multiple candidates (a directory, a file.pyx, and __pycache__/*.pyc entries are all possible).

I also notices that there is no (longer a) description of the simple default search for default installations: sys.modules, builtin modules, and sys.path directories. Up to here, first found rules. If there were such a simplified description, it could be followed by a description of resolution of conflicts within a sys.path directory.
History
Date User Action Args
2013-02-09 00:46:57terry.reedysetnosy: + terry.reedy

messages: + msg181710
versions: + Python 3.3, Python 3.4, - Python 3.2
2013-02-08 18:54:55eric.snowsetmessages: + msg181692
2013-02-08 17:44:16r.david.murraysetmessages: + msg181688
2013-02-08 17:40:34eric.araujosetnosy: + eric.araujo
messages: + msg181687
2013-02-03 14:54:43shaisetmessages: + msg181273
2013-02-03 14:48:56r.david.murraysetmessages: + msg181272
2013-02-03 14:16:08shaisetmessages: + msg181269
2013-02-03 14:04:40r.david.murraysetnosy: + docs@python
messages: + msg181268

assignee: docs@python
components: + Documentation, - Interpreter Core
2013-02-03 05:07:23eric.snowsetmessages: + msg181242
2013-02-03 04:29:03Arfreversetnosy: + Arfrever
2013-02-03 01:31:14r.david.murraysetmessages: + msg181236
2013-02-03 00:02:49eric.snowsetnosy: + eric.snow
messages: + msg181234
2013-02-02 23:51:05shaisetmessages: + msg181233
2013-02-02 23:39:33r.david.murraysetnosy: + r.david.murray
messages: + msg181232
2013-02-02 22:41:08shaicreate