Consider the following directory structure:

a-\
  __init__.py
  b.py
  b-|
    __init__.py

Now, in Python (I checked 2.7.3 and 3.2.3, haven't seen the issue mentioned anywhere so I suspect it is also in later Pythons), if you import a.b, you always get the package (that is, the b folder), and the module (b.py) is silently ignored. I tested by putting the line """print("I'm a package")""" in a/b/__init__.py and """print("I'm a module")""" in a/b.py.

This becomes a real problem with tools which find modules dynamically, like test harnesses.

I'd expect that in such cases, Python should "avoid the temptation to guess", and raise an ImportError.

Thanks, Shai.

This is behavior has been true since packages were introduced, and is not going to change.  However, I agree that it could be better documented.

In Python3 something that needs to do dynamic module discovery can use importlib to be sure of using the same logic that Python itself uses.  (Well, that's completely true only for 3.3+, but it is 99+% true for 3.1 and 3.2 as well).

Thanks for the quick response.

If this isn't changing, I'd definitely want better documentation. In particular, the rationale behind this should be explained.

I submitted the bug because a co-worker unintentionally caused a whole suite of tests to be ignored.

Thanks again,
Shai.

Other than the language reference (with its updated info on imports--thanks Barry!), what other documentation would benefit from a note on this?  Somwhere in http://docs.python.org/dev/tutorial/modules.html?

It was mostly the language reference I was thinking of, since that's where I think one would naturally go to find out about unexpected behavior of the import statement, but a note in the tutorial is probably not a bad idea.  I'm not sure if this rises to the level of a FAQ or not, but perhaps it does.

Technically I suppose the importlib documentation should also mention it where appropriate if it does not already, but I wouldn't expect that to be a very discoverable location for it.

By the way, I seem to remember there being some detail about namespace packages that potentially affected this, but I don't remember if that was a proposal or something that wound up in the final implementation.  Do you?

> By the way,...

Yeah, PEP 420 (implemented in 3.3) introduced namespace packages.  The new behavior you're thinking of is where a package doesn't need a __init__.py.  So path-based lookup for modules, the order goes like this (for "import spam.eggs"):

1. look for a directory named "spam" with a __init__.py,
2. look for a file named spam.py,
3. look for a directory named "spam" (becomes an namespace package),
4. raise ImportError (used to be step 3).

Once spam gets loaded, spam.eggs gets imported...

Ah, yes.

To clarify for Shai, by the way, the reason this stuff can't change (and the reason there is a new step 3 instead of changing the whole algorithm to be something more sensible) is because of the requirement of maintaining backward compatibility.

Hi,

> the reason this stuff can't change [... is] backward compatibility.

Thanks, but this is still unclear to me. The required fix for code that would break because of the change I propose, is removal of dead code which looks misleadingly alive. 

Is the backward-compatibility requirement really that strict?

Yes.  Imagine you have a deployed application, and there happens to be an xx.py file that is masked by a package in it.  You upgrade from pythonX.Y.Z to X.Y.Z+1, and your application is suddenly throwing an error.  Yes it is easy to fix, but we prefer not to break things that way.

Now, could we throw an error in Python 3.4?  It could be discussed, at least.

Oh, sure, this was unclear of me. I thought you were talking about Python 3.4. I wasn't really expecting this to be fixed in the stable branches.

Thanks,
Shai.

I knew that a package would win over a module, but an initless package does not?  Yuck :(

It has to be that way to preserve backward compatibility, since IIUC before the PEP there was no such thing as "an initless package", it was just a directory that was ignored by import.

Deprecating pkg/__init__.py and having pkg.py coexist with pkg/ was on the table in an earlier proposal (PEP 402).  In that case pkg/__init__.py would have been tried first for backward compatbility (until eliminated in Python 4 or whenever).  PEP 420 (namespace packages) took a more conservative approach, leaving the question of pkg.py coexisting with pkg/ on the table.

I still find the idea appealing of replacing pkg/__init__.py with simply pkg.py + pkg/.  PEP 402 outlines the rationale pretty well.  Considering that PEP 420 made __init__.py-less packages legal, deprecating __init__.py isn't a huge leap.  The challenge of deciding if a directory is a package is tricky when there is not marker (like __init__.py is), but PEP 420 already tackled that for the most part.

Regardless, it would definitely require a new PEP (likely derived from 402) and some caution, especially since you could argue that people may be relying on the current precedence policy.  It would also take a little bit of work for the implementation, and a bunch of work to make sure the stdlib is happy.

I looked through both the old 2.7 import statement doc and the new 3.3 import statement doc and import system chapter (5. The import system) and could not find anything about what a Path Based Finder path entry finder does when a particular path entry has multiple candidates (a directory, a file.pyx, and __pycache__/*.pyc entries are all possible).

I also notices that there is no (longer a) description of the simple default search for default installations: sys.modules, builtin modules, and sys.path directories. Up to here, first found rules. If there were such a simplified description, it could be followed by a description of resolution of conflicts within a sys.path directory.