The overall scope of this issue is that current Python documentation gives vague, sometimes incorrect, information about the set of Python features involved in modularizing functionality.  This issue presents an obstacle to programmers making smooth transitions from a single module, to collections of modules and packages, then on to neatly organized common packages shared between projects.

The problem affects documentation of:

import and from...import statements
------------------------------------
The Language Reference is way too complicated for the mainstream case. Exactly what variants of arguments are possible, and what are their effects? What are the interactions with package features, such as whether or not modules have been explicitly imported into package __init_.py?

sys.path
---------
Typical consituents; range of alternatives for adding more dirs
                   
Module site.py
--------------
Multiple serious errors in the file docstring, relating to site-packages directories and .pth files

.pth files
-----------
Incorrectly described in site.py, and then vaguely described in other docs.
Are .pth files processed everywhere on sys.path? Can they be interative? (No to both).

package structure
-----------------
Details of package structure have evidently changed over Python versions. Current docs are unclear on points such as:
-- is __init__.py needed on subpackage directories?
-- the __all__ variable: Does it act generally to limit visibility of a module or package's attributes, or does pertain only to the 'from...import *' statement?

Details:
=========

Language Reference
-------------------
 http://docs.python.org/py3k/reference/simple_stmts.html#the-import-statement
The description of the import statement is extensive, but dauntingly complicated for the reader trying to understand the mainstream case of simply importing modules or packages that are on sys.path.  This is because the algorithm for finding modules tries numerous esoteric strategies before falling back on the plain-old-file-system method. 

(Even now that I have a good understanding of the plain-old-file variations of import, I reread this and find it hard to comprehend, and disorganized and incomplete in presenting the available variations of the statement.)

Grammar issue: the grammar shown for the import statement shows:
relative_module ::=  "."* module | "."+

... which implies that relative module could have zero leading dots. I believe an actual relative path is required to have at least one dot (PEP 328).  Evidently, in this grammar, 'relative_module' really means "relative or absolute path to module or package", so it would be quite helpful to change to:

relative_path ::=  "."+ module | "."+
from_path     ::= (relative_path | module)

etc.  (Really 'module' is not quite right here either since it's used to mean module-or-package.)
                                                                                                 
site.py:
--------
Module site.py implements the site-package related features. The docstring has multiple problems with consequences in other docs.

1. Does not mention user-specific site-package directories (implemented by addusersitepackages() )

2. Seriously misleading discussion of .pth files.  In the docstring the example shows using pth files, called "package configuration files" in their comments, to point to actual package directories bar and foo located within the site-packages directory.  This is an absolutely incorrect use of pth files:  If foo and bar are packages in .../site-packages/, they do not need to be pointed to, they are already on sys.path.  

If the package dirs ARE pointed to by foo.pth and bar.pth, the modules inside them will be exposed directly to sys.path, possibly precipitating name collisions.  Further, programmers following this example will create packages in which import statements will appear to magically perform relative imports without leading dots, leading to confusion over how the import statement is supposed to work.

It may be that this discussion is held over from a time when "package" perhaps meant "Just a Bunch of Files in a Directory"?

3. The docstring (or other docs) should make clear that .pth files are ONLY processed within site-package directories (ie: only by site.py).

4. Bug: Minor: In addsitepackages(), the library directory for Windows (the else clause) is shown as lower-case 'lib' instead of 'Lib'. This has some possibility of causing problems when running from a case-sensitive server.  In any case, if read as documentation it is misleading.

Tutorial
---------
6. Modules:  http://docs.python.org/py3k/tutorial/modules.html  

1. Discussion (6.1.2. The Module Search Path) is good as far as it goes, but it doesn't mention the site-package directories.

2. Section 6.4. Packages:  Discussion of __init__.py does describe the purpose of these files.  However, the discussion in relation to subpackages should mention that for subdirectories to be accessible they must in fact be made into subpackages.  I.e.: there is not form of import that can reach into a subdir of a package *unless* it's flagged as a subpackage using __init__.py.  I have read elsewhere that there were some versions of Python where  __init__.py was not needed on subdirs within a package.

3. Section 6.4. Packages: The discussion of __all__ should note that it works *only* in conjunction with 'from...import *', and is not a general mechanism for limiting visibility of attributes. Attributes not in the __all__ list are still accessible using other forms of import.

4. Section 6.4. Packages: "Note that when using from package import item" and following.  Draws a contrast between:

    from package import item          vs 
    import item.subitem.subsubitem

However, this muddles the roles of the arguments to import, and notably uses 'item' in two different ways.  Instead the discussion can be presented as a comparison of:

    from   PackageOrModule import ModuleOrAttribute      vs 
    import PackageOrModule

... where it can be pointed out that the PackageOrModule 'dotted path' argument follows the *same* rules in both cases (except for relative paths). The *salient* contrast is that only the 'from' form has a *second* argument which can be an attribute.

5.  Footnote: (somewhat unrelated, but...) says "the execution of a module-level function enters the function name in the module’s global symbol table."  This is surely incorrect -- it is the execution of the function's *def* that enters the function name in the symbol table.

Standard Library Reference
---------------------------
1) 27.13. site — Site-specific configuration hook  http://docs.python.org/py3k/library/site.html This is documentation for site.py, and is a page that might well come up in a search for '.pth'. 

1a) This page may simply be importing the docstring from site.py?  In any case it repeats the ommissions and errors noted above for site.py.

2) 27.1. sys — System-specific parameters and functions  http://docs.python.org/py3k/library/sys.html  Documentation for sys.path.  OK as far as it goes, but:

2a) Could helpfully point to a discussion of the typical items to be found in sys.path under normal circumstances

2b) It does point to the site.py module documentation as the authoritative info on using .pth files (which is seriously flawed as noted above).

3) 29.3. pkgutil — Package extension utility. http://docs.python.org/py3k/library/pkgutil.html  

3a) The info for pkgutil.extend_path() describes how .pkg files are similar to .pth files, and their entries should point to package directories. As I noted above, so far as I can see, package directories should be within a directory on sys,path, but should not themselves be included in the path, otherwise it breaks their capability to work properly as packages.

'Installing Python Modules' document: 
--------------------------------------
http://docs.python.org/py3k/install/index.html This may well be consulted for info on how to organize source files, though it is basically the doc for Distutils.

1. Main problem is that it seems quite out-of-date; "Windows has no concept of a user’s home directory, " and so on.

2. 'How installation works' > table.  For Windows suggests 'prefix' (default: C:\Python) as an installation directory.  This is indeed one of the possible 'site-package' directories, but surely it is deprecated in favor of C:\Python\Lib\site-packages, which this section does not mention.

3. Does not mention user-specific site-package directories.

4. 'Modifying Python's Search Path' > "The most convenient way is to add a path configuration file to a directory that’s already on Python’s path". This is incorrect.  (a) .pth files are only processed in site-package directories. (b) Clarifying an additional point of confusion -- as a consequence of (a) .pth files cannot be chained.

5. Points to docs for site.py... with it's flaws noted above.

PEP 302 New Import Hooks
------------------------
Given the vagueness elsewhere in the docs, one might go hunting for ground truth in the PEPs. One must allow for PEPs having been superceeded, but nonetheless, outdated info that is now wrong is an additional stumbling block for learners.

1. Section 'Specification part 1: The importer Protocol'.  Discussion says that in an import statement, a path (with no leading  '.') is first treated as relative. This is now incorrect (as spelled out in PEP 328.)  It would be helpful to insert a note in PEP 302 pointing out that the later revision invalidates this passage.

Thanks for the detailed review.  I’d like to work on a number of them.  I think I’ll open focused bug reports and make this one depend on them; this will let interested people see the new bugs and possibly give feedback.

> The Language Reference is way too complicated for the mainstream case.
Well, it is not a user guide, but a language reference.  Its scope is different from the library reference.

> Exactly what variants of arguments are possible, and what are their effects?
Does http://docs.python.org/dev/library/functions#__import__ help? Does http://docs.python.org/dev/library/importlib ?

> What are the interactions with package features, such as whether or not
> modules have been explicitly imported into package __init_.py?
Ah, that’s a common issue.  I’ll look into the turorial and docs for import and make a patch.

> Details of package structure have evidently changed over Python versions.
I don’t think so.

> Current docs are unclear on points such as:
> -- is __init__.py needed on subpackage directories?
Yes, it always has.  I think there was some discussion about removing them in py3k, but this was rejected.

> -- the __all__ variable: Does it act generally to limit visibility of a
> module or package's attributes, or does pertain only to the
> 'from...import *' statement?
Both.  http://docs.python.org/dev/genindex-_ leads me to http://docs.python.org/dev/reference/simple_stmts#index-44 and http://docs.python.org/dev/c-api/import#index-0  The doc is there; I however agree that the tutorial or libref could have the same info.

> Language Reference
> Grammar issue: the grammar shown for the import statement shows: [snip]
I have no inkling on that one; I’ll create another bug and ask the import experts.

> Really 'module' is not quite right here either since it's used to mean module-or-package.
A package is a type of module.
                                                                                                
> site.py:
I think I’ll focus first on my work about improving the site.rst documentation, then open another bug to synchronize the docstrings.

> Seriously misleading discussion of .pth files.  [snip]
Agreed.  I believe this section dates back to the pre-package days, where projects like PIL had a module, a .pth file and a directory with modules on sys.path.

> The docstring (or other docs) should make clear that .pth files
> are ONLY processed within site-package directories (ie: only by site.py)
I’ll add a section about the “site directories” in the docs.  They’re defined in site.py or by users calling site.addsitedir.

> In addsitepackages(), the library directory for Windows (the else clause)
> is shown as lower-case 'lib' instead of 'Lib'.
I don’t see any else clause in the 2.7 or 3.3 code.  Otherwise you’re right.


> Tutorial [snip]
All valid points.


> Standard Library Reference
> site
> This page may simply be importing the docstring from site.py?
> In any case it repeats the ommissions and errors noted above for site.py.
We don’t use docstring extraction in CPython.  Code and docs are updated alongside, or like here, get oudated in parallel.

> sys
> Could helpfully point to a discussion of the typical items to
> be found in sys.path under normal circumstances
Hm, this would be very platform-specific.  What use cases would that help?

> pkgutil
I’ll have to read again the code and PEPs to comment on that one.

> 'Installing Python Modules' document
> "Windows has no concept of a user’s home directory, " and so on.
The author probably meant that there was no $HOME environment variable, ~ shortcut and all that.  Even though Windows assigns a directory for each user, it’s not quite a home dir, but the parent of a settings dir, a documents dir, etc.  (Just like on Mac OS X, there is a UNIX-style home dir, but the tools present another directory for users’ documents, IIRC.)

> For Windows suggests 'prefix' (default: C:\Python) as an installation directory.
> This is indeed one of the possible 'site-package' directories, but surely it is
> deprecated in favor of C:\Python\Lib\site-packages, which this section does not mention.
Don’t confuse the prefix and the install dir.  The directory for Python modules is computed as prefix + Lib/site-packages.

> Does not mention user-specific site-package directories.
That’s #10745.  I documented it in http://docs.python.org/dev/packaging/commandref#install-dist but it’s not done for distutils docs yet.

> 'Modifying Python's Search Path' [snip]
> (a) .pth files are only processed in site-package directories.
Yes.  I’ll probably link to the site docs after we write out a clean section on .pth files.  There are other parts of the distutils docs that duplicate information found in docs.python.org/dev/using, for example; I’ll remove them from the packaging (distutils2) docs.

> (b) Clarifying an additional point of confusion [...]
We don’t over-explain every implication in the docs; see http://docs.python.org/dev/documenting/style.html#affirmative-tone

> PEP 302 New Import Hooks
This would be a valid point to raise on python-dev.

That was my initial feeback; I think I’ve covered all of your points.  Some time next week I’ll open the focused bugs and start on patches.

Hi Eric,  Thanks for starting to review this, and your responses are encouraging. Some comments inline below.

FWIW, along the way I accumulated my own notes on this topic, on some pages here:

grahamwideman.wikispaces.com
(Left navigation panel...)
Software development > Python > Organization for common modules

Might be of interest as feedback on the digging process I needed in order to get some clarity on these issues, and also shows my references.

>> Exactly what variants of arguments are possible, and what are their effects?
>Does http://docs.python.org/dev/library/functions#__import__ help? Does http://docs.python.org/dev/library/importlib ?

Well somewhat overkill -- because the matter of interest was args for from... and import, while the docs you mention are for more complicated underlying functions. (Interesting nonetheless.)

>> Current docs are unclear on points such as:
>> -- is __init__.py needed on subpackage directories?
>Yes, it always has.  I think there was some discussion about removing them in py3k, but this was rejected.
I came to same conclusion.. but have seen it described otherwise (in at least one book), so good to state this explicitly.

>> -- the __all__ variable: Does it act generally to limit visibility of a
>> module or package's attributes, or does pertain only to the
>> 'from...import *' statement?
> Both. 

I'm pretty sure that's not correct -- pretty sure that __all__ only specifies what's included in from...import *, and does not prevent access via from...import specific_attrib.  But I may have tested incorrectly.
                                                                                               
>> Seriously misleading discussion of .pth files.  [snip]
>Agreed. 

Cool -- I think it's well worth fixing this area for sure!

>> In addsitepackages(), the library directory for Windows (the else clause)
>> is shown as lower-case 'lib' instead of 'Lib'.
>I don’t see any else clause in the 2.7 or 3.3 code.  Otherwise you’re right.

Sorry, the lowecase 'lib' issue is in 
  getsitepackages()... 
  if sys.platform in(...) ... 
  else:...
    sitepackages.append(os.path.join(prefix, "lib", "site-packages"))

>> sys
>> Could helpfully point to a discussion of the typical items to
>> be found in sys.path under normal circumstances
>Hm, this would be very platform-specific.  What use cases would that help?

It would demystify how python already knows how to find various things under vanilla circumstances.

>> 'Installing Python Modules' document
>> "Windows has no concept of a user’s home directory, " and so on.
>The author probably meant that there was no $HOME environment variable, ~ shortcut and all that.

Fair enough, but in actuality there *is* a user-specific location (on Windows) examined by site.py, which is in %APPDATA%\Python\.

>> For Windows suggests 'prefix' (default: C:\Python) as an installation directory.
>> This is indeed one of the possible 'site-package' directories, but surely it is
>> deprecated in favor of C:\Python\Lib\site-packages, which this section does not mention.
>Don’t confuse the prefix and the install dir.  The directory for Python modules is computed as prefix + Lib/site-packages.

Currently, under "Alternate installation: Windows (the prefix scheme)", it says:
    python setup.py install --prefix="\Temp\Python"
    to install modules to the \Temp\Python directory on the current drive.
Does this really mean "install modules to \Temp\Python\Lib\site-packages"?
(And as a side point, surely installing under the Temp directory is a strange location to pick for an example?)

>That was my initial feeback; I think I’ve covered all of your points. 
Looking forward!

Thanks for getting started with such a detailed review on this Graham. We've known the documentation in this area has been flawed for a long time, but actually *fixing* seemed like such a big task that it has tended to get pushed to the bottom of our respective to-do lists.

And so as things have been tweaked, the already flawed documentation has fallen even further out of date (as it wasn't always clear where updates should be included).

Simply breaking it down into a smaller list of easier to tackle problems is a big step towards getting something done about it.

A couple of specific notes:

__all__ only affects import *, and may also affect documentation tools (e.g. pydoc will respect __all__ when deciding what to display). It has no effect on attribute retrieval from modules.

pkgutil.extend_path() is used to modify pkg.__path__ attributes, *not* sys.path. It is used to allow a single package to span multiple directories, forming the basis for "namespace packages" (see http://www.python.org/dev/peps/pep-0382/#namespace-packages-today). 

Packages are technically a kind of module, but we're somewhat inconsistent in our use of the term "module". Sometimes it means all kinds of modules (frozen, builtin, extension, package, compiled, source), sometimes it means anything-but-a-package and occasionally it specifically means pure Python modules. In theory, context should make it clear which interpretation is intended. In practice, the boundaries between the first two meanings get a little blurry.

Hi Nick: Thanks for your additional points. Comments inline:

> __all__ only affects import *, and may also affect documentation tools (e.g. pydoc will respect __all__ when deciding what to display). It has no effect on attribute retrieval from modules.

That's indeed my understanding. So the doc (6. Simple statements) which says that __all__ determines the list of "public names" is a bit of a red herring.  Attributes are accessible (ie: public) regardless of whether on the __all__ list.  Instead the __all__ list establishes the list of names imported by *, and makes those names reference-able without a module prefix. (Plus gives hints about intent to doc tools.)

> pkgutil.extend_path() is used to modify pkg.__path__ attributes, *not* sys.path. 

Understood, and perhaps my point was obtuse.  I was pointing out that the doc for extend_path discusses .pkg entries which point to package dirs, and that this, it says, is like .pth files. I claim that an entry in a .pth files should NOT point to a package dir, but rather to one level up: to a dir that *contains* package dirs. (Pointing a .pth entry directly at a package dir will break package behavior by exposing the constituent modules to sys.path.)  Hence the doc for extend_path is misleadingly suggesting a wrong idea about .pth files.

"Public name" is a term that describes a convention, not anything enforced by the interpreter. Names starting with underscores typically aren't public either (unless documented otherwise), but that has no effect on the ability to retrieve them as attributes. A glossary entry defining the term may be worthwhile (perhaps with a reference to pydoc.visiblename())

Dirs mentioned in .pkg files *should* be added to "the path". It's just that the path in question is pkg.__path__, not sys.path. That could probably be made clearer, but the docs aren't wrong as they stand.

> "Public name" is a term that describes a convention, not anything enforced by the interpreter. 

And I guess that's really the main point. In other languages Public means accessible, and Private means not so.  In Python, Public means "suggested for outside consumption", and Private means not so intended, but nonetheless accessible. If that was reiterated near the discussion of __all__ it would be most helpful.  

>  Dirs mentioned in .pkg files *should* be added to the [...] pkg.__path__, not sys.path. 
> That could probably be made clearer, but the docs aren't wrong as they stand.

Again I've not managed to draw attention to the exact point of contention. 
1. A dir added to a .pkg file evidently should be an actual package dir.  
2. A dir added to a .pth file should NOT be an actual package dir. It should be the dir at the level above.

Thus the entries in .pkg and .pth files point to different kinds of things, yet the doc I pointed to asserts they are the same in this regard.

Ah, now I get your point re .pkg vs .pth. I naturally read that as ".pth entries are to sys.path entries as .pkg entries are to pkg.__path__ entries", without any hint that those are the same *kind* of thing. However, I already know that sys.path and pkg.__path__ entries are different, and now I see how it could be misleading for anyone that wasn't as familiar with the distinction.

(This has actually been one of the historic problems with the import documentation - we *don't always know* when something is unclear, because we unconsciously fill in the missing pieces)

>>> Exactly what variants of arguments are possible, and what are their
>>> effects?
>> Does http://docs.python.org/dev/library/functions#__import__ help?
>> Does http://docs.python.org/dev/library/importlib ?
> Well somewhat overkill -- because the matter of interest was args for
> from... and import, while the docs you mention are for more
> complicated underlying functions. (Interesting nonetheless.)

importlib.import_module should not be overkill, as it was designed specifically to be more easy and correct to use than __import__.

>>> -- the __all__ variable: Does it act generally to limit visibility
>>> of a module or package's attributes, or does pertain only to the
>>> 'from...import *' statement?
>> Both. 
> I'm pretty sure that's not correct -- pretty sure that __all__ only
> specifies what's included in from...import *, and does not prevent
> access via from...import specific_attrib.  But I may have tested
> incorrectly.

Sorry, I should have clarified.  I meant that __all__ limits what “from x import *” can see (for somemodule.__all__), and what pydoc and other tools display (for someobject.__all__).  Direct reference “x._lalala” or explicit import “from x import explicit_name” don’t look at __all__.
                                                     
>>> In addsitepackages(), the library directory for Windows (the else
>>> clause) is shown as lower-case 'lib' instead of 'Lib'.

Please report that as a new bug.

>>> sys
>>> Could helpfully point to a discussion of the typical items to
>>> be found in sys.path under normal circumstances
>> Hm, this would be very platform-specific.  What use cases would that
>> help?
> It would demystify how python already knows how to find various
> things under vanilla circumstances.

I’m all for demystification, so OK.  We already have platform-specific examples in site and/or sysconfig docs, so once more wouldn’t harm.

>>> 'Installing Python Modules' document
>>> "Windows has no concept of a user’s home directory, " and so on.
>> The author probably meant that there was no $HOME environment
>> variable, ~ shortcut and all that.
> Fair enough, but in actuality there *is* a user-specific location (on
> Windows) examined by site.py, which is in %APPDATA%\Python\.

That does not map to the home concept at all.  Anyway, it does not really add value to say that one OS doesn’t have something that another OS has, let’s just say that X uses something and that Y uses soemthing else.

>> Don’t confuse the prefix and the install dir.  The directory for
>> Python modules is computed as prefix + Lib/site-packages.
> Currently, under "Alternate installation: Windows (the prefix
> scheme)", it says:
>    python setup.py install --prefix="\Temp\Python" [...]
> Does this really mean "install modules to \Temp\Python\Lib\site-packages"?

I don’t know, try it in a shell!

> (And as a side point, surely installing under the Temp directory is a
> strange location to pick for an example?)

Well, the docs gotta pick something.  A tempdir is not worse than something else to demonstrate how to use a tool.

We need to have one discussion thread for each issue, to make this huge doc bug manageable.  I will go over your first message again and open one report for each point, this week if I have time.

I noticed an inconsistency in the docs, and I think it falls in the "Language Reference" section of this ticket. In the definition of the import statement, after:

        | "from" module "import" "*"

we should add:

        | "from" relative_module "import" "*"


It is also possible to replace this line by:

        | "from" (module | relative_module) "import" "*"

FYI, I have updated the site module and distutils/packaging docs to document the PEP 370 features.  There has been no feedback, so I will backport to 2.7 and close the reports.  See linked bugs to read the changesets.