site.py ha two limitations that make difficult to use pth files on my
linux installation (gobolinux):
- it does not process pth files that are located in directories that are
already present in os.path at the time the main method is invoked
- it does not process directory recursively

Please find attached a patch that solves both. Basically known_paths
became a set representing the directories that have been processed.
Duplicates in os.path are avoided by looking directly into it.

While I am not sure how I feel about searching every entry on sys.path,
I know I don't like the idea of recursing the directory tree.  pth files
are meant to only be at the top-level of a directory that packages and
modules are checked for.

What I meant for recursively was not to scan the full directory tree,
but just to scan entries that have been added to the os.path. I think
that this was the original intention as
http://docs.python.org/inst/search-path.html states:
"Any directories added to the search path will be scanned in turn for
.pth files". 
Also http://docs.python.org/lib/module-site.html states:
"For each of the distinct head-tail combinations, it sees if it refers
to an existing directory, and if so, adds it to sys.path and also
inspects the newly added path for configuration files."

For what you meant by recursion, that makes more sense.

But as for whether this is correct or not, read the next paragraph in
the site docs
(http://docs.python.org/dev/library/site.html#module-site).  It says "A
path configuration file is a file whose name has the form package.pth
and exists in one of the four directories mentioned above".  So the
"newly added path" is the four paths mentioned in that paragraph (e.g.,
the ones that involve sys.prefix and sys.exec_prefix.

So the docs do not suggest that any recursive check is done, nor does it
need to worry about pre-existing directories.  It just needs to check
site-packages locations and site-python.

If site.py is not checking those directories, it is a bug, otherwise I
think the current behaviour matches the documentation.

This make sense. I hope I'm not annoying you.

But what I did was to read
(http://docs.python.org/inst/search-path.html) before reading
(http://docs.python.org/dev/library/site.html#module-site) as the latter
is referenced by the first via a html link.

The part of my interest of the first document says:
"The expected convention for locally installed packages is to put them
in the .../site-packages/ directory, but you may want to install Python
modules into some arbitrary directory. For example, your site may have a
convention of keeping all software related to the web server under /www.
Add-on Python modules might then belong in /www/python, and in order to
import them, this directory must be added to sys.path. There are several
different ways to add the directory.

The most convenient way is to add a path configuration file to a
directory that's already on Python's path, usually to the
.../site-packages/ directory. Path configuration files have an extension
of .pth, and each line must contain a single path that will be appended
to sys.path. (Because the new paths are appended to sys.path, modules in
the added directories will not override standard modules. This means you
can't use this mechanism for installing fixed versions of standard
modules.)"

So the gobolinux maintainer did: it added a file into this dir and it is
actually parsed and some dirs are appended to sys.path.

And after that the document says what already quoted:
"Paths can be absolute or relative, in which case they're relative to
the directory containing the .pth file. Any directories added to the
search path will be scanned in turn for .pth files. See site module
documentation for more information."

So why repeat here that some other directories will be scanned for pth
files, while you describe the accepted format for paths in .pth files?
If the only scanned directories are the ones already cited (a directory
that's already on Python's path), IMO there is no need to specify
further, unless you mean that the "single path appended to sys.path"
that you are describing "will be scanned *in turn* for .pth file".

On the other hand the second document doesn't say explicitly that the
*only* scanned directories are the ones derived from sys.prefix and
sys.exec_prefix. Indeed on darwin (Mac Os X) additional directories are
actually parsed.

I have made this a documentation bug as obviously there is some needed
clarification in how .pth files are handled.

If you have any suggested wording, Giambattista, then please feel free
to submit patches against the 2.6 docs (easier to work with since they
are in reStructuredText format).  Otherwise I will try to get to this
when I can (which might be a while =).

Please find attached a patch for
http://docs.python.org/dev/install/index.html.
I think removing a statement is enough to make things clear.
About darwin additional directories I'm unsure whether it need further
documentation as it seems very platform-related.
This is the comment I found in site.py
# for framework builds *only* we add the standard Apple
# locations. Currently only per-user, but /Library and
# /Network/Library could be added too


Eventually I used this command in pth file to extend the scanned
directories:
"import site;
site.addsitedir('/System/Links/Libraries/python2.5/site-packages/', set())"

I think this is more maintainable and clear than changing site.py, but I
don't think it will be forward compatible.

Fixed in r59033 for Python 2.6.  Thanks for the help!