The current site.py has three major deficiencies:

(1) All site dirs must exist on the filesystem:  Since PEP 302 (New 
Import Hooks) was adopted, this is not necessarily true.  
sys.meta_path and sys.path_hooks can have valid uses for non-
existent paths.  Even the standard zipimport hook supports in-zip-
file paths (i.e. foo.zip/bar).

(2) The directories added to sys.path by .pth files are not scanned 
for further .pth files.  If they were, you could make life much easier 
on developers and users of multi-user systems.  For example, it 
would be possible for an administrator to drop in a .pth file into the 
system-wide site-packages to allow users to have their own local 
site-packages folder.  Currently, you could try this, but it wouldn't 
work because many packages such as PIL, Numeric, and PyObjC 
take advantage of .pth files during their installation.

(3) To support the above use case, .pth files should be allowed to 
use os.path.expanduser(), so you can toss a tilde in front and do the 
right thing.  Currently, the only way to support (2) is to use an ugly 
"import" pth hook.

Attached is a patch to CVS HEAD that:
(1) Removes the os.path.exists() / os.path.isdir() restrictions
(2) Changes the .pth reader to use addsitedir() (top down) rather than 
sys.path.append()
(3) makepath() uses os.path.expanduser() after the os.path.join().

Logged In: YES 
user_id=92689

Patch looks good, the new functionality is great to have, recommend apply.

Logged In: YES 
user_id=6380

Hm. Looking for .pth files only in site-packages was
intentional; it will slow down startup considerably if you
have to do an os.listdir() of every directory on sys.path
(even worse if some are on NFS!). I'd recommend against that
part.

Logged In: YES 
user_id=139309

If you read closer, it only scans directories added by .pth files, not all 
directories on sys.path.  This is to facilitate better encapsulation, so an 
admin user can create a .pth file that says "~/lib/python2.4/site-
packages" and any user that starts Python will have their own site-
packages dir, where .pth files work, without resorting to PYTHONPATH.  
This way you can use distutils to install packages into your homedir, 
and they will actually work as expected.

PYTHONPATH is utterly disastrous if you regularly deal with several 
different Python interpreters.

Logged In: YES 
user_id=6380

It still looks like a potentially considerable slowdown
(since every directory added by a .pth file is scanned
recursively), and I'm doubting the usefulness.

Why should a site admin have the power to make Python search
in an arbitrary place in my home directory? That could cause
surprises too!

When you are regularly switching between multiple Python
interpreters, wouldn't it make more sense to have a few
aliases (or shell scripts, .bat files, etc.) that set
PYTHONPATH without exporting it? Methinks that many people
have dealt with this issue *without* feeling the need to
patch site.py.

(BTW I'm okay with the other changes, from your description.)

Logged In: YES 
user_id=139309

Well in practice, .pth files aren't used a lot (most people will see maybe 
wxPython, PIL, and Numeric), so those three os.listdir() probably aren't 
going to be significant.  If users are creating pth files themselves (which 
they probably should be doing, during development), then they'd likely 
take advantage of the new features.

As far as power for site admins goes.. why should they have power to 
install Python at all?  Lots of things an admin could do can cause 
surprises.

PYTHONPATH is only useful when dealing with the command prompt 
directly, which is probably not the case for users of Mac OS X and 
Windows.  It shouldn't have to be, anyway.  Mac OS X doesn't even 
ship with a GUI interface to edit login environment variables, and the 
normal rc scripts aren't run in the context of LaunchServices.  Also, 
consider the situation where a user of some shared web server is writing 
CGI scripts that depend on Numeric installed in their home directory.  
They'd need to know to add the following to the top of *EVERY* cgi 
script:

import site
import os
site.addsitedir(os.path.expanduser("~/lib/python" + sys.version[:3] + "/
site-packages"))

If the admin had configured a directory to put things by installing a .pth 
file that pointed there, they simply would have to tell users what flags to 
pass to distutils.

For Mac OS X, there is a workaround, in that ~/Library/Python/2.X/site-
packages is a second site dir, but this doesn't need to be a workaround 
and should be available elsewhere.

Logged In: YES 
user_id=6380

Sorry, I'm still -1, but I don't have the time to discuss it
further. All your examples seem to be Mac OS X based;
perhaps you can come up with a Mac OS X specific solution?

Logged In: YES 
user_id=139309

Did you read my whole comment?  I did come up with an example that is 
not Mac OS X based: CGIs on a web server.

Logged In: YES 
user_id=6380

Do such shared web servers really exist? It sounds like an
invitation to being hacked to me. Given that this is a last
resort (it would make more sense to petition the ISP to
install Numeric) I don't see that patching sys.path at the
top of the cgi script is so bad.

Logged In: YES 
user_id=139309

And if the listdir performance is the only killer, the "recursive site" 
protocol could be changed slightly -- it could depend on the existence of 
some file (e.g. "python-site"), so the os.listdir() turns into a false 
os.path.exists() in the general case (where it is not intended to be added 
as another site directory).

Alternatively, some token in the .pth file could say "this points to 
another site dir".  Currently that (undocumented) token actually exists, 
but it looks like this:
import site; site.addsitedir("....")

Logged In: YES 
user_id=139309

Such shared web servers do exist, and are actually quite common for 
low-cost hosting (i.e. pythonmac.org is sitting on one of these right now 
- dreamhost.com).

However, they setuid first so that they are not a security hole waiting to 
happen.  This is done with Apache using the suEXEC module <http://
httpd.apache.org/docs/suexec.html>.

Petitioning the ISP to go through the trouble of auditing and installing 
some package for all users doesn't do you much good if you're trying to 
get something up and running that day.

I'm going to check the patch for my new PEP about per user site directories.

The feature request is superseded by my PEP 370
(http://www.python.org/dev/peps/pep-0370/). As far as I understand your
proposal the PEP covers your use cases.