>What is the difference between a
>"platform-level" site directory and a "user-specified" site directory?

The former is defined by the executing interpreter's main 'site' 
module, which is sometimes patched on certain platforms to add extra 
site directories.

>If one
>sets up a virtualenv, directories effectively specified by the user become
>"platform-level" even though the actual "platform" lives somewhere else.

It's interpreter-defined, and so still "platform" for these purposes.

>Directories like the one I've mentioned *can* be "otherwise detected" as
>supporting .pth files; it seems like the only thing one would need would be to
>get rid of the "-E" flag to the python interpreter.  Why is that not 
>a desirable
>thing to do?

It's to keep other PYTHONPATH directories (with an 
easy_install-supplied 'site' patch) from creating a false positive 
indication that the directory works as a standalone 'site' directory.

To be honest, I can't be 100% sure this is still necessary; it's been 
a couple years now.    I'm just not clear on how the benefit of 
changing it now, outweighs the risk of breaking something else, given 
that --site-dirs was left in precisely to allow overriding the 
automatic detection.  (Earlier versions of easy_install didn't *have* 
.pth detection; you HAD to use --site-dirs if the original, unpatched 
python.org site.py wouldn't have considered it a "site" 
directory.  The detection was added so that --site-dirs wouldn't be 
needed in the most common cases.)

I still seem to be missing something.  What is the difference between a
"platform-level" site directory and a "user-specified" site directory?  If one
sets up a virtualenv, directories effectively specified by the user become
"platform-level" even though the actual "platform" lives somewhere else.

To be honest, I am looking forward to getting rid of this particular feature of
combinator and suggesting that the 'site' functionality just be achieved by
suggesting that users create a virtualenv in ~/.local, but I still think this is
interesting for automated applications that might want to easy_install things to
bootstrap themselves but have a different idea of what a "site" directory is,
and un-breaking setuptools for users who have a customized PYTHONPATH setup that
isn't lucky enough to be a released open-source project.

Directories like the one I've mentioned *can* be "otherwise detected" as
supporting .pth files; it seems like the only thing one would need would be to
get rid of the "-E" flag to the python interpreter.  Why is that not a desirable
thing to do?

>Am I missing something about its theory of operation?

The .pth test is intended for detecting platform-level "site" 
directories, not user-specified site directories, and the --site-dirs 
option is intended to force "site" treatment for directories that 
cannot otherwise be detected as supporting .pth files.

Thanks for the detailed description of the workaround.  It still sounds like
there is a bug in setuptools though, and its .pth execution capability detection
could be more thorough.  Am I missing something about its theory of operation?

At 09:45 PM 4/3/2009 +0000, Glyph Lefkowitz wrote:
> > It also looks like you may not have PYTHONPATH set at the time of 
> installation.
>
>I do.  The shell log I entered previously didn't omit any intermediary setup
>steps.  PYTHONPATH doesn't contain the installation target directory though;
>just the directory containing my bootstrapping sitecustomize.

Okay, then you need to use the --site-dirs option to tell 
easy_install that no, really, it's already a site directory and it 
doesn't need any special processing.  If the target directory isn't 
on PYTHONPATH or listed in --site-dirs, it's checked for .pth 
execution capability...  and that check occurs without an active 
PYTHONPATH.  So, either the target directory needs to be on 
PYTHONPATH, you need to use --site-dirs, or you need to use --multi-version.

> But do you process 'import' lines in the .pth files?

Yes.

  glyph@illidan:~$ echo 'import sys; print "hello .pth"' >
.local/lib/python2.5/site-packages/hello.pth
  glyph@illidan:~$ python -c ''
  hello .pth

> It also looks like you may not have PYTHONPATH set at the time of installation.

I do.  The shell log I entered previously didn't omit any intermediary setup
steps.  PYTHONPATH doesn't contain the installation target directory though;
just the directory containing my bootstrapping sitecustomize.

> We also specifically honor .pth files by making sure it's in the site dirs list,

But do you process 'import' lines in the .pth files?  If not, the .pth test code
won't run, which is why the checking would fail.  It also looks like you may not
have PYTHONPATH set at the time of installation.  Fixing either of those would
allow installation to proceed correctly.

Please also note that you can use the --site-dirs option to force easy_install
to treat the target directory as a "site" directory without testing it, and you
can also use --multi-version if you will be responsible for making sure the
egg(s) are on sys.path.  In either case, easy_install will not do any validation
on the target directory, and in the --multi-version case it will not install a
site.py or do any .pth manipulation either.

Combinator, the path-management tool that we use at Divmod (and many folks use
for managing their path for hacking on Twisted) interacts with the current
setuptools checking in a weird way.

We replace sitecustomize by setting PYTHONPATH and manipulate sys.path there. 
We also specifically honor .pth files by making sure it's in the site dirs list,
but somehow setuptools doesn't notice, and throws a bogus warning.

For example:

glyph@illidan:~/Projects/Twisted/trunk$ python setup.py install --prefix ~/.local
running install
Checking .pth file support in /home/glyph/.local/lib/python2.5/site-packages/
/usr/bin/python -E -c pass
TEST FAILED: /home/glyph/.local/lib/python2.5/site-packages/ does NOT support
.pth files
error: bad install directory or PYTHONPATH

You are attempting to install a package to a directory (blah blah blah, you've
all probably read this warning a dozen times)
glyph@illidan:~/Projects/Twisted/trunk$ python -c "import sys; print
'/home/glyph/.local/lib/python2.5/site-packages' in sys.path"
True

It would be nice if the check took sitecustomize path-customization into account.

The mkdir patch needs to happen in check_site_dir(), not in finalize_options(),
as the latter has no business writing anything to disk.

As for the warn-and-proceed, I think it would be better fixed by adding to the
existing error message, so that it explains the availability of --site-dirs and
--multi-version.  (IOW, there are already existing options to say, "I'll make it
a site dir" or "I'll be putting the egg on sys.path myself", and either one
would allow you to proceed with installation.)

See also mailing list discussion:
http://mail.python.org/pipermail/distutils-sig/2008-November/010488.html

Here is a patch to create --prefix= subtree if it doesn't already exist and
another patch to treat "that's not on the sys.path" as a warning instead of a
hard error.  This fixes one of the most common complaints that I heard when I
asked a bunch of people who refused to use setuptools why they refused to use it.