A user has demonstrated that this is a regression in distribute: http://bitbucket.org/tarek/distribute/issue/142/easy_install-will-install-a-package-that-is-already

FYI this issue appears in distribute-0.6.10: http://bitbucket.org/tarek/distribute/issue/141/easy_install-pywin32-binary-succeeds-and-then-says-it-cant-find
Nobody has yet, to my knowledge, tried to reproduce this with pywin32 (following the minimal test case in msg325) and setuptools. If someone does so attempt, please update this ticket either confirming that this was fixed for pywin32 and setuptools >= 0.6c10 or informing us that this was not fixed for that case.

setuptools 0.6c10 is released with a fix for this issue.

The following tickets for Tahoe-LAFS may be related.  Hopefully someone will test 
setuptools-0.6c10 with Tahoe-LAFS and report back. 
http://allmydata.org/trac/tahoe/ticket/717 
http://allmydata.org/trac/tahoe/ticket/657

Okay, after extensive code review, it looks like my fix for issue 65 will handle
this as well, at least for the cases shown here (both the Linux and Windows
examples).  That is, it will prevent the packages from being reinstalled.

It may, however, add a spurious entry to easy-install.pth for each
system-defined directory (e.g. /var/lib/python-support/python2.6) that aren't
listed in site-dirs.  

Please note that I highly recommend upstream system packagers add the site-dirs
variable to their configuration in Python's $PREFIX/lib/distutils/distutils.cfg,
using a section like this:

[easy_install]
site_dirs = /opt/whatever/someplace/blah, /var/wtf/libraries/go/here

Listing all the places where the system installs Python libraries.  This will
not only fix this issue for existing versions of setuptools, but will also
prevent redundant entries from being added to easy-install.pth once I roll out
the base level fix.

Please note that this particular aspect (knowing which directories are
site-owned) is not something I can fix in setuptools itself; there is no a
priori way to know whether the directories on sys.path were put there by the OS
or by a user/script/whatever munging sys.path before calling easy_install
programmatically.  (And there are plenty of tools that use easy_install
programmatically.)

Zooko, does your patch fix any of these issues?

I have the same problem in ubuntu jaunty both with original setuptools and
ubuntu setuptools. You can see a log of the way I show the problem here:

  http://www.e-den.it/misc/setuptools.log

You can see that if the dateutil module and python_dateutil-1.4.1.egg-info are
in /var/lib/python-support/python2.6 (sys.path[10]) tey are not detected by
setuptools (but are detected by pip) and if they are in
/usr/local/lib/python2.6/dist-packages/  (sys.path[14]) they are. 
The commands I issued whose log is in the url given above are:

apt-get install python-dateutil python-setuptools
ipython -c "import sys; sys.path"
easy_install pip
pip install python-dateutil
easy_install python-dateutil
mv /var/lib/python-support/python2.6/dateutil/  
/var/lib/python-support/python2.6/python_dateutil-1.4.1.egg-info/
/usr/local/lib/python2.6/dist-packages/
pip install python-dateutil
easy_install python-dateutil

I have a similar problem with pywin32.  I've described this over on
http://allmydata.org/trac/tahoe/ticket/756 (if pywin32 has been manually
installed, setuptools still doesn't detect it), but here is the minimal test case:

Put the following into a file named {{{setup.py}}} in a new empty directory:

{{{
import pkg_resources
if pkg_resources.require("pywin32"):
    print "Yes, pkg_resources says that pywin32 is already installed."
from setuptools import setup
setup(name='mindeponpywin32', install_requires=['pywin32'])
}}}

Make sure pywin32 is installed (by executing the installer for it and clickety
clickety clicking).

Then run the following command:

{{{
$ python ./setup.py develop
}}}

It will say something like:

{{{
Yes, pkg_resources says that pywin32 is already installed.
running develop
}}}
...
{{{
Processing dependencies for mindeponpywin32==0.0.0
Searching for pywin32
Reading http://pypi.python.org/simple/pywin32/
Reading http://sf.net/projects/pywin32
Reading http://sourceforge.net/project/showfiles.php?group=78018
No local packages or download links found for pywin32
error: Could not find suitable distribution for Requirement.parse('pywin32')
}}}

So, would this patch change it to search the sys.path for items that are
supposed to be installed?

HACK yukyuk:~/playground/setuptools/dw-0.6c9+zookopatches$ darcs diff -u -ppath
Mon Feb  2 17:41:09 MST 2009  zooko@zooko.com
  * add sys.path to the index to search for easy_install
diff -rN -u old-dw-0.6c9+zookopatches/setuptools/command/easy_install.py
new-dw-0.6c9+zookopatches/setuptools/command/easy_install.py
--- old-dw-0.6c9+zookopatches/setuptools/command/easy_install.py       
2009-02-02 17:45:45.000000000 -0700
+++ new-dw-0.6c9+zookopatches/setuptools/command/easy_install.py       
2009-02-02 17:45:45.000000000 -0700
@@ -168,7 +168,7 @@
             hosts = ['*']
         if self.package_index is None:
             self.package_index = self.create_index(
-                self.index_url, search_path = self.shadow_path, hosts=hosts,
+                self.index_url, search_path = self.shadow_path+sys.path,
hosts=hosts,
             )
         self.local_index = Environment(self.shadow_path+sys.path)

> (Interestingly, it *does* search all of sys.path to resolve the dependencies
of anything it actually installs.)

Wait a minute, the problems that we've been having here are when we install
allmydata-tahoe, which depends upon Nevow and Twisted, and setuptools is not
satisfied by the distribution of Nevow or of Twisted which is present in the
sys.path but not in a "site" directory.  So why do you say that setuptools does
search the entire sys.path?  That doesn't seem to be what's happening here.

So the particular part of this that is relevant to this ticket is that Rob was
having trouble with "./setup.py develop" attempting to install Twisted, even
though Twisted was already installed (by Apple themselves) in the
/System/Library hierarchy.  Adding
"--site-dirs=/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python"
fixed that particular problem.


I'm changing the title of this ticket to indicate that it is not debian-specific.

zooko helped me sort out a build problem today that seems like it's the same (or 
at least a very similar) problem as this; but on MacOS X (specifically 10.5.5)

bascially I wound up with some weird linking problems due to twisted being 
installed in multiple locations, and one wound up importing bits of the other.

blowing away the easy_install'd version from within "/Library/..." and tweaking 
our setup.py to pass 
"--site-dirs 
/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python" 
fixed the problem.

so I'm not sure if this is a setuptools problem, or an apple not-packaging-
things-properly problem, but my guess is either way, it's going to bite other 
people.

See the launchpad ticket for further discussion:

https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035

Yep, adding --site-dirs=/var/lib/python-support/python2.5 fixes this on Hardy.

So, try the --site-dirs.  My guess is, that will fix it, and that it should be
added to the distutils.cfg file for the distro as an interim fix.

It appears that easy_install does NOT search sys.path for a requirement that is
specified on the command line; it only searches "site" directories for such. 
Why?  I don't know.  I'll have to research the history and see if that was done
to fix a bug, before deciding whether to change that.

(Interestingly, it *does* search all of sys.path to resolve the dependencies of
anything it actually installs.)

So, it looks like Nevow-0.9.26.egg-info *was* on the sys.path.  See also the
launchpad ticket where it appears that simplejson has a similar problem in
Debian sid.

Please see also the launchpad ticket where people are discussing the hypothesis
that this was fixed in Debian sid by changing the tool used to produce Nevow
Debian packages from "pysupport" to "pycentral":

https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035

Hm...

>>> for p in sys.path:
...  try:
...   for f in os.listdir(p):
...    if '.egg-info' in f:
...     print p, f
...  except OSError:
...   pass
... 
/usr/lib/python2.5 wsgiref.egg-info
/usr/lib/python2.5/lib-dynload Python-2.5.2.egg-info
/usr/lib/python2.5/site-packages zope.interface-3.3.1.egg-info
/usr/lib/python2.5/site-packages setuptools.egg-info
/usr/lib/python2.5/site-packages pyutil.egg-info
/usr/lib/python2.5/site-packages command_not_found-0.1.egg-info
/usr/lib/python2.5/site-packages argparse.egg-info
/usr/lib/python2.5/site-packages python_apt-0.6.17.egg-info
/usr/lib/python2.5/site-packages pycryptopp.egg-info
/usr/lib/python2.5/site-packages zfec.egg-info
/usr/lib/python2.5/site-packages Twisted_Web-0.7.0.egg-info
/usr/lib/python2.5/site-packages Twisted-2.5.0.egg-info
/usr/lib/python2.5/site-packages pycrypto-2.0.1.egg-info
/var/lib/python-support/python2.5 GnuPGInterface-0.3.2.egg-info
/var/lib/python-support/python2.5 pyOpenSSL-0.6.egg-info
/var/lib/python-support/python2.5 Nevow-0.9.26.egg-info
/var/lib/python-support/python2.5 simplejson-1.7.3.egg-info

You haven't answered my question.  Is there a nevow*.egg-info  or
nevow*.egg-link in any directory on sys.path (e.g., in
/var/lib/python-support/python2.5), or not?  Neither comment you just made even
remotely touches on that question.

If there is NOT such a file, then there is a distribution problem.  My guess
would be that there is some distro-specific site.py hackery, but it might be
more subtle than that.

However, if there IS such a file, then there MAY be a setuptools problem.  In
which case, you might try running easy_install
--site-dirs=/var/lib/python-support/python2.5 (or whatever path the .egg-info is
in) and see if that fixes it.

The long and varied history of people attempting to debug this issue in Debian,
Ubuntu, Tahoe, and so on has often included people saying "pkg_resources find
it, but then easy_install ignores it, so this much be a bug in easy_install".

Hm.  But why does this work:

zooko@deharo2:~$ python -c "import pkg_resources;print
pkg_resources.require('nevow');import nevow;print nevow,nevow.__version__"
[Nevow 0.9.26 (/var/lib/python-support/python2.5)]
<module 'nevow' from '/var/lib/python-support/python2.5/nevow/__init__.pyc'> 0.9.26

Of course, you should check whether the .egg-info is symlinked into any of the
directories that *are* on sys.path.  If it is, then there might conceivably be a
setuptools problem here.  But if the .egg-info is not located directly within a
directory on sys.path (i.e., "somesyspathdir/Nevow-0.9.26.egg-info), then it's a
downstream packaging fail, not a setuptools problem.

That would be why, then.  Can't find a package that's not on sys.path.  The
.egg-info has to be installed alongside the code, not shoved off in some data
directory.  Sounds like a downstream or packaging problem to me.

No:

zooko@deharo2:~$ python -c "import sys;print sys.path"
['', '/usr/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg',
'/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2',
'/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload',
'/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages',
'/var/lib/python-support/python2.5']

Is /usr/share/python-support/python-nevow/ on sys.path?

It kind of looks like it isn't checking the installed distributions at all...

$ sudo easy_install -vv nevow 2>&1 | tee z.log.txt
Searching for nevow
Reading http://pypi.python.org/simple/nevow/
Couldn't find index page for 'nevow' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
Reading http://pypi.python.org/simple/Nevow/
Reading http://divmod.org/trac/wiki/DivmodNevow
Found link:
http://divmod.org/trac/attachment/wiki/SoftwareReleases/Nevow-0.9.31.tar.gz?format=raw
Reading http://divmod.org/projects/nevow
Found link: http://divmod.org/static/projects/nevow/Nevow-0.5.0.tar.gz
Reading http://www.divmod.org/
Best match: Nevow 0.9.31
Downloading
http://divmod.org/trac/attachment/wiki/SoftwareReleases/Nevow-0.9.31.tar.gz?format=raw

I then hit C-c to stop it.

Run it in super-verbose mode (-vv) and attach or link to the trace, then.  Just
the one specific easy_install, showing the command-line options used.  i.e.,
please show how to reproduce it without using a makefile, and with enough
verbosity to see what distributions it's checking to meet the requirement.

No, there is no --always-copy config in Nevow:

http://divmod.org/trac/browser/trunk/Nevow

If you have access to an Ubuntu Hardy box, you can reproduce this issue with:

sudo apt-get install python-nevow
sudo easy_install nevow

The latter command ought to finish without fetching anything from the network or
changing the system, but instead it starts fetching Nevow in order to install it.

I just reproduced this on a Hardy box.  Here are some details about the egg-info
which comes with the python-nevow package:

zooko@deharo2:~$ dpkg --listfiles python-nevow | grep -i egg
/usr/share/python-support/python-nevow/Nevow-0.9.26.egg-info
zooko@deharo2:~$ 
zooko@deharo2:~$ cat /usr/share/python-support/python-nevow/Nevow-0.9.26.egg-info
Metadata-Version: 1.0
Name: Nevow
Version: 0.9.26
Summary: Web Application Construction Kit
Home-page: http://divmod.org/trac/wiki/DivmodNevow
Author: Divmod, Inc.
Author-email: support@divmod.org
License: MIT
Description: UNKNOWN
Platform: any
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Development Status :: 4 - Beta
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content

zooko@deharo2:~$ python -c "import pkg_resources;print
pkg_resources.require('setuptools')"
[setuptools 0.6c8 (/usr/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg)]

Are you using --always-copy?  Note that --always-copy will not work with
system-installed packages, as it currently has no way to identify what files are
part of the distribution.  Thus, it can't copy them out.

This appears to be a bug in upstream setuptools, not in the Debian/Ubuntu
packaging.  I earlier thought that I had confirmed that it did *not* happen with
pristine setuptools, but just now I tried to reproduce it and it happens with
pristine setuptools.  Also a Debian developer tried to reproduce it on his
Debian-sid/lenny system and the problem did not manifest there!

I suspect that it might have something to do with
http://bugs.python.org/setuptools/issue20

This bug has now been added to Ubuntu -- it is present in Ubuntu Hardy and the
current prerelease of Ubuntu Ibex.

https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035

Tahoe ticket http://allmydata.org/trac/tahoe/ticket/229 -- describes a
problem that I found on my home system, running debian/sid, when upgrading
from the debian python-setuptools-0.6c7 to 0.6c8 . I have Nevow installed on
this box, in a way that is visible to setuptools: specifically {{{import
pkg_resources; pkg_resources.require("Nevow")}}} succeeds.

However, with 0.6c8 installed, "easy_install Nevow" nevertheless attempts to
download and install the latest version according to pypi. 0.6c7 did not: it
saw the existing Nevow and refrained from downloading a new copy.

I haven't done a lot of analysis of this yet. When Zooko and I looked at it
about a month ago, we concluded that it was something specific to the debian
0.6c8 package: installing 0.6c8 from source on a different system did not
seem to exhibit the bug. Downgrading to the debian 0.6c7 package resolved the
problem.

This has been reported to Debian:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=475440

I believe that it is a bug specific to the Debian package of setuptools 0.6c8.
It doesn't happen with setuptools 0.6c8 on other platforms, and it doesn't
happen with the Debian package of setuptools 0.6c7.