The bug tracker for setuptools 0.7 or higher is on BitBucket

 

Issue17

Title easy_install will install a package that is already there
Priority bug Status resolved
Superseder Nosy List evilrob, joss, pje, sandro, zooko
Assigned To Keywords

Created on 2008-05-31.01:13:10 by zooko, last changed 2010-04-28.20:20:26 by zooko.

Messages
msg512 (view) Author: zooko Date: 2010-04-15.04:55:42
A user has demonstrated that this is a regression in distribute: http://bitbucket.org/tarek/distribute/issue/142/easy_install-will-install-a-package-that-is-already
msg511 (view) Author: zooko Date: 2010-04-15.04:20:00
FYI this issue appears in distribute-0.6.10: http://bitbucket.org/tarek/distribute/issue/141/easy_install-pywin32-binary-succeeds-and-then-says-it-cant-find
Nobody has yet, to my knowledge, tried to reproduce this with pywin32 (following the minimal test case in msg325) and setuptools. If someone does so attempt, please update this ticket either confirming that this was fixed for pywin32 and setuptools >= 0.6c10 or informing us that this was not fixed for that case.
msg416 (view) Author: pje Date: 2009-10-19.19:15:28
setuptools 0.6c10 is released with a fix for this issue.
msg409 (view) Author: zooko Date: 2009-10-14.15:10:20
The following tickets for Tahoe-LAFS may be related.  Hopefully someone will test 
setuptools-0.6c10 with Tahoe-LAFS and report back. 
http://allmydata.org/trac/tahoe/ticket/717 
http://allmydata.org/trac/tahoe/ticket/657
msg402 (view) Author: pje Date: 2009-10-11.21:11:25
Okay, after extensive code review, it looks like my fix for issue 65 will handle
this as well, at least for the cases shown here (both the Linux and Windows
examples).  That is, it will prevent the packages from being reinstalled.

It may, however, add a spurious entry to easy-install.pth for each
system-defined directory (e.g. /var/lib/python-support/python2.6) that aren't
listed in site-dirs.  

Please note that I highly recommend upstream system packagers add the site-dirs
variable to their configuration in Python's $PREFIX/lib/distutils/distutils.cfg,
using a section like this:

[easy_install]
site_dirs = /opt/whatever/someplace/blah, /var/wtf/libraries/go/here

Listing all the places where the system installs Python libraries.  This will
not only fix this issue for existing versions of setuptools, but will also
prevent redundant entries from being added to easy-install.pth once I roll out
the base level fix.

Please note that this particular aspect (knowing which directories are
site-owned) is not something I can fix in setuptools itself; there is no a
priori way to know whether the directories on sys.path were put there by the OS
or by a user/script/whatever munging sys.path before calling easy_install
programmatically.  (And there are plenty of tools that use easy_install
programmatically.)
msg372 (view) Author: pje Date: 2009-10-10.19:53:22
Zooko, does your patch fix any of these issues?
msg352 (view) Author: sandro Date: 2009-08-27.11:05:31
I have the same problem in ubuntu jaunty both with original setuptools and
ubuntu setuptools. You can see a log of the way I show the problem here:

  http://www.e-den.it/misc/setuptools.log

You can see that if the dateutil module and python_dateutil-1.4.1.egg-info are
in /var/lib/python-support/python2.6 (sys.path[10]) tey are not detected by
setuptools (but are detected by pip) and if they are in
/usr/local/lib/python2.6/dist-packages/  (sys.path[14]) they are. 
The commands I issued whose log is in the url given above are:

apt-get install python-dateutil python-setuptools
ipython -c "import sys; sys.path"
easy_install pip
pip install python-dateutil
easy_install python-dateutil
mv /var/lib/python-support/python2.6/dateutil/  
/var/lib/python-support/python2.6/python_dateutil-1.4.1.egg-info/
/usr/local/lib/python2.6/dist-packages/
pip install python-dateutil
easy_install python-dateutil
msg325 (view) Author: zooko Date: 2009-07-12.01:41:10
I have a similar problem with pywin32.  I've described this over on
http://allmydata.org/trac/tahoe/ticket/756 (if pywin32 has been manually
installed, setuptools still doesn't detect it), but here is the minimal test case:

Put the following into a file named {{{setup.py}}} in a new empty directory:

{{{
import pkg_resources
if pkg_resources.require("pywin32"):
    print "Yes, pkg_resources says that pywin32 is already installed."
from setuptools import setup
setup(name='mindeponpywin32', install_requires=['pywin32'])
}}}

Make sure pywin32 is installed (by executing the installer for it and clickety
clickety clicking).

Then run the following command:

{{{
$ python ./setup.py develop
}}}

It will say something like:

{{{
Yes, pkg_resources says that pywin32 is already installed.
running develop
}}}
...
{{{
Processing dependencies for mindeponpywin32==0.0.0
Searching for pywin32
Reading http://pypi.python.org/simple/pywin32/
Reading http://sf.net/projects/pywin32
Reading http://sourceforge.net/project/showfiles.php?group=78018
No local packages or download links found for pywin32
error: Could not find suitable distribution for Requirement.parse('pywin32')
}}}
msg234 (view) Author: zooko Date: 2009-02-03.00:46:43
So, would this patch change it to search the sys.path for items that are
supposed to be installed?

HACK yukyuk:~/playground/setuptools/dw-0.6c9+zookopatches$ darcs diff -u -ppath
Mon Feb  2 17:41:09 MST 2009  zooko@zooko.com
  * add sys.path to the index to search for easy_install
diff -rN -u old-dw-0.6c9+zookopatches/setuptools/command/easy_install.py
new-dw-0.6c9+zookopatches/setuptools/command/easy_install.py
--- old-dw-0.6c9+zookopatches/setuptools/command/easy_install.py       
2009-02-02 17:45:45.000000000 -0700
+++ new-dw-0.6c9+zookopatches/setuptools/command/easy_install.py       
2009-02-02 17:45:45.000000000 -0700
@@ -168,7 +168,7 @@
             hosts = ['*']
         if self.package_index is None:
             self.package_index = self.create_index(
-                self.index_url, search_path = self.shadow_path, hosts=hosts,
+                self.index_url, search_path = self.shadow_path+sys.path,
hosts=hosts,
             )
         self.local_index = Environment(self.shadow_path+sys.path)
msg199 (view) Author: zooko Date: 2008-10-22.01:36:28
> (Interestingly, it *does* search all of sys.path to resolve the dependencies
of anything it actually installs.)

Wait a minute, the problems that we've been having here are when we install
allmydata-tahoe, which depends upon Nevow and Twisted, and setuptools is not
satisfied by the distribution of Nevow or of Twisted which is present in the
sys.path but not in a "site" directory.  So why do you say that setuptools does
search the entire sys.path?  That doesn't seem to be what's happening here.
msg182 (view) Author: zooko Date: 2008-09-24.18:01:33
So the particular part of this that is relevant to this ticket is that Rob was
having trouble with "./setup.py develop" attempting to install Twisted, even
though Twisted was already installed (by Apple themselves) in the
/System/Library hierarchy.  Adding
"--site-dirs=/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python"
fixed that particular problem.


I'm changing the title of this ticket to indicate that it is not debian-specific.
msg181 (view) Author: evilrob Date: 2008-09-24.17:35:15
zooko helped me sort out a build problem today that seems like it's the same (or 
at least a very similar) problem as this; but on MacOS X (specifically 10.5.5)

bascially I wound up with some weird linking problems due to twisted being 
installed in multiple locations, and one wound up importing bits of the other.

blowing away the easy_install'd version from within "/Library/..." and tweaking 
our setup.py to pass 
"--site-dirs 
/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python" 
fixed the problem.

so I'm not sure if this is a setuptools problem, or an apple not-packaging-
things-properly problem, but my guess is either way, it's going to bite other 
people.
msg126 (view) Author: zooko Date: 2008-08-22.11:12:48
See the launchpad ticket for further discussion:

https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035
msg125 (view) Author: zooko Date: 2008-08-22.11:03:46
Yep, adding --site-dirs=/var/lib/python-support/python2.5 fixes this on Hardy.
msg124 (view) Author: pje Date: 2008-08-22.03:31:24
So, try the --site-dirs.  My guess is, that will fix it, and that it should be
added to the distutils.cfg file for the distro as an interim fix.

It appears that easy_install does NOT search sys.path for a requirement that is
specified on the command line; it only searches "site" directories for such. 
Why?  I don't know.  I'll have to research the history and see if that was done
to fix a bug, before deciding whether to change that.

(Interestingly, it *does* search all of sys.path to resolve the dependencies of
anything it actually installs.)
msg123 (view) Author: zooko Date: 2008-08-22.03:00:27
So, it looks like Nevow-0.9.26.egg-info *was* on the sys.path.  See also the
launchpad ticket where it appears that simplejson has a similar problem in
Debian sid.
msg122 (view) Author: zooko Date: 2008-08-22.02:59:43
Please see also the launchpad ticket where people are discussing the hypothesis
that this was fixed in Debian sid by changing the tool used to produce Nevow
Debian packages from "pysupport" to "pycentral":

https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035
msg121 (view) Author: zooko Date: 2008-08-22.02:58:57
Hm...

>>> for p in sys.path:
...  try:
...   for f in os.listdir(p):
...    if '.egg-info' in f:
...     print p, f
...  except OSError:
...   pass
... 
/usr/lib/python2.5 wsgiref.egg-info
/usr/lib/python2.5/lib-dynload Python-2.5.2.egg-info
/usr/lib/python2.5/site-packages zope.interface-3.3.1.egg-info
/usr/lib/python2.5/site-packages setuptools.egg-info
/usr/lib/python2.5/site-packages pyutil.egg-info
/usr/lib/python2.5/site-packages command_not_found-0.1.egg-info
/usr/lib/python2.5/site-packages argparse.egg-info
/usr/lib/python2.5/site-packages python_apt-0.6.17.egg-info
/usr/lib/python2.5/site-packages pycryptopp.egg-info
/usr/lib/python2.5/site-packages zfec.egg-info
/usr/lib/python2.5/site-packages Twisted_Web-0.7.0.egg-info
/usr/lib/python2.5/site-packages Twisted-2.5.0.egg-info
/usr/lib/python2.5/site-packages pycrypto-2.0.1.egg-info
/var/lib/python-support/python2.5 GnuPGInterface-0.3.2.egg-info
/var/lib/python-support/python2.5 pyOpenSSL-0.6.egg-info
/var/lib/python-support/python2.5 Nevow-0.9.26.egg-info
/var/lib/python-support/python2.5 simplejson-1.7.3.egg-info
msg120 (view) Author: pje Date: 2008-08-22.02:48:56
You haven't answered my question.  Is there a nevow*.egg-info  or
nevow*.egg-link in any directory on sys.path (e.g., in
/var/lib/python-support/python2.5), or not?  Neither comment you just made even
remotely touches on that question.

If there is NOT such a file, then there is a distribution problem.  My guess
would be that there is some distro-specific site.py hackery, but it might be
more subtle than that.

However, if there IS such a file, then there MAY be a setuptools problem.  In
which case, you might try running easy_install
--site-dirs=/var/lib/python-support/python2.5 (or whatever path the .egg-info is
in) and see if that fixes it.
msg118 (view) Author: zooko Date: 2008-08-21.22:42:17
The long and varied history of people attempting to debug this issue in Debian,
Ubuntu, Tahoe, and so on has often included people saying "pkg_resources find
it, but then easy_install ignores it, so this much be a bug in easy_install".
msg117 (view) Author: zooko Date: 2008-08-21.22:41:23
Hm.  But why does this work:

zooko@deharo2:~$ python -c "import pkg_resources;print
pkg_resources.require('nevow');import nevow;print nevow,nevow.__version__"
[Nevow 0.9.26 (/var/lib/python-support/python2.5)]
<module 'nevow' from '/var/lib/python-support/python2.5/nevow/__init__.pyc'> 0.9.26
msg116 (view) Author: pje Date: 2008-08-21.22:13:51
Of course, you should check whether the .egg-info is symlinked into any of the
directories that *are* on sys.path.  If it is, then there might conceivably be a
setuptools problem here.  But if the .egg-info is not located directly within a
directory on sys.path (i.e., "somesyspathdir/Nevow-0.9.26.egg-info), then it's a
downstream packaging fail, not a setuptools problem.
msg115 (view) Author: pje Date: 2008-08-21.22:11:19
That would be why, then.  Can't find a package that's not on sys.path.  The
.egg-info has to be installed alongside the code, not shoved off in some data
directory.  Sounds like a downstream or packaging problem to me.
msg114 (view) Author: zooko Date: 2008-08-21.22:07:07
No:

zooko@deharo2:~$ python -c "import sys;print sys.path"
['', '/usr/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg',
'/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2',
'/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload',
'/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages',
'/var/lib/python-support/python2.5']
msg112 (view) Author: pje Date: 2008-08-21.21:26:49
Is /usr/share/python-support/python-nevow/ on sys.path?
msg110 (view) Author: zooko Date: 2008-08-21.20:37:14
It kind of looks like it isn't checking the installed distributions at all...
msg109 (view) Author: zooko Date: 2008-08-21.20:36:37
$ sudo easy_install -vv nevow 2>&1 | tee z.log.txt
Searching for nevow
Reading http://pypi.python.org/simple/nevow/
Couldn't find index page for 'nevow' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
Reading http://pypi.python.org/simple/Nevow/
Reading http://divmod.org/trac/wiki/DivmodNevow
Found link:
http://divmod.org/trac/attachment/wiki/SoftwareReleases/Nevow-0.9.31.tar.gz?format=raw
Reading http://divmod.org/projects/nevow
Found link: http://divmod.org/static/projects/nevow/Nevow-0.5.0.tar.gz
Reading http://www.divmod.org/
Best match: Nevow 0.9.31
Downloading
http://divmod.org/trac/attachment/wiki/SoftwareReleases/Nevow-0.9.31.tar.gz?format=raw

I then hit C-c to stop it.
msg107 (view) Author: pje Date: 2008-08-21.19:54:31
Run it in super-verbose mode (-vv) and attach or link to the trace, then.  Just
the one specific easy_install, showing the command-line options used.  i.e.,
please show how to reproduce it without using a makefile, and with enough
verbosity to see what distributions it's checking to meet the requirement.
msg102 (view) Author: zooko Date: 2008-08-21.19:14:03
No, there is no --always-copy config in Nevow:

http://divmod.org/trac/browser/trunk/Nevow

If you have access to an Ubuntu Hardy box, you can reproduce this issue with:

sudo apt-get install python-nevow
sudo easy_install nevow

The latter command ought to finish without fetching anything from the network or
changing the system, but instead it starts fetching Nevow in order to install it.

I just reproduced this on a Hardy box.  Here are some details about the egg-info
which comes with the python-nevow package:

zooko@deharo2:~$ dpkg --listfiles python-nevow | grep -i egg
/usr/share/python-support/python-nevow/Nevow-0.9.26.egg-info
zooko@deharo2:~$ 
zooko@deharo2:~$ cat /usr/share/python-support/python-nevow/Nevow-0.9.26.egg-info
Metadata-Version: 1.0
Name: Nevow
Version: 0.9.26
Summary: Web Application Construction Kit
Home-page: http://divmod.org/trac/wiki/DivmodNevow
Author: Divmod, Inc.
Author-email: support@divmod.org
License: MIT
Description: UNKNOWN
Platform: any
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Development Status :: 4 - Beta
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content

zooko@deharo2:~$ python -c "import pkg_resources;print
pkg_resources.require('setuptools')"
[setuptools 0.6c8 (/usr/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg)]
msg100 (view) Author: pje Date: 2008-08-21.19:02:27
Are you using --always-copy?  Note that --always-copy will not work with
system-installed packages, as it currently has no way to identify what files are
part of the distribution.  Thus, it can't copy them out.
msg67 (view) Author: zooko Date: 2008-08-01.21:33:44
This appears to be a bug in upstream setuptools, not in the Debian/Ubuntu
packaging.  I earlier thought that I had confirmed that it did *not* happen with
pristine setuptools, but just now I tried to reproduce it and it happens with
pristine setuptools.  Also a Debian developer tried to reproduce it on his
Debian-sid/lenny system and the problem did not manifest there!

I suspect that it might have something to do with
http://bugs.python.org/setuptools/issue20
msg65 (view) Author: zooko Date: 2008-08-01.18:28:25
This bug has now been added to Ubuntu -- it is present in Ubuntu Hardy and the
current prerelease of Ubuntu Ibex.

https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035
msg27 (view) Author: zooko Date: 2008-05-31.01:13:10
Tahoe ticket http://allmydata.org/trac/tahoe/ticket/229 -- describes a
problem that I found on my home system, running debian/sid, when upgrading
from the debian python-setuptools-0.6c7 to 0.6c8 . I have Nevow installed on
this box, in a way that is visible to setuptools: specifically {{{import
pkg_resources; pkg_resources.require("Nevow")}}} succeeds.

However, with 0.6c8 installed, "easy_install Nevow" nevertheless attempts to
download and install the latest version according to pypi. 0.6c7 did not: it
saw the existing Nevow and refrained from downloading a new copy.

I haven't done a lot of analysis of this yet. When Zooko and I looked at it
about a month ago, we concluded that it was something specific to the debian
0.6c8 package: installing 0.6c8 from source on a different system did not
seem to exhibit the bug. Downgrading to the debian 0.6c7 package resolved the
problem.

This has been reported to Debian:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=475440

I believe that it is a bug specific to the Debian package of setuptools 0.6c8.
It doesn't happen with setuptools 0.6c8 on other platforms, and it doesn't
happen with the Debian package of setuptools 0.6c7.
History
Date User Action Args
2010-04-28 20:20:26zookosetstatus: chatting -> resolved
2010-04-15 04:55:44zookosetmessages: + msg512
2010-04-15 04:20:01zookosetstatus: resolved -> chatting
messages: + msg511
2009-10-19 19:15:28pjesetstatus: in-progress -> resolved
messages: + msg416
2009-10-14 15:10:20zookosetmessages: + msg409
2009-10-11 21:11:26pjesetstatus: chatting -> in-progress
messages: + msg402
2009-10-10 19:53:22pjesetmessages: + msg372
2009-08-27 11:05:32sandrosetnosy: + sandro
messages: + msg352
2009-07-12 01:41:10zookosetmessages: + msg325
2009-02-03 00:46:44zookosetmessages: + msg234
2008-10-22 01:36:28zookosetmessages: + msg199
2008-09-24 18:01:33zookosetmessages: + msg182
title: easy_install (debian sid version 0.6c8) will install a package that is already there -> easy_install will install a package that is already there
2008-09-24 17:35:15evilrobsetnosy: + evilrob
messages: + msg181
2008-08-27 20:53:29josssetnosy: + joss
2008-08-22 11:12:49zookosetmessages: + msg126
2008-08-22 11:03:46zookosetmessages: + msg125
2008-08-22 03:31:25pjesetmessages: + msg124
2008-08-22 03:00:27zookosetmessages: + msg123
2008-08-22 02:59:43zookosetmessages: + msg122
2008-08-22 02:58:58zookosetmessages: + msg121
2008-08-22 02:48:56pjesetmessages: + msg120
2008-08-21 22:42:17zookosetmessages: + msg118
2008-08-21 22:41:23zookosetmessages: + msg117
2008-08-21 22:13:51pjesetmessages: + msg116
2008-08-21 22:11:19pjesetmessages: + msg115
2008-08-21 22:07:07zookosetmessages: + msg114
2008-08-21 21:26:49pjesetmessages: + msg112
2008-08-21 20:37:14zookosetmessages: + msg110
2008-08-21 20:36:37zookosetmessages: + msg109
2008-08-21 19:54:32pjesetmessages: + msg107
2008-08-21 19:14:03zookosetmessages: + msg102
2008-08-21 19:02:27pjesetnosy: + pje
messages: + msg100
2008-08-01 21:33:45zookosetmessages: + msg67
2008-08-01 18:28:25zookosetstatus: unread -> chatting
messages: + msg65
2008-05-31 01:13:10zookocreate