Issue141

If you're reporting an issue for setuptools 0.7 or higher, please use BitBucket

Title Setuptools looks for dependencies even when installed
Priority bug Status chatting
Superseder Nosy List fonnesbeck, pje
Assigned To Keywords

Created on 2012-07-07.15:27:09 by fonnesbeck, last changed 2012-07-10.20:50:17 by pje.

Files
File name Uploaded Type Edit Remove
Terminal Saved Output fonnesbeck, 2012-07-10.15:00:10 application/octet-stream
Messages
msg689 (view) Author: pje Date: 2012-07-10.20:50:17
There's definitely something weird with the OS-provided distutils platform strings, then.  In particular, if they're going to make various platform subtags like "intel" and "x86_64", they need to update the local version of compatible_platforms() to support detecting what you can actually use.

You may want to submit a bug report to Apple, and reference this issue here.  I don't have sufficient knowledge of OS X build matters to know what the right fix for this is.
msg688 (view) Author: fonnesbeck Date: 2012-07-10.19:49:26
I think this is because I explicitly build 64-bit (not universal) versions of scipy, by setting flags for -arch x86_64.
msg687 (view) Author: pje Date: 2012-07-10.19:35:00
Ok, from the output, it looks like your system's actual platform string is "macosx-10.7-intel", NOT macosx-10.7-x86_64.  The packages with the latter string, how did you install them?  Was it with a different Python or easy_install?

Basically, the only thing I can see to fix this is to either rename those eggs (and edit easy-install.pth accordingly), or allow easy_install to rebuild them (once) with the correct platform tags.  Either way, the problem should be gone after that...  as long as you don't do again whatever built/installed the eggs tagged as x86-64 versions.
msg686 (view) Author: fonnesbeck Date: 2012-07-10.15:00:10
I get a false:

In [1]: from pkg_resources import get_build_platform, get_platform, compatible_platforms

In [2]: compatible_platforms('macosx-10.7-x86_64', get_platform())
Out[2]: False

Also, here is a full output from 'sudo DISTUTILS_DEBUG=yes easy_install -vvvn mcex'. The build fails, of course, but the full output is here.
msg680 (view) Author: pje Date: 2012-07-09.20:59:14
For some reason the following didn't end up in my last note:

  >>> from pkg_resources import get_build_platform, get_platform, compatible_platforms
  >>> compatible_platforms('macosx-10.7-x86_64', get_platform())
  True

Hopefully it's visible now...
msg679 (view) Author: pje Date: 2012-07-09.20:55:24
On Mon, Jul 9, 2012 at 4:34 PM, Chris Fonnesbeck <fonnesbeck@gmail.com> wrote:
>
> On Monday, July 9, 2012 at 3:31 PM, PJ Eby wrote:
> > Is that the full output? It looks like it gets cut off long before it should have been finished.
>
> No, because I terminated it manually when it tries to fetch SciPy from PyPI. I did not want to install another SciPy.

That's why there's an 'n' in the -vvvn  -- it will keep it from
actually installing.  I need the full output, if the below isn't the
problem...

This bit is intriguing:

> ./scipy-0.10.1-py2.7-macosx-10.7-x86_64.egg
> ./statsmodels-0.5.0-py2.7-macosx-10.7-intel.egg

It looks as though you have some eggs with a "macosx-10.7-x86_64"
platform tag and some with a "macosx-10.7-intel" platform tag.  By
default, setuptools considers these different platforms and not
compatible.  Unless you have a copy of setuptools that's been patched
by Apple or someone else to treat these as compatible, easy_install
will not recognize it as supplying the requirement.

One way to test this would be to try easy_installing other packages
listed with the x86_64 tag and see if they end up downloading again,
vs. the ones with 'intel' tags continuing to work.  Another way would
be to do this at a Python prompt:

True

If that doesn't say "True", then somehow scipy, numpy, and pandas were
built with an incompatible platform tag.
msg678 (view) Author: fonnesbeck Date: 2012-07-09.20:35:04
On Monday, July 9, 2012 at 3:31 PM, PJ Eby wrote:
> Is that the full output? It looks like it gets cut off long before it should have been finished. 

No, because I terminated it manually when it tries to fetch SciPy from PyPI. I did not want to install another SciPy. 
> Also, did you ever try the "python -m easy_install" variant? 

This did not alter the behavior. 
> Last, but not least, can you send the entire contents of /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/easy-install.pth?

There is no easy-install.pth in that directory, but there is one in /Library/Python/2.7/site-packages. The contents are as follows:

import sys; sys.__plen = len(sys.path)
./pip-1.1.post1-py2.7.egg
./matplotlib-1.2.x-py2.7-macosx-10.7-intel.egg
./readline-6.2.2-py2.7-macosx-10.7-intel.egg
./setuptools-0.6c11-py2.7.egg
./basemap-1.0.3-py2.7-macosx-10.7-intel.egg
./gradient_samplers-0.1-py2.7.egg
./Numdifftools-0.3.5-py2.7.egg
./distribute-0.6.26-py2.7.egg
./pytest-2.2.3-py2.7.egg
./pytools-2011.5-py2.7.egg
./scipy-0.10.1-py2.7-macosx-10.7-x86_64.egg
./statsmodels-0.5.0-py2.7-macosx-10.7-intel.egg
/Users/fonnescj/Code/mcex
/Users/fonnescj/Code/Theano
./pyopencl-2011.2-py2.7-macosx-10.7-intel.egg
/Users/fonnescj/Code/pymc
./ipython-0.13.beta1-py2.7.egg
./numpy-1.8.0.dev_e15d0bd_20120629-py2.7-macosx-10.7-x86_64.egg
./pandas-0.7.3_20120629-py2.7-macosx-10.7-x86_64.egg
./scikit_learn-0.12_git-py2.7-macosx-10.7-intel.egg

import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new) 

Many thanks,
cf
msg677 (view) Author: pje Date: 2012-07-09.20:31:14
On Mon, Jul 9, 2012 at 1:53 PM, Chris Fonnesbeck <fonnesbeck@gmail.com>wrote:

> That worked. I've posted the output here: http://pastebin.com/p6Csfgp1
>

Is that the full output?  It looks like it gets cut off long before it
should have been finished.  Also, did you ever try the "python -m
easy_install" variant?  Last, but not least, can you send the entire
contents of
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/easy-install.pth?
msg673 (view) Author: fonnesbeck Date: 2012-07-09.17:53:33
That worked. I've posted the output here: http://pastebin.com/p6Csfgp1 

setup.py for mcex contains the following with respect to dependencies:

    setup(name = DISTNAME,
version = VERSION,
maintainer = MAINTAINER,
maintainer_email = MAINTAINER_EMAIL,
description = DESCRIPTION,
license = LICENSE,
url = URL,
long_description = LONG_DESCRIPTION,
packages =
['mcex','mcex/distributions','mcex/history','mcex/step_methods'],
classifiers =classifiers,
install_requires=['theano','numpy','scipy','numdifftools'])
msg671 (view) Author: pje Date: 2012-07-09.16:59:07
Something is very wrong: DISTUTILS_DEBUG should dump a large amount of
debug information before it even gets to looking for mcex.  Try:

sudo DISTUTILS_DEBUG=yes easy_install -vvvn mcex

Also, do you have a setup.py or requires.txt for
/Users/fonnescj/Code/mcex?  What does it say for scipy?
msg668 (view) Author: fonnesbeck Date: 2012-07-09.14:48:43
I'm using the Python 2.7.1 that ships with OSX 10.7, and keep no other versions on my system to prevent these sorts of problems. There is also only one SciPy on my system, and I'm very aware of what is and is not installed, as I use these tools so regularly.

The debug and verbose settings did not add very much to the output. For example:


$ sudo easy_install -vvv mcex
Searching for mcex
Best match: mcex 0.1
mcex 0.1 is already the active version in easy-install.pth

Using /Users/fonnescj/Code/mcex
Processing dependencies for mcex
Searching for scipy
Reading http://pypi.python.org/simple/scipy/
Found link: http://pypi.python.org/packages/source/s/scipy/scipy-0.10.0.tar.gz#md5=e357c08425fd031dce63bc4905789088
Found link: http://pypi.python.org/packages/source/s/scipy/scipy-0.10.0.zip#md5=d1a4242266739433dcfe2096b0ab4007
Found link: http://pypi.python.org/packages/source/s/scipy/scipy-0.8.0.tar.gz#md5=f0bfc6141b90e6a31555b31486602251
Found link: http://pypi.python.org/packages/source/s/scipy/scipy-0.10.1.zip#md5=4156cc1b765eb186de9518a94b6c3518
Found link: http://pypi.python.org/packages/source/s/scipy/scipy-0.10.1.tar.gz#md5=6ad976549e22e04ca93e70cf55b70a22
Found link: http://pypi.python.org/packages/source/s/scipy/scipy-0.9.0.zip#md5=a37933c9e3c4fdf8d087624cd7dcb47d
Found link: http://pypi.python.org/packages/source/s/scipy/scipy-0.9.0.tar.gz#md5=ebfef6e8e82d15c875a4ee6a46d4e1cd
Reading http://www.scipy.org
Reading http://sourceforge.net/project/showfiles.php?group_id=27747&package_id=19531
Reading http://new.scipy.org/Wiki/Download
Best match: scipy 0.10.1
Downloading http://pypi.python.org/packages/source/s/scipy/scipy-0.10.1.zip#md5=4156cc1b765eb186de9518a94b6c3518
^Cinterrupted

In particular, it does not say anything about what it finds on my system. I confirmed that DISTUTILS_DEBUG=yes is in my env.
msg667 (view) Author: pje Date: 2012-07-09.13:18:33
So, are you still getting the behavior now?  If so, run a trace using "DISTUTILS_DEBUG=yes" in your environment.

Also, on OS X, I understand that there can be certain challenges using easy_install depending on whether you're using stock OS X Python + setuptools, or the MacPorts version, or some combination.  If I understand correctly, it's entirely possible that you've been using one version of SciPy, but it's not on the sys.path of the easy_install you're using.  Using the DISTUTILS_DEBUG setting, plus the "-vvv" option to easy_install, should give us more insight into exactly what files are being found where (or not) and what is going on with that.

Also, you might compare the effect of "python -m easy_install" vs. running "easy_install" directly.  The "python -m" version has the advantage of using whatever "python" you're running, whereas the "easy_install" command you're running could potentially be linked to a different Python instance or sys.path setup.
msg666 (view) Author: fonnesbeck Date: 2012-07-08.17:27:06
I am sure it is installed because I use scipy almost every day.

easy-install.pth contains the following entry:

./scipy-0.10.1-py2.7-macosx-10.7-x86_64.egg
msg665 (view) Author: pje Date: 2012-07-07.22:18:02
How do you know scipy is installed?  Is it listed in the same easy_install.pth file as everything else on your system?
msg664 (view) Author: fonnesbeck Date: 2012-07-07.15:27:09
When installing a package using setuptools, the installer looks for and tries to install dependencies even when they are already satisfied. For example, a package that requires scipy (no version requirement specified) looks for a version online even though I have a very current version already installed:


Processing dependencies for mcex==0.1
Searching for scipy
Reading http://pypi.python.org/simple/scipy/
Reading http://www.scipy.org
Reading http://sourceforge.net/project/showfiles.php?group_id=27747&package_id=19531
Reading http://new.scipy.org/Wiki/Download
Best match: scipy 0.10.1
History
Date User Action Args
2012-07-10 20:50:17pjesetmessages: + msg689
2012-07-10 19:49:26fonnesbecksetmessages: + msg688
2012-07-10 19:35:00pjesetmessages: + msg687
2012-07-10 15:00:11fonnesbecksetfiles: + Terminal Saved Output
messages: + msg686
2012-07-09 20:59:14pjesetmessages: + msg680
2012-07-09 20:55:24pjesetmessages: + msg679
2012-07-09 20:38:46pjesetfiles: - unnamed
2012-07-09 20:38:41pjesetfiles: - unnamed
2012-07-09 20:35:04fonnesbecksetmessages: + msg678
2012-07-09 20:31:14pjesetfiles: + unnamed
messages: + msg677
2012-07-09 17:53:34fonnesbecksetmessages: + msg673
2012-07-09 16:59:07pjesetfiles: + unnamed
messages: + msg671
2012-07-09 14:48:43fonnesbecksetmessages: + msg668
2012-07-09 13:18:33pjesetmessages: + msg667
2012-07-08 17:27:06fonnesbecksetmessages: + msg666
2012-07-07 22:18:03pjesetstatus: unread -> chatting
nosy: + pje
messages: + msg665
2012-07-07 15:27:09fonnesbeckcreate