Issue19

Title Will setuptools on Mac Python accept fat eggs?
Priority wish Status chatting
Superseder Nosy List natefoo, pje, ronaldoussoren, zooko
Assigned To Keywords

Created on 2008-06-02.23:50:10 by zooko, last changed 2009-11-06.21:24:21 by natefoo.

Files
File name Uploaded Type Edit Remove
fat.patch natefoo, 2009-11-06.21:24:21 application/octet-stream
Messages
msg457 (view) Author: natefoo Date: 2009-11-06.21:24:21
I have dealt quite extensively with this problem, and it isn't fixed in the
2.5.4 builds on python.org as reported here, so the solution is not as simple as
upgrading.  I have the attached workaround in an app I work on but haven't
tested it everywhere yet.
msg403 (view) Author: pje Date: 2009-10-11.21:22:27
Zooko, ISTM that the most expedient way for you to move forward is to find a
10.5 user who's having the problem, and test to see whether renaming it works.

It also sounds to me as though none of us are able to champion any sort of
setuptools-based fix, as nobody has a complete enough picture of what's
happening or what's supposed to happen.  Without that, this issue is dead in the
water.  I'll leave it open a little while longer, but close it if there's no new
activity.
msg401 (view) Author: zooko Date: 2009-10-11.19:53:00
Ronald, PJE: Could you help me understand the issue for the purposes of helping
my users work around it (and also anyone else who discovers this ticket)?

As far as I understand the Apple-supplied Python that comes with 10.5 will
refuse to install "fat" eggs.  An effective work-around would be name your fat
egg "i386" and "ppc", e.g. using symlinks.  Is that right?
msg400 (view) Author: ronaldoussoren Date: 2009-10-11.19:38:57
PJE: I would have to research the exact semantics that would be needed for this, 
and don't have time to do that right now.  AFAIK /usr/bin/python on OSX 10.5 
does build eggs that should have been labeled as "fat" but does not do that due 
to the way Apple builds their binaries. 

I'm -1 on including a fix for this issue before someone has documented exactly 
what the issue is, how setuptools can work around the OS bug, and how that 
affects existing installations.  If that is not done a fix could break an 
existing installation when a users upgrades the setuptools installation for 
/usr/bin/python and could therefore break system components because Apple ships 
setuptools 0.6c9 and uses setuptools-using packages in their software.

The sw_vers removal patch is completely unrelated to this issue and is safe to 
apply.
msg399 (view) Author: pje Date: 2009-10-11.19:16:03
Setuptools doesn't do any special processing on "fat" for *any* release, so it
will only accept a "fat" egg if the local distutils says "fat" is the current
platform -- which IIUC means 10.5 users are out of luck for receiving "fat"
packages.  (Zooko: one possible workaround for you would be to make the exact
same .egg available under three names, by symlinking or copying it to a pair of
i1386 and ppc versions.)

Meanwhile, ISTM that to correctly support fat vs. ppc vs. i1386, there needs to
be explicit support for that in pkg_resources.compatible_platforms(), and maybe
get_supported_platform() as well.

If I understand correctly, get_supported_platform() is returning whatever the
distutils says, currently, but it should NOT return 'fat', even if the distutils
returns 'fat'.  (Because although it *builds* fat, it really only suppports i386
or ppc, and can also run fat.)

get_build_platform(), OTOH, should be patched to return 'fat' in the special
case situation, and whatever the distutils says, otherwise.

Finally, compatible_platforms() should return false if the machine types differ,
and neither one is 'fat'.  (That is, a fat binary will run on i386 or ppc, and
matching machines will also work.)

Ronald, on 10.5, is the distutils actually building "fat" binaries even though
the platform string says something else?  If that's the case, then I think the
above changes are all that's needed, and I can probably implement this myself
(although a code review by someone with more understanding of the version
strings would be helpful.)

(Btw, Ronald, I'm also in the process of integrating your 'sw_vers'-removal
patch, so it should go out in the same release as this, if not sooner.)
msg397 (view) Author: zooko Date: 2009-10-11.18:18:56
I'm not going to work on a patch for this anytime soon.  I guess I need to
recommend that my Mac OS 10.5 users install Python from python.org?  Hm.  :-/ 
No wait, this issue only affects people *building* binary eggs, right?  My Mac
OS 10.5 users can install and use binary eggs just fine even if they are using
the Apple-supplied Python.  Is that right?
msg396 (view) Author: ronaldoussoren Date: 2009-10-11.17:57:36
Note that setuptools doesn't need explicit support for fat binaries, that's 
already in distutils itself. This would be a workaround for a bug in Apple's 
installation of Python on OSX 10.5.

That said, I won't work on a patch because the issue doesn't exist in 10.6 or 
any of the installers on the python.org website.
msg394 (view) Author: pje Date: 2009-10-11.14:48:38
Reclassifying as "wish", since this is 1) a feature (setuptools never had
explicit support for "fat" binaries), and 2) a distutils workaround and perhaps
even a significant change to build processes.

This also isn't going anywhere without someone providing a tested patch that,
AFAICT, needs to be verified against multiple Python and OS/X versions, as well
as including more tests for setuptools' platform utilities to verify its operation.

Zooko, if you want to keep this one open, you will need to drive the
implementation or at least detailed specification of a fix.  Just describing the
problem isn't going to help, since I'm not on a Mac, and certainly not on 10.5.
msg393 (view) Author: ronaldoussoren Date: 2009-10-11.10:55:08
PJE: this issue is still relevant, the issue affects the Apple supplied Python 
on OSX 10.5 (but not 10.6).

The python distribution on OSX 10.5 is a universal binary, but that is not 
reflected in the output of distutils.util.get_platform(): that returns the 
architecture of the machine you're running on (ppc or i386) and not "fat".

The way to work around this issue is to test if the 'machine' value is in 
('i386', 'ppc') and sys.platform == 'darwin' and 
sys.prefix.startswith('/System').  If all of these are true the value of machine 
should be patched to be 'fat'.
msg387 (view) Author: pje Date: 2009-10-10.23:41:56
Ronald, could you perhaps comment on this issue?  Is it worth keeping an issue
open for this, or is it a dead parrot?  Thanks!
msg87 (view) Author: zooko Date: 2008-08-19.15:36:59
Here is the link to the distutils-sig message where Ronald Oussoren wrote that
comment:

http://mail.python.org/pipermail/distutils-sig/2008-June/009540.html
msg41 (view) Author: zooko Date: 2008-06-08.04:00:58
On Jun 3, 2008, at 1:12 AM, Ronald Oussoren wrote:

> That's an issue in Apple's build of Python and not something we can  
> fix.

So what is the status of this issue now, from the perspective of a user?

Do we have to avoid building binary distributions on certain versions  
of Mac OS X, or certain CPU architectures?  Do we need to build  
separate "PPC" and "x86" binary distributions?

Or is everything going to work fine and, as a user of Python and  
distutils and setuptools, I can ignore this issue?

Regards,

Zooko
msg34 (view) Author: zooko Date: 2008-06-02.23:51:19
Here is the ticket to track this issue as it affects the allmydata.org Tahoe
project:

http://allmydata.org/trac/tahoe/ticket/212 # easy_install of a fat binary .egg
which was built on Mac OS 10.4 fails on Mac OS 10.5
msg33 (view) Author: zooko Date: 2008-06-02.23:50:10
There was a time, last November, when, if I understand it correctly, setuptools
on the standard Python that comes with Mac OS X wouldn't accept fat eggs for Mac
OS X 10.4 or higher.

Here is the pythonmac-sig discussion:

http://www.nabble.com/does-pkg_resources-think-that-%22macosx-10.3%22-is-incompatible-with-10.5--to13865060.html#a13865060

Ronald Oussoren has committed the fix for the comparison of Mac OS X version
numbers from '<' to '>=', so as of Python 2.5.2 that problem is fixed.  However,
he mentioned other problems which I didn't understand:

"""the config/Makefile in  
Apple's Python.framework isn't configured for building universal  
binaries.

And to make matters even worse: I'm pretty sure that setuptools used  
to know that 'fat' builds are compatible with 'i386' and 'ppc'  
architectures (at least on OSX), but that code no longer seems to be  
there.
"""

What is the status of this issue now?
History
Date User Action Args
2009-11-06 21:24:21natefoosetfiles: + fat.patch
nosy: + natefoo
messages: + msg457
2009-10-11 21:22:27pjesetmessages: + msg403
2009-10-11 19:53:00zookosetmessages: + msg401
2009-10-11 19:38:57ronaldoussorensetmessages: + msg400
2009-10-11 19:16:04pjesetmessages: + msg399
2009-10-11 18:18:56zookosetmessages: + msg397
2009-10-11 17:57:37ronaldoussorensetmessages: + msg396
2009-10-11 14:48:39pjesetpriority: bug -> wish
messages: + msg394
2009-10-11 10:55:08ronaldoussorensetmessages: + msg393
2009-10-10 23:41:56pjesetnosy: + pje, ronaldoussoren
messages: + msg387
2008-08-19 15:37:00zookosetmessages: + msg87
2008-06-08 04:00:58zookosetmessages: + msg41
2008-06-02 23:51:19zookosetstatus: unread -> chatting
messages: + msg34
2008-06-02 23:50:11zookocreate