In Lib/distutils/util.py in the get_platform() function there's OS 
specific code to create a string which describes the current platform. 
This usually includes the OS + version + arch. 

FreeBSD specific code is missing from this function. Currently 
get_platform() returns a string specific to the security patch level of 
freebsd. For example:

freebsd-7.2-RELEASE-p3-i386

This results in eggs that only work on a specific patch level release of 
FreeBSD and are not portable between (for example) 7.2-RELEASE-p2 and 
7.2-RELEASE-p3.

However FreeBSD is actually binary compatible within a major version 
number. For example 7.1 and 7.2 are binary compatible.

This patch adds freebsd specific code to get_platform() after which it 
will return a string like:

freebsd-7-i386

Stef Walter wrote:
> 
> New submission from Stef Walter <stef@memberwebs.com>:
> 
> In Lib/distutils/util.py in the get_platform() function there's OS 
> specific code to create a string which describes the current platform. 
> This usually includes the OS + version + arch. 
> 
> FreeBSD specific code is missing from this function. Currently 
> get_platform() returns a string specific to the security patch level of 
> freebsd. For example:
> 
> freebsd-7.2-RELEASE-p3-i386
> 
> This results in eggs that only work on a specific patch level release of 
> FreeBSD and are not portable between (for example) 7.2-RELEASE-p2 and 
> 7.2-RELEASE-p3.
> 
> However FreeBSD is actually binary compatible within a major version 
> number. For example 7.1 and 7.2 are binary compatible.
> 
> This patch adds freebsd specific code to get_platform() after which it 
> will return a string like:
> 
> freebsd-7-i386

I think this is more a problem with easy_install than with
distutils itself.

get_platform() is meant to return a platform string, nothing
more, nothing less.

It is more meant for human consumption than for scripts to
use as basis for checking whether a particular platform
is binary compatible to what the user wants to install
a distribution archive to.

What we could do is provide a new distutils API
binary_compatible_platform() which takes the platform string
as used in the distribution archive and compares it to the
one returned by distutils on the target system.

Note that there are various level of compatibility to
consider here:

 * whether the ABI is the same (binary compatible)
 * whether the code is 32-bit and needs to run on a 64-bit
   system
 * whether the typically installed software base is
   the same
 * whether the target system provides a compatibility layer
   which can be used or not
 * whether the distribution provides code for more than
   one platform (e.g. Mac universal builds), so that it'll
   run on more than just one architecture

It is usually easier for the user to decide by looking at
the file name whether a certain package is suitable or not
than to have some script know about all the differences
between the various OS versions.

Some examples for get_platform() strings which are in fact
compatible:

darwin-8.11.0-Power_Macintosh (the Python 2.3 way)
macosx-10.4-fat (the Python 2.5+ way)
macosx-10.4-ppc (actually "fat", but not identified as such
                 due to a bug in distutils for Python 2.4)

Aside: For eGenix we have decided to simply drop the get_platform()
check in our prebuilt archives altogether - there are just too
many dimensions to the problem. Instead, we use the sys.platform
variable which provides a much more practical (though, not perfect)
solution to the problem of detecting incompatibilities early
on in the installation process.

I agree with your comments, and the solution you're proposing solves the 
problem (and several others) for the long term. 

However in the short term, could this patch be committed? Most other OS's 
(including openbsd and netbsd) have OS specific code in get_platform(). 
FreeBSD is notably missing from that function. 

Obviously this would not preclude work on a better all encompassing 
solution.

Stef Walter wrote:
> 
> Stef Walter <stef@memberwebs.com> added the comment:
> 
> I agree with your comments, and the solution you're proposing solves the 
> problem (and several others) for the long term. 
> 
> However in the short term, could this patch be committed? Most other OS's 
> (including openbsd and netbsd) have OS specific code in get_platform(). 
> FreeBSD is notably missing from that function. 

The code in get_platform() tries to gather as much information as
possible regarding a platform, rather than limiting the amount of
information.

That's why there are so many entries for the various OSes.

Please contact the maintainer of easy_install to get it fixed
to handle the specific FreeBSD case. For other systems they will
likely have to apply similar patches, e.g. for Mac OS X, Sun,
AIX, etc. - all of these include version and release information
in the get_platform() output.

Regarding a short term fix, there's no a lot we can do: the next
Python release is 2.7 and changing this for 2.6 is out of the question,
since it would break other tools that rely on the established
naming scheme.

easy_install has a different release cycle, so it's easier
to get a fix for it, or just apply one yourself.

Other OSs have special cases in get_platform() to specifically limit the 
amount of code, and make proper decisions with regard to package 
compatibility. 

Here's an example this commit for Mac OS X: http://svn.python.org/view?
view=rev&revision=67988

It was discussed here at this issue: http://bugs.python.org/issue4064

Another example is how linux has no version information at all (ie: 
linux-i586). Perhaps this is why the easy_install authors thought their 
package system worked. They only tested it on linux?

Yes I agree that obviously this cannot be changed for 2.6. But it would 
be great to get this code in for python 2.7

Anyway, this is ultimately your call, since I don't have the 50,000 foot 
view over the entire situation. 

FWIW, I've had to patch python in a very large set of client 
installations. This patch has become a routine in order to unbreak 
python wrt to platform dependent packages.

Stef Walter wrote:
> Other OSs have special cases in get_platform() to specifically limit the 
> amount of code, and make proper decisions with regard to package 
> compatibility. 
>
> Here's an example this commit for Mac OS X: http://svn.python.org/view?
> view=rev&revision=67988
> 
> It was discussed here at this issue: http://bugs.python.org/issue4064

Well, if you try to install .egg files built as universal binaries
on a system that uses a non-SDK build of Python on a PPC system,
you'll have a similar problem.

easy_install will look for '...-ppc', but the file is name '...-fat'.

But you do have a point: the '10.4' is actually an indicator for
the SDK version, not the version of the OS where the package
was built.

> Another example is how linux has no version information at all (ie: 
> linux-i586).

Right... the Linux major version doesn't change that often,
while the minor ones do change very often and don't really
give the user any useful information w/r to Python extensions.

As a result, using "Linux-2.6.22.19-0.4-default" in the name
would cause more user concern than necessary.

> Perhaps this is why the easy_install authors thought their 
> package system worked. They only tested it on linux?

Probably. Most .eggs are Python-only, so they don't even need a
platform string. The others are mostly for Windows.

> Yes I agree that obviously this cannot be changed for 2.6. But it would 
> be great to get this code in for python 2.7
> 
> Anyway, this is ultimately your call, since I don't have the 50,000 foot 
> view over the entire situation. 

Is that binary compatibility scheme documented somewhere ?

If so, we could switch to '%s-%s' % (sys.platform, machine)
for Python 2.7.

> FWIW, I've had to patch python in a very large set of client 
> installations. This patch has become a routine in order to unbreak 
> python wrt to platform dependent packages.

Since this only affects easy_install/setuptools, it's probably
easier to just create a patched egg for that and then use it
with the normal Python installation.

Marc-Andre Lemburg wrote:
> Is that binary compatibility scheme documented somewhere ?

Not sure, it's been referred to and adhered to many times in the FreeBSD
community, but I'm not sure where it's documented. I'll ask around on
the FreeBSD mailing lists and post my findings here.

FWIW, the freebsd kernel and package system have options specifically
for compatibility with previous major versions. ie: FreeBSD 8 has
COMPAT_FREEBSD7 and COMPAT_FREEBSD6 kernel options, and has
misc/compat7x and misc/compat6x libraries available for install. But
again, I'll let you know for sure.

>> FWIW, I've had to patch python in a very large set of client 
>> installations. This patch has become a routine in order to unbreak 
>> python wrt to platform dependent packages.
> 
> Since this only affects easy_install/setuptools, it's probably
> easier to just create a patched egg for that and then use it
> with the normal Python installation.

True, I could give that a shot.

About FreeBSD ABI compatibility between minor versions:

Julian Elischer wrote:
> It is a policy of the project but I don't think our policies are 
> written down as such. I think you will find it referenced in
> many places in a sideways manner rather than directly.

http://docs.freebsd.org/cgi/getmsg.cgi?fetch=141507+0+current/freebsd-
hackers

This patch was a short-term fix.  Distutils is closed to anything but bug fixes and distutils2 can get a better fix.  Objections if I close this?

Shrug. I guess you can close it.

This is still a bothersome issue, but we've taken to patching every version of python downstream before deploying them. All for a simple three line patch.

> This is still a bothersome issue, but we've taken to patching every version of python
> downstream before deploying them. All for a simple three line patch.

Sorry about the unsatisfactory situation.  Could we start anew and define exactly what the problem is, so that distutils2 can be free of it?  (I’m afraid distutils can’t be changed: even undocumented, the platform string used for FreeBSD is certainly used by tools out there that we don’t want to break.  I second the suggestion to bring up the issue to the projects responsible for eggs, i.e. setuptools and distribute, not distutils.)

Good plan.

So the issue is:

 * Platform specific eggs are built containing a path that has the full
   patch level of the freebsd kernel, like "8.2-RELEASE-p2". The "-p2"
   part is updated for every security patch of FreeBSD.
 * Thus when you apply a security patch to FreeBSD, platform specific
   eggs built for that version of FreeBSD (before the security patch
   was applied) are no longer considered compatible.

FYI, FreeBSD has an unwritten policy of keeping all 8.x releases
backwards compatible with one another. So platform specific eggs built
for 8.1 would work without inherent problems on 8.2 or 8.3.

But at the very least, platform specific eggs should not be dependent on
the patch level of the FreeBSD kernel.

On 11/20/2011 03:38 PM, Éric Araujo wrote:
> 
> Éric Araujo <merwok@netwok.org> added the comment:
> 
>> This is still a bothersome issue, but we've taken to patching every version of python
>> downstream before deploying them. All for a simple three line patch.
> 
> Sorry about the unsatisfactory situation.  Could we start anew and define exactly what the problem is, so that distutils2 can be free of it?  (I’m afraid distutils can’t be changed: even undocumented, the platform string used for FreeBSD is certainly used by tools out there that we don’t want to break.  I second the suggestion to bring up the issue to the projects responsible for eggs, i.e. setuptools and distribute, not distutils.)
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue6983>
> _______________________________________

Thanks for the reply.  distutils2 won’t support eggs, and I don’t think setuptools or distribute will be reworked to work on distutils2, so if your problem is only with eggs there is nothing that we can do to solve it in distutils2.  (As I said, distutils is frozen and can’t be changed to address this either.)  Currently distutils2 build sdists and bdists of the dumb, msi and wininst varieties.  If these formats have the same issue that you describe, then we can change them.

Closing this issue; judging by the comments, the fix should be pushed off to setuptools/distribute/whatever.