distutils.util.get_platform() semantically differs on (a) Windows and OS X, and on (b) Linux.

Windows/OS X: the return value is derived from the architecture of the *interpreter*, hence for 32-bit Python running on a 64-bit system, get_platform() = 'win32'/'macosx-10.6-i386' (32-bit).

Linux: the return value is derived from the architecture of the *OS*, hence for 32-bit Python running on 64-bit Linux get_platform() = 'linux-x86_64' (64-bit).

Based on a discussion on distutils-sig, the Linux behaviour is probably wrong and should be changed. https://mail.python.org/pipermail/distutils-sig/2013-August/subject.html

My context (where this hit me): I was installing the 32-bit version of the Perforce API (compiled module) on 64-bit Windows and on 64-bit Linux. My command-line was

  python3.3-32 setup.py install --root FOO --install-platlib=lib.$PLAT

(note the '-32' and the '$PLAT')

On Windows, this installed the 32-bit version of the API into "FOO\lib.win32". On Linux, this installed the 64-bit version of the API into "FOO/lib.linux-x86_64".

Do you want to upload a patch?

Unfortunately, I don't have a patch.

Some thoughts:

To discover a 32-bit interpreter running on a 64-bit system, we could use
platform.architecture(), which returns
    >>> platform.architecture()
    ('32bit', 'ELF')

What then, though? How do you turn '32bit' to 'linux-i386'? The naive solution
would be to hard-code this as an exception:
- if 32-on-64, use 'i386'
- otherwise, use os.uname()[4], i.e. 'i386' or 'x86_64'

I suspect that's ultra naive, though. The get_platform() code deals with a
number of Unix-like systems (Solaris, AIX, ...). Even if we accepted this
solution only for Linux, leaving the other OSs broken, is 'i386' always the
right answer, or would 'i586' or similar be appropriate in some cases?

We could also take inspiration from Mac OS (_osx_support.get_platform_osx()),
which basically ignores os.uname()[4] completely and constructs the return value
from a set of hard-coded values ('i386' being one of them).

What do you think?

Hello,

> To discover a 32-bit interpreter running on a 64-bit system, we could
> use
> platform.architecture(), which returns
>     >>> platform.architecture()
>     ('32bit', 'ELF')

Just use (sys.maxsize < 2**32).

> What then, though? How do you turn '32bit' to 'linux-i386'?

I don't know. How does distutils normally do? :-)

Are you asking *what* distutils does?

It tackles the problem completely differently on Windows, Unix, and OS X.

> Are you asking *what* distutils does?

Yup :-) I'm not a distutils maintainer, so I hardly know how it does
things internally.

> It tackles the problem completely differently on Windows, Unix, and
> OS X.

Ah... but does it compute the result by itself or simply queries some
kind of external information?
If the former, then it should be simple to give it the right bitness
value (32 vs. 64). If the latter, things will get a bit more...
interesting :-)

It's very hacky on all of Windows, Unix, and OS X. That's why I don't feel confident to propose a solution.

On Unix, specifically, the return value is heavily based on os.uname(). It seems that the maintainers of OS X have started with the same but then chose to diverge and instead use a list of values.

FTR the Mac OS code does some normalization: http://hg.python.org/cpython/file/bda5a87df1c8/Lib/_osx_support.py#l473

Code for linux just returns the value from uname, as Same said: http://hg.python.org/cpython/file/bda5a87df1c8/Lib/distutils/util.py#l75

> On Unix, specifically, the return value is heavily based on os.uname().

Ouch. Then I'm afraid this is a probably a won't fix :-/

Well, the maintainers of Mac OS didn't consider it a won't fix - and have this working properly. I don't see why we couldn't try to copy what they did.

Actually, I think the impact of changing this for 32-bit Python on 64-bit Linux should be quite small, no?

> Well, the maintainers of Mac OS didn't consider it a won't fix - and
> have this working properly. I don't see why we couldn't try to copy
> what they did.

Ah, ok. Then a patch would be welcome :-)

> Actually, I think the impact of changing this for 32-bit Python on
> 64-bit Linux should be quite small, no?

Well, if you can get the semantics reliably right, yes, probably.

A compatibility issue here is that the value provided by get_platform() is also used outside of Distutils, in particular by pkg_resources (provided by setuptools) and by pip, in both cases to help determine whether a binary distribution of an extension module is compatible with the python being used.  It's also used in Distutils to form the name of the build directory for extension modules.  I believe at the moment pip requires an exact match of the get_platform() value for most platforms, other than OS X, so it is possible that making this change could break downloads of existing binary egg-based distributions. (On the other hand, binary distributions for Linux platforms are known to be problematic because of API differences that are not reflected in the platform-derived string.)  So any change in behavior here should probably be limited to a feature release (like 3.4) and in coordination with the Pypa packaging developers (setuptools, pip, wheel, et al) to minimize disruptions.

Thanks for the context.

> A compatibility issue here is that the value provided by get_platform() is
> also used outside of Distutils, in particular by pkg_resources (provided by
> setuptools) and by pip, in both cases to help determine whether a binary
> distribution of an extension module is compatible with the python being used.

If your wording is correct, and get_platform() really is used to determine "the
python being used," then we could actually be improving things by fixing this.
Currently, get_platform() doesn't tell you the bitness of the *python* being
used, but the bitness of the *OS* being used (the two of which only differ on
32-on-64).

Has there been any progress made on fixing this? I ran into this trying to install numpy via pip, 32-bit python installation on 64-bit Centos 6.4. It get's the compile flags right, but not the linker:

C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -m32 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC
compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/python/ia32/include/python2.7 -c'
gcc: _configtest.c
gcc -pthread _configtest.o -o _configtest
_configtest.o: could not read symbols: File in wrong format
collect2: ld returned 1 exit status

Someone worked around this by changing ccompiler.py, line 693 to:
runtime_library_dirs=None, debug=0, extra_preargs=['-m32'],

See: http://stackoverflow.com/questions/11265057/how-do-i-install-a-32-bit-version-of-numpy

I tried using the setarch command, which alters the output of uname, but it didn't change anything:
setarch i686 pip install numpy

This changes the output of uname from
Linux centos63-vm 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 18:37:12 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

to 

Linux centos63-vm 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 18:37:12 UTC 2013 i686 i686 i386 GNU/Linux

So if get_platform really depends on uname, then why doesn't this work?

beep!

while reading to learn...

FYI: AIX does not return even a hint of the underlying platform (assumption is probably 64-bit) - but the value of the interpreter is anyone's guess

root@x064:[/data/prj/aixtools/python/python-2.7.10]python
Python 2.7.10 (default, Mar  1 2016, 12:55:35) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.util
>>> distutils.util.get_platform()
'aix-5.3'
>>> import os
>>> os.uname()
('AIX', 'x064', '3', '5', '00XXXXXXXXXX')


Going through the rest of the preceding messages:
a) platform.architecture() - seems accurate for AIX, is import so expensive that it is not preferred?
>>> import platform
>>> platform.architecture()
('32bit', '')

b) sys.maxsize < 2**16 is a good hint, but it is possible to have a 64-bit int with only 32-bit pointers - based on assumption!
>>> import os, sys
>>> sys.maxsize
2147483647
>>> 2**32
4294967296L
>>> 2**31
2147483648L

> If your wording is correct, and get_platform() really is used to determine "the
> python being used," then we could actually be improving things by fixing this.
> Currently, get_platform() doesn't tell you the bitness of the *python* being
> used, but the bitness of the *OS* being used (the two of which only differ on
> 32-on-64).

So, it seems get_platform() does not have a uniform result that could ever be parsed by internal or external modules. It is always dependent on something external to python (e.g., uname() output).

IMHO: determining and adding the python bitness, as MacOS has done (i.e., I assume it is either (32-bit) or (64-bit) at the end) is a move forward - and I would be willing to submit a couple of somethings so that that value could be determined - leave it to a committer to decide what makes best sense.

Of course, it is "sad" if it breaks existing things, but being non-uniform as it is worse (imho :smile:)

Thanks for the analysis. I agree with you. If there's much push-back, maybe we could introduce an alternative interface, i.e. let get_platform() do its thing, deprecate it, and introduce something like get_interpreter_platform().

If you need assistance (re: AIX), just ask.

However, I am guessing you have enough to move forward.

FYI: build as 64-bit (and shall only build as 64-bit from now I expect)
and the output works as:
==============================
aixtools.python:aixtools.python.man.en_US:2.7.11.0::I:C:::::N:man pages::::0::
aixtools.python:aixtools.python.rte:2.7.11.0::I:C:::::N:built 04-Mar-2016 1232 UTC::::0::
==============================
root@x064:[/data/prj/aixtools/python/python-2.7.11]./python
Python 2.7.11 (default, Mar  4 2016, 12:29:39) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.is_python_build()
True
>>> import distutils.util
>>> distutils.util.get_platform()
'aix-5.3'
>>> import os
>>> os.uname()
('AIX', 'x064', '3', '5', '00C291F54C00')
>>> import platform
>>> platform.architecture()
('64bit', '')
>>> import os, sys
>>> sys.maxsize
9223372036854775807
>>> 2**32
4294967296
>>> 

I was wondering if there is a PEP statement somewhere re: distutils.util.get_platform() semantics

There are so many places where there are references to where 32-bit and 64-bit are referred to - and, in particular, the value of os.uname()[4]

For the discussion I went back to 32-bit Python on AIX

A)
michael@x071:[/data/prj/python/archive/Python-2.7.3]python
Python 2.7.3 (default, Sep 26 2013, 20:43:10) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.platform()
'AIX-1-00C291F54C00-powerpc-32bit'
>>> import os
>>> os.uname()[4]
'00C291F54C00'
>>> import sys
>>> sys.maxsize
2147483647

B)
root@x066:~# python
Python 2.7.3 (default, Mar 14 2014, 12:37:29)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.platform()
'Linux-3.2.0-4-powerpc64-ppc64-with-debian-7.8'
>>> import os
>>> os.uname()[4]
'ppc64'
>>> import sys
>>> sys.maxsize
2147483647
>>>

The data comes from the same hardware - just different OS.

re: os.uname()[4] and platform.platform()
for AIX os.uname()[4] says next to nothing - the only bit relavant to the "machine" are the letters C4 and those are likely to be found on any POWER 32 and 64-bit (no longer have direct access to 32-bit hardware so must trust the documentation on that one) - for AIX platform.platform() does give proper machine and interpreter size.

For Linux (debian in this case) os.uname()[4] identifies the hardware platform, and platform.platform() "looks" 64-bit to me, but sys.maxsize clearly shows this is 32-bit.

I am willing to work on a patch - even digging on the POWER Linux side as well - but to write a patch the desired output is needed.

And, I would suggest replacing the -m output with either -M or -p output:
michael@x071:[/data/prj/python/archive/Python-2.7.3]uname -m -M
00C291F54C00 IBM,8203-E4A
michael@x071:[/data/prj/python/archive/Python-2.7.3]uname -m -p
00C291F54C00 powerpc

Again - what is the preferred output. IMHO - platform.platform() comes very close for what I would be expecting.
AIX (noise) powerpc 32bit 
or
AIX (noise) powerpc 64bit

Comments? (and has this ever been discussed/specified in a PEP? In another discussion someone mentioned something about a wheel specification - but that refers back to distutils and IMHO this is "core" first, and distutils (should be) following)

FYI: This is 'actual' as I am working on an implementation of a cloud-init distro for AIX and it is very difficult to figure out the correct approach for a replacement value for os.uname[4] - when comparing with "Linux" logic

I was thinking of using

platform.platform().split("-")[3]

but both are 32-bit on 64-bit hardware:

>>> platform.platform().split("-")
['AIX', '1', '00C291F54C00', 'powerpc', '32bit']
>>> platform.platform().split("-")[4]
'32bit'
>>> platform.platform().split("-")[3]
'powerpc'

>>> platform.platform().split("-")
['Linux', '3.2.0', '4', 'powerpc64', 'ppc64', 'with', 'debian', '7.8']
>>> platform.platform().split("-")[4]
'ppc64'
>>> platform.platform().split("-")[3]
'powerpc64'

This - also - seems tricky re: the placement of the -
>>> platform.platform()
'Linux-3.2.0-4-powerpc64-ppc64-with-debian-7.8'

compared with:
>>> platform.platform()
'AIX-1-00C291F54C00-powerpc-32bit'

Truely, some guidance from "the powers that be" is needed if there is ever to be uniformity. And if it 'cannot' be patched on 'old' versions, at least there will be a way to hack clarity into code (imho, a patch is better rather than hacks creating continued diversity - everyone continues to use their understanding of intent, creating new diverse wishes (aka features not benefits)) for what the code "must" do - because there are so many projects that used it this way (and others that used it that way) ....

:)

Michael,

Thanks for reopening this. You say you're using "64-bit hardware", but what bitness is your OS and the Python interpreter?

If you read my original issue description, I only had this issue with 32-bit Python on a 64-bit Linux system (on 64-bit hardware).

You say you have 64-bit hardware but are both your AIX and Linux 64-bit? And what about your Python? (Can you try invoking python2.7-32 and python2.7-64 explicitly?) Please add this detail.

Thanks,
Sam

Back again - I understood a lot less then, maybe more now..

iirc, get_platform() asin sysconfig.get_platform() and distutils.util.get_platform() are suppposed to return a suitable PEP425 tag that identifies the ABI of the running interpreter - eg.g, 32-bit even though operating on 64-bot OS and hardware.

As far as AIX goes, for years this was not the case. More recently (for Python 3.9) the logic has been added so that the bitness of the interpreter can be identified.

i.e., on AIX `get_platform()` now returns:
'aix-5307-0747-64' - where the 64 (or 32) identifies the bitness of the interpreter.

So, my question now - are the PEP425 tags returned by other platforms adequate? If yes, then I see little reason to not close this issue as resolved (elsewhere).

Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools