classification
Title: distutils.utils.get_platform() for 32-bit Python on a 64-bit machine
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: Arfrever, Dima.Tisnek, Michael.Felt, djones, eric.araujo, jaraco, jkloth, madprog, ned.deily, pitrou, sferencik, steve.dower, tarek
Priority: normal Keywords:

Created on 2013-09-09 13:03 by sferencik, last changed 2021-02-03 18:07 by steve.dower. This issue is now closed.

Messages (24)
msg197363 - (view) Author: Sam Ferencik (sferencik) Date: 2013-09-09 13:03
distutils.util.get_platform() semantically differs on (a) Windows and OS X, and on (b) Linux.

Windows/OS X: the return value is derived from the architecture of the *interpreter*, hence for 32-bit Python running on a 64-bit system, get_platform() = 'win32'/'macosx-10.6-i386' (32-bit).

Linux: the return value is derived from the architecture of the *OS*, hence for 32-bit Python running on 64-bit Linux get_platform() = 'linux-x86_64' (64-bit).

Based on a discussion on distutils-sig, the Linux behaviour is probably wrong and should be changed. https://mail.python.org/pipermail/distutils-sig/2013-August/subject.html

My context (where this hit me): I was installing the 32-bit version of the Perforce API (compiled module) on 64-bit Windows and on 64-bit Linux. My command-line was

  python3.3-32 setup.py install --root FOO --install-platlib=lib.$PLAT

(note the '-32' and the '$PLAT')

On Windows, this installed the 32-bit version of the API into "FOO\lib.win32". On Linux, this installed the 64-bit version of the API into "FOO/lib.linux-x86_64".
msg197385 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-09-09 19:37
Do you want to upload a patch?
msg197475 - (view) Author: Sam Ferencik (sferencik) Date: 2013-09-11 07:26
Unfortunately, I don't have a patch.

Some thoughts:

To discover a 32-bit interpreter running on a 64-bit system, we could use
platform.architecture(), which returns
    >>> platform.architecture()
    ('32bit', 'ELF')

What then, though? How do you turn '32bit' to 'linux-i386'? The naive solution
would be to hard-code this as an exception:
- if 32-on-64, use 'i386'
- otherwise, use os.uname()[4], i.e. 'i386' or 'x86_64'

I suspect that's ultra naive, though. The get_platform() code deals with a
number of Unix-like systems (Solaris, AIX, ...). Even if we accepted this
solution only for Linux, leaving the other OSs broken, is 'i386' always the
right answer, or would 'i586' or similar be appropriate in some cases?

We could also take inspiration from Mac OS (_osx_support.get_platform_osx()),
which basically ignores os.uname()[4] completely and constructs the return value
from a set of hard-coded values ('i386' being one of them).

What do you think?
msg197535 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-09-12 15:18
Hello,

> To discover a 32-bit interpreter running on a 64-bit system, we could
> use
> platform.architecture(), which returns
>     >>> platform.architecture()
>     ('32bit', 'ELF')

Just use (sys.maxsize < 2**32).

> What then, though? How do you turn '32bit' to 'linux-i386'?

I don't know. How does distutils normally do? :-)
msg197584 - (view) Author: Sam Ferencik (sferencik) Date: 2013-09-13 14:57
Are you asking *what* distutils does?

It tackles the problem completely differently on Windows, Unix, and OS X.
msg197586 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-09-13 15:14
> Are you asking *what* distutils does?

Yup :-) I'm not a distutils maintainer, so I hardly know how it does
things internally.

> It tackles the problem completely differently on Windows, Unix, and
> OS X.

Ah... but does it compute the result by itself or simply queries some
kind of external information?
If the former, then it should be simple to give it the right bitness
value (32 vs. 64). If the latter, things will get a bit more...
interesting :-)
msg197589 - (view) Author: Sam Ferencik (sferencik) Date: 2013-09-13 15:20
It's very hacky on all of Windows, Unix, and OS X. That's why I don't feel confident to propose a solution.

On Unix, specifically, the return value is heavily based on os.uname(). It seems that the maintainers of OS X have started with the same but then chose to diverge and instead use a list of values.
msg197592 - (view) Author: √Čric Araujo (eric.araujo) * (Python committer) Date: 2013-09-13 15:33
FTR the Mac OS code does some normalization: http://hg.python.org/cpython/file/bda5a87df1c8/Lib/_osx_support.py#l473

Code for linux just returns the value from uname, as Same said: http://hg.python.org/cpython/file/bda5a87df1c8/Lib/distutils/util.py#l75
msg197596 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-09-13 15:36
> On Unix, specifically, the return value is heavily based on os.uname().

Ouch. Then I'm afraid this is a probably a won't fix :-/
msg197600 - (view) Author: Sam Ferencik (sferencik) Date: 2013-09-13 15:45
Well, the maintainers of Mac OS didn't consider it a won't fix - and have this working properly. I don't see why we couldn't try to copy what they did.

Actually, I think the impact of changing this for 32-bit Python on 64-bit Linux should be quite small, no?
msg197607 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-09-13 16:05
> Well, the maintainers of Mac OS didn't consider it a won't fix - and
> have this working properly. I don't see why we couldn't try to copy
> what they did.

Ah, ok. Then a patch would be welcome :-)

> Actually, I think the impact of changing this for 32-bit Python on
> 64-bit Linux should be quite small, no?

Well, if you can get the semantics reliably right, yes, probably.
msg197668 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-09-13 21:44
A compatibility issue here is that the value provided by get_platform() is also used outside of Distutils, in particular by pkg_resources (provided by setuptools) and by pip, in both cases to help determine whether a binary distribution of an extension module is compatible with the python being used.  It's also used in Distutils to form the name of the build directory for extension modules.  I believe at the moment pip requires an exact match of the get_platform() value for most platforms, other than OS X, so it is possible that making this change could break downloads of existing binary egg-based distributions. (On the other hand, binary distributions for Linux platforms are known to be problematic because of API differences that are not reflected in the platform-derived string.)  So any change in behavior here should probably be limited to a feature release (like 3.4) and in coordination with the Pypa packaging developers (setuptools, pip, wheel, et al) to minimize disruptions.
msg197912 - (view) Author: Sam Ferencik (sferencik) Date: 2013-09-16 15:59
Thanks for the context.

> A compatibility issue here is that the value provided by get_platform() is
> also used outside of Distutils, in particular by pkg_resources (provided by
> setuptools) and by pip, in both cases to help determine whether a binary
> distribution of an extension module is compatible with the python being used.

If your wording is correct, and get_platform() really is used to determine "the
python being used," then we could actually be improving things by fixing this.
Currently, get_platform() doesn't tell you the bitness of the *python* being
used, but the bitness of the *OS* being used (the two of which only differ on
32-on-64).
msg204135 - (view) Author: David Jones (djones) Date: 2013-11-24 00:12
Has there been any progress made on fixing this? I ran into this trying to install numpy via pip, 32-bit python installation on 64-bit Centos 6.4. It get's the compile flags right, but not the linker:

C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -m32 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC
compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/python/ia32/include/python2.7 -c'
gcc: _configtest.c
gcc -pthread _configtest.o -o _configtest
_configtest.o: could not read symbols: File in wrong format
collect2: ld returned 1 exit status

Someone worked around this by changing ccompiler.py, line 693 to:
runtime_library_dirs=None, debug=0, extra_preargs=['-m32'],

See: http://stackoverflow.com/questions/11265057/how-do-i-install-a-32-bit-version-of-numpy

I tried using the setarch command, which alters the output of uname, but it didn't change anything:
setarch i686 pip install numpy

This changes the output of uname from
Linux centos63-vm 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 18:37:12 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

to 

Linux centos63-vm 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 18:37:12 UTC 2013 i686 i686 i386 GNU/Linux

So if get_platform really depends on uname, then why doesn't this work?
msg246402 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2015-07-07 11:47
beep!
msg261050 - (view) Author: Michael Felt (Michael.Felt) * Date: 2016-03-01 13:59
while reading to learn...

FYI: AIX does not return even a hint of the underlying platform (assumption is probably 64-bit) - but the value of the interpreter is anyone's guess

root@x064:[/data/prj/aixtools/python/python-2.7.10]python
Python 2.7.10 (default, Mar  1 2016, 12:55:35) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.util
>>> distutils.util.get_platform()
'aix-5.3'
>>> import os
>>> os.uname()
('AIX', 'x064', '3', '5', '00XXXXXXXXXX')


Going through the rest of the preceding messages:
a) platform.architecture() - seems accurate for AIX, is import so expensive that it is not preferred?
>>> import platform
>>> platform.architecture()
('32bit', '')

b) sys.maxsize < 2**16 is a good hint, but it is possible to have a 64-bit int with only 32-bit pointers - based on assumption!
>>> import os, sys
>>> sys.maxsize
2147483647
>>> 2**32
4294967296L
>>> 2**31
2147483648L

> If your wording is correct, and get_platform() really is used to determine "the
> python being used," then we could actually be improving things by fixing this.
> Currently, get_platform() doesn't tell you the bitness of the *python* being
> used, but the bitness of the *OS* being used (the two of which only differ on
> 32-on-64).

So, it seems get_platform() does not have a uniform result that could ever be parsed by internal or external modules. It is always dependent on something external to python (e.g., uname() output).

IMHO: determining and adding the python bitness, as MacOS has done (i.e., I assume it is either (32-bit) or (64-bit) at the end) is a move forward - and I would be willing to submit a couple of somethings so that that value could be determined - leave it to a committer to decide what makes best sense.

Of course, it is "sad" if it breaks existing things, but being non-uniform as it is worse (imho :smile:)
msg261051 - (view) Author: Sam Ferencik (sferencik) Date: 2016-03-01 14:09
Thanks for the analysis. I agree with you. If there's much push-back, maybe we could introduce an alternative interface, i.e. let get_platform() do its thing, deprecate it, and introduce something like get_interpreter_platform().
msg261057 - (view) Author: Michael Felt (Michael.Felt) * Date: 2016-03-01 16:20
If you need assistance (re: AIX), just ask.

However, I am guessing you have enough to move forward.
msg261187 - (view) Author: Michael Felt (Michael.Felt) * Date: 2016-03-04 13:38
FYI: build as 64-bit (and shall only build as 64-bit from now I expect)
and the output works as:
==============================
aixtools.python:aixtools.python.man.en_US:2.7.11.0::I:C:::::N:man pages::::0::
aixtools.python:aixtools.python.rte:2.7.11.0::I:C:::::N:built 04-Mar-2016 1232 UTC::::0::
==============================
root@x064:[/data/prj/aixtools/python/python-2.7.11]./python
Python 2.7.11 (default, Mar  4 2016, 12:29:39) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.is_python_build()
True
>>> import distutils.util
>>> distutils.util.get_platform()
'aix-5.3'
>>> import os
>>> os.uname()
('AIX', 'x064', '3', '5', '00C291F54C00')
>>> import platform
>>> platform.architecture()
('64bit', '')
>>> import os, sys
>>> sys.maxsize
9223372036854775807
>>> 2**32
4294967296
>>> 

I was wondering if there is a PEP statement somewhere re: distutils.util.get_platform() semantics
msg279824 - (view) Author: Michael Felt (Michael.Felt) * Date: 2016-10-31 20:09
There are so many places where there are references to where 32-bit and 64-bit are referred to - and, in particular, the value of os.uname()[4]

For the discussion I went back to 32-bit Python on AIX

A)
michael@x071:[/data/prj/python/archive/Python-2.7.3]python
Python 2.7.3 (default, Sep 26 2013, 20:43:10) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.platform()
'AIX-1-00C291F54C00-powerpc-32bit'
>>> import os
>>> os.uname()[4]
'00C291F54C00'
>>> import sys
>>> sys.maxsize
2147483647

B)
root@x066:~# python
Python 2.7.3 (default, Mar 14 2014, 12:37:29)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.platform()
'Linux-3.2.0-4-powerpc64-ppc64-with-debian-7.8'
>>> import os
>>> os.uname()[4]
'ppc64'
>>> import sys
>>> sys.maxsize
2147483647
>>>

The data comes from the same hardware - just different OS.

re: os.uname()[4] and platform.platform()
for AIX os.uname()[4] says next to nothing - the only bit relavant to the "machine" are the letters C4 and those are likely to be found on any POWER 32 and 64-bit (no longer have direct access to 32-bit hardware so must trust the documentation on that one) - for AIX platform.platform() does give proper machine and interpreter size.

For Linux (debian in this case) os.uname()[4] identifies the hardware platform, and platform.platform() "looks" 64-bit to me, but sys.maxsize clearly shows this is 32-bit.

I am willing to work on a patch - even digging on the POWER Linux side as well - but to write a patch the desired output is needed.

And, I would suggest replacing the -m output with either -M or -p output:
michael@x071:[/data/prj/python/archive/Python-2.7.3]uname -m -M
00C291F54C00 IBM,8203-E4A
michael@x071:[/data/prj/python/archive/Python-2.7.3]uname -m -p
00C291F54C00 powerpc

Again - what is the preferred output. IMHO - platform.platform() comes very close for what I would be expecting.
AIX (noise) powerpc 32bit 
or
AIX (noise) powerpc 64bit

Comments? (and has this ever been discussed/specified in a PEP? In another discussion someone mentioned something about a wheel specification - but that refers back to distutils and IMHO this is "core" first, and distutils (should be) following)
msg279825 - (view) Author: Michael Felt (Michael.Felt) * Date: 2016-10-31 20:34
FYI: This is 'actual' as I am working on an implementation of a cloud-init distro for AIX and it is very difficult to figure out the correct approach for a replacement value for os.uname[4] - when comparing with "Linux" logic

I was thinking of using

platform.platform().split("-")[3]

but both are 32-bit on 64-bit hardware:

>>> platform.platform().split("-")
['AIX', '1', '00C291F54C00', 'powerpc', '32bit']
>>> platform.platform().split("-")[4]
'32bit'
>>> platform.platform().split("-")[3]
'powerpc'

>>> platform.platform().split("-")
['Linux', '3.2.0', '4', 'powerpc64', 'ppc64', 'with', 'debian', '7.8']
>>> platform.platform().split("-")[4]
'ppc64'
>>> platform.platform().split("-")[3]
'powerpc64'

This - also - seems tricky re: the placement of the -
>>> platform.platform()
'Linux-3.2.0-4-powerpc64-ppc64-with-debian-7.8'

compared with:
>>> platform.platform()
'AIX-1-00C291F54C00-powerpc-32bit'

Truely, some guidance from "the powers that be" is needed if there is ever to be uniformity. And if it 'cannot' be patched on 'old' versions, at least there will be a way to hack clarity into code (imho, a patch is better rather than hacks creating continued diversity - everyone continues to use their understanding of intent, creating new diverse wishes (aka features not benefits)) for what the code "must" do - because there are so many projects that used it this way (and others that used it that way) ....

:)
msg279863 - (view) Author: Sam Ferencik (sferencik) Date: 2016-11-01 09:37
Michael,

Thanks for reopening this. You say you're using "64-bit hardware", but what bitness is your OS and the Python interpreter?

If you read my original issue description, I only had this issue with 32-bit Python on a 64-bit Linux system (on 64-bit hardware).

You say you have 64-bit hardware but are both your AIX and Linux 64-bit? And what about your Python? (Can you try invoking python2.7-32 and python2.7-64 explicitly?) Please add this detail.

Thanks,
Sam
msg365074 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-03-26 13:52
Back again - I understood a lot less then, maybe more now..

iirc, get_platform() asin sysconfig.get_platform() and distutils.util.get_platform() are suppposed to return a suitable PEP425 tag that identifies the ABI of the running interpreter - eg.g, 32-bit even though operating on 64-bot OS and hardware.

As far as AIX goes, for years this was not the case. More recently (for Python 3.9) the logic has been added so that the bitness of the interpreter can be identified.

i.e., on AIX `get_platform()` now returns:
'aix-5307-0747-64' - where the 64 (or 32) identifies the bitness of the interpreter.

So, my question now - are the PEP425 tags returned by other platforms adequate? If yes, then I see little reason to not close this issue as resolved (elsewhere).
msg386264 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-03 18:07
Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools
History
Date User Action Args
2021-02-03 18:07:57steve.dowersetstatus: open -> closed

nosy: + steve.dower
messages: + msg386264

resolution: out of date
stage: needs patch -> resolved
2020-03-26 13:52:43Michael.Feltsetmessages: + msg365074
2016-11-01 09:37:50sferenciksetmessages: + msg279863
2016-10-31 20:34:26Michael.Feltsetmessages: + msg279825
2016-10-31 20:09:49Michael.Feltsetmessages: + msg279824
2016-03-29 15:59:30madprogsetnosy: + madprog
2016-03-04 13:38:53Michael.Feltsetmessages: + msg261187
2016-03-01 16:20:58Michael.Feltsetmessages: + msg261057
2016-03-01 14:09:39sferenciksetmessages: + msg261051
2016-03-01 13:59:27Michael.Feltsetnosy: + Michael.Felt
messages: + msg261050
2015-12-02 16:34:57jaracosetstage: needs patch
versions: + Python 3.5, - Python 3.3
2015-07-07 11:47:10Dima.Tisneksetnosy: + Dima.Tisnek
messages: + msg246402
2015-01-13 23:19:00jaracosetnosy: + jaraco
2013-11-24 00:14:14gregory.p.smithsettitle: distutils.utils.get_platform() for 32-bit Python on a 64-bit machine -> distutils.utils.get_platform() for 32-bit Python on a 64-bit machine
2013-11-24 00:12:45djonessetnosy: + djones
messages: + msg204135
2013-09-16 15:59:02sferenciksetmessages: + msg197912
2013-09-13 21:44:22ned.deilysetnosy: + ned.deily
messages: + msg197668
2013-09-13 16:05:00pitrousetmessages: + msg197607
2013-09-13 15:45:32sferenciksetmessages: + msg197600
2013-09-13 15:36:21pitrousetmessages: + msg197596
2013-09-13 15:33:01eric.araujosetmessages: + msg197592
2013-09-13 15:20:00sferenciksetmessages: + msg197589
2013-09-13 15:14:13pitrousetmessages: + msg197586
title: distutils.utils.get_platform() for 32-bit Python on a 64-bit machine -> distutils.utils.get_platform() for 32-bit Python on a 64-bit machine
2013-09-13 14:57:16sferenciksetmessages: + msg197584
2013-09-12 15:18:52pitrousetmessages: + msg197535
title: distutils.utils.get_platform() for 32-bit Python on a 64-bit machine -> distutils.utils.get_platform() for 32-bit Python on a 64-bit machine
2013-09-11 07:26:48sferenciksetmessages: + msg197475
2013-09-10 21:17:56Arfreversetnosy: + Arfrever
2013-09-09 19:37:09pitrousetnosy: + pitrou

messages: + msg197385
versions: + Python 2.7, Python 3.4
2013-09-09 18:42:49jklothsetnosy: + jkloth
2013-09-09 13:03:01sferencikcreate