classification
Title: rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)
Type: Stage: needs patch
Components: Installation Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, barry, doko, freakboy3742, koobs, larry, ncoghlan, ned.deily, python-dev, rkuska, sYnfo, thomas-petazzoni, vstinner, wscullin, xdegaye, yan12125, zach.ware
Priority: release blocker Keywords: needs review, patch

Created on 2015-04-15 19:04 by doko, last changed 2016-08-31 18:22 by yan12125. This issue is now closed.

Files
File name Uploaded Description Edit
platdir.diff doko, 2015-04-15 19:04 patch
platdir2.diff doko, 2016-06-04 23:13 review
issue23968-android-regression.patch yan12125, 2016-08-30 08:20 Fix a regression after changeset 5c77488830bc review
Messages (32)
msg241143 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2015-04-15 19:04
Currently there is only one platform directory for all linux architectures, there are two reasons to have a separate directory for each architecture:

 - the contents of the files in this directory differ
   (most prominent case are the RTLD_* constants on various
   architectures).

 - The _sysconfigdata.py module is installed in the platdir.
   A cross build of the extensions in the standard library
   is done by running the python interpreter for the build
   machine with the platdir for the host machine.  This
   currently works for cross builds targeting a different
   os, but not cross building from one linux architecture
   to another.

This would then remove Lib/plat-linux and introduce some Lib/plat-$(PLATFORM_TRIPLET) directories.
msg243874 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2015-05-23 00:26
ping? I would like to get this into 3.5.
msg243896 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-05-23 08:17
+1 from me.
msg244362 - (view) Author: Matěj Stuchlík (sYnfo) Date: 2015-05-29 09:14
+1 from the Python team from Fedora, the patch looks good from downstream standpoint.
msg244411 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2015-05-29 20:09
I don't know what problems I might have run into previously, but it's working now and seems fine to me.  +1
msg244605 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2015-06-01 18:24
What Lib/plat-$(PLATFORM_TRIPLET) directories will be introduced?

This patch would make situation worse on rarely used architectures (these without Lib/plat-$(PLATFORM_TRIPLET) distributed in repository of CPython) if patch for bug #12619 is not applied.
msg248423 - (view) Author: William Scullin (wscullin) Date: 2015-08-11 18:48
This would likely improve life for folks with cross-compile toolchains.
msg267342 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-06-04 23:13
attaching an updated patch. besides renaming of the platform directory, the following changes are made:

 - it makes the name of the internal _sysconfigdata module dependent
   on the abiflags
 - it moves the _sysconfigdata module into the platform directory.
   This address as well #22724, it should address #18083 as well,
   provided that some multiarch tuples for solaris are defined.
 - it fixes regeneration of the Python modules from C headers for
   cross builds.
 - it introduces a sys.implementation._multiarch attribute to construct
   the name of the platform directory without having the
   _sysconfigdata module available (which now is located in the platform
   dir)

One outstanding issue probably is the generation of the platform directory in the source tree, not in the build tree (but this is existing behaviour).

This should address #12619 (this patch probably needs an update),
msg267352 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-06-05 00:22
This may be of interest for iOS and Android cross-compilation, so I've added Xavier de Gaye to the nosy list, while Russell Keith-Magee added himself.
msg268528 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-06-14 06:55
New changeset 78d2cb7f66b6 by doko in branch 'default':
- Issue #23968: Rename the platform directory from plat-$(MACHDEP) to
https://hg.python.org/cpython/rev/78d2cb7f66b6
msg268530 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-06-14 06:57
now checked in, after 3.6.0a2, to watch behaviour on different platforms. Still needs doc updates.
msg268536 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-06-14 07:22
New changeset a062c60c9f7d by doko in branch 'default':
- Issue #23968: Update distutils/sysconfig.py to look for the renamed
https://hg.python.org/cpython/rev/a062c60c9f7d
msg268829 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-06-18 19:59
New changeset e2b09c00ee24 by Ned Deily in branch 'default':
Issue #23968: Fix installs of the renamed config directory for OS X
https://hg.python.org/cpython/rev/e2b09c00ee24
msg268833 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-06-18 20:47
There is a problem with the current implementation.  When running ./configure using compilers other than a current gcc, for example, when using clang or older versions of gcc, the build variable MULTIARCH may not get set because those compilers do not support --print-multiarch:

configure.ac:744
MULTIARCH=$($CC --print-multiarch 2>/dev/null)

But, a bit later in ./configure, PLATFORM_TRIPLET gets set as a result of running conftest.c.  The end result is that PLATFORM_TRIPET is non-empty but MULTIARCH is empty.  Still later in configure.ac (around 4477:

if test x$PLATFORM_TRIPLET = x; then
  LIBPL='$(prefix)'"/lib/python${VERSION}/config-${LDVERSION}"
else
  LIBPL='$(prefix)'"/lib/python${VERSION}/config-${LDVERSION}-${PLATFORM_TRIPLET}"
fi

So, the value of LIBPL is extended to include PLATFORM_TRIPLET regardless of whether MULTIARCH has a value.

This causes problems at runtime because get_makefile_filename() in Lib/sysconfig.py now uses the value of sys.implemntation's _multiarch attribute (whose value is compiled in sysmodule.c from MULTIARCH) to form the path to the config directory (the equivalent of LIBPL).  In these cases, sys.implementation will have no_multiarch attribute so sysconfig looks for the config directory in the pre-3.6 location (e.g. .../lib/python3.6/config-3.6m) even though it was installed in the new location (.../lib/python3.6/config-3.6m-darwin).  Fortunately, both test_sysconfig and test_distutils catch this:

======================================================================
FAIL: test_srcdir (test.test_sysconfig.TestSysConfig)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/py/dev/3x/root/uxd/lib/python3.6/test/test_sysconfig.py", line 346, in test_srcdir
    self.assertTrue(os.path.isdir(srcdir), srcdir)
AssertionError: False is not true : /py/dev/3x/root/uxd/lib/python3.6/config-3.6dm

======================================================================
FAIL: test_get_makefile_filename (test.test_sysconfig.MakefileTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/py/dev/3x/root/uxd/lib/python3.6/test/test_sysconfig.py", line 414, in test_get_makefile_filename
    self.assertTrue(os.path.isfile(makefile), makefile)
AssertionError: False is not true : /py/dev/3x/root/uxd/lib/python3.6/config-3.6dm/Makefile

======================================================================
FAIL: test_srcdir (distutils.tests.test_sysconfig.SysconfigTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/py/dev/3x/root/uxd/lib/python3.6/distutils/tests/test_sysconfig.py", line 61, in test_srcdir
    self.assertTrue(os.path.isdir(srcdir), srcdir)
AssertionError: False is not true : /py/dev/3x/root/uxd/lib/python3.6/config-3.6dm

----------------------------------------------------------------------

I'm not sure what the best solution is.
msg269237 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-06-25 13:24
The same problem occurs when cross-compiling for Android and running test_sysconfig and test_distutils.

Some other points:
* The generation of the platform directory breaks the support of read only source trees. The files of this  directory are not tracked by mercurial and printed by 'hg status'.

* The new files in the platform directory are regenerated from C headers of the build machine.
msg269240 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-06-25 14:55
> The same problem occurs when cross-compiling for Android and running test_sysconfig and test_distutils.

This is misleading. Actually, test_sysconfig and test_distutils do not fail when run from the source tree, but fail on an installed python, whatever the compilation mode: native or cross-compilation.
msg269360 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-06-27 08:28
Regeneration of the platdir files needs to be aware of the value of sysroot [1]. For example on Android cross-builds, sysroot must be set to the path of the headers and libraries for the targeted Android API level corresponding to a version of the Android libc and a version of Android. So, automatically regenerating those files in this case requires to look into the CC, CFLAGS and CPPFLAGS environment variables for sysroot. OTOH RTLD_* constants are exposed in the posix module (issue 13226) and the platdir files usefulness is limited now.

Cross building from one linux architecture to another is possible with gnu make VPATH support by building outside the source tree. Android is also a linux architecture and building outside the source tree makes sense, not only to work around the problem described in the second item of msg241143, but also to build for different Android API levels and identify the results of those builds.

It is not clear that the changes made in this issue fixes correctly issue 22724, see msg269359.

I think the scope of this isssue should be restricted to multiarch build systems, i.e. when gcc has been configured with '--enable-multiarch' [2].

[1] Options for Directory Search: https://gcc.gnu.org/onlinedocs/gcc/Directory-Options.html
[2] Installing GCC: Configuration: https://gcc.gnu.org/install/configure.html
msg271636 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-29 16:26
To reproduce the problem on any system where Python is built with configure:

$ mkdir -p tmp/build tmp/install
$ cd tmp
$ hg clone https://hg.python.org/cpython cpython
$ cd build
$ $(cd ../cpython && pwd)/configure --prefix=$(cd ../install && pwd) && make install
$ ../install/bin/python3 -m test -v test_sysconfig
[snip]
======================================================================
FAIL: test_srcdir (test.test_sysconfig.TestSysConfig)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/path_to/tmp/install/lib/python3.6/test/test_sysconfig.py", line 346, in test
_srcdir
    self.assertTrue(os.path.isdir(srcdir), srcdir)
AssertionError: False is not true : /path_to/tmp/install/lib/python3.6/config-3.6m

======================================================================
FAIL: test_get_makefile_filename (test.test_sysconfig.MakefileTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/path_to/tmp/install/lib/python3.6/test/test_sysconfig.py", line 414, in test
_get_makefile_filename
    self.assertTrue(os.path.isfile(makefile), makefile)
AssertionError: False is not true : /path_to/tmp/install/lib/python3.6/config-3.6m/Makefile

----------------------------------------------------------------------
Ran 21 tests in 0.063s

FAILED (failures=2, skipped=2)
test test_sysconfig failed
test_sysconfig failed
1 test failed:
    test_sysconfig
Total duration: 0:00:01
msg271690 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-30 14:50
This problem occured for the first time on the "installed Python" buildbot (setup by Zachary) at
http://buildbot.python.org/all/builders/x86%20Gentoo%20Installed%20with%20X%203.x/builds/747
on the day changeset 78d2cb7f66b6 has been commited (see msg268528).

Those "installed Python" buildbot are very helpful, thanks Zachary.
msg271691 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-30 14:54
The reason I would like this problem fixed is that the test suite is run on an installed Python when cross-compiling, obviously you cannot run the test suite with the cross-compiled Python on the source tree of the build system.
msg271692 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-30 15:00
It's not just broken for cross-compiling.  As I noted earlier, it's broken when using other than gcc, such as is the case with current OS X systems.  This needs to be fixed before release.
msg271693 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-30 15:06
It is also broken for gcc, see the buildbot 'compile' output.
It seems broken for all non windows builds whatever the compiler.
msg272790 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-15 18:41
New changeset ddc4bdae5e41 by Ned Deily in branch 'default':
Issue #23968: Make OS X installer build script aware of renamed platform
https://hg.python.org/cpython/rev/ddc4bdae5e41
msg273179 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-08-20 01:22
Any update on this issue?

test_srcdir() of test_sysconfig still fails on "x86 Gentoo Installed with X 3.x" buildbot. I opened the issue #27798 which was closed a duplicate of this issue.

First failure of the buildbot:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Installed%20with%20X%203.x/builds/747

This build contains many changes. I don't know which one introduced the issue?

* change a062c60c9f7d60db2b63af88e2d676e617398d01: Update distutils/sysconfig.py to look for the renamed _sysconfigdata module too
* change 78d2cb7f66b602ec496800fbbe71e590c0f56b00: Rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)

Ned Deily: "This needs to be fixed before release."

I agree. I use the release blocker priority as a reminder. Well, if the issue cannot be fixed, I suggest to revert changes related to this issue.

"It's not just broken for cross-compiling.  As I noted earlier, it's broken when using other than gcc, such as is the case with current OS X systems."

test_sysconfig fails on "x86 Gentoo Installed with X 3.x" buildbot which uses GCC. Extract of the configure output:

   checking for gcc... gcc
msg273182 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-08-20 01:56
please could somebody test the attached patch (I'm also trying to setup a gentoo env)? The idea is to keep the multiarch and triplet macros in sync.

diff -r a7f3678f9509 configure.ac
--- a/configure.ac	Sat Aug 20 03:05:13 2016 +0200
+++ b/configure.ac	Sat Aug 20 03:54:39 2016 +0200
@@ -882,6 +882,8 @@
   if test x$PLATFORM_TRIPLET != x$MULTIARCH; then
     AC_MSG_ERROR([internal configure error for the platform triplet, please file a bug report])
   fi
+elif test x$PLATFORM_TRIPLET != x && test x$MULTIARCH = x; then
+  MULTIARCH=$PLATFORM_TRIPLET
 fi
 if test x$PLATFORM_TRIPLET = x; then
   PLATDIR=plat-$MACHDEP
msg273185 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-08-20 04:18
I've just tried doko's patch on Arch Linux. Before this patch there are three test failures in test_distutils and test_sysconfig, just like Gentoo. After the patch, all tests in test_distutils and test_sysconfig pass.

On my PC both `gcc -print-multiarch` and `gcc --print-multiarch` print an empty string.

By the way, could you attach this patch file to roundup, so that others can download it?

Reference: Arch Linux's build script of python-hg: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-hg
msg273875 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-29 18:03
New changeset 5c77488830bc by doko in branch 'default':
- Issue #23968, keep platform_triplet and multiarch macros in sync
https://hg.python.org/cpython/rev/5c77488830bc
msg273876 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-08-29 18:04
now fixed.
msg273903 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-08-30 08:05
Changeset 5c77488830bc brings a regression - Android build fails with the following message:

export H2PY; H2PY="$PYTHON_FOR_BUILD /home/yen/python3-android/src/cpython/build-target/../Tools/scripts/h2py.py"; \
cd ../Lib/plat-aarch64-linux-gnu;  ./regen
+ _PYTHON_PROJECT_BASE=/home/yen/python3-android/src/cpython/build-target _PYTHON_HOST_PLATFORM=linux-aarch64 PYTHONPATH=/home/yen/python3-android/src/cpython/build-target/build/lib.linux-aarch64-3.6:../Lib:../Lib/plat-aarch64-linux-gnu python3.6 /home/yen/python3-android/src/cpython/build-target/../Tools/scripts/h2py.py -i '(u_long)' /opt/android-ndk/platforms/android-21/arch-arm64/usr/include/sys/types.h /opt/android-ndk/platforms/android-21/arch-arm64/usr/include/netinet/in.h /opt/android-ndk/platforms/android-21/arch-arm64/usr/include/dlfcn.h
./regen: line 33: _PYTHON_PROJECT_BASE=/home/yen/python3-android/src/cpython/build-target: No such file or directory
make[1]: *** [Makefile:1324: ../Lib/plat-aarch64-linux-gnu] Error 127
make[1]: Leaving directory '/home/yen/python3-android/src/cpython/build-target'
make: *** [Makefile:36: python] Error 2

The previous changeset 1902e1d79e25 builds fine. Prepending `PYTHON_FOR_BUILD` in configure.ac with `env` makes the whole build work again.

From the message seems plat-* directories are re-generated with a different name. Is it expected?
msg273904 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-08-30 08:20
Oh didn't see the title of this issue. Here's the patch for Android.
msg274042 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-08-31 17:23
I don't see how the suggested android fix is related, and what it is supposed to fix. It adds the env command before setting the environment variables, which should be a no-op.

However I think we should introduce different platform triplets for android.  Please follow-up in http://bugs.python.org/issue27917
msg274056 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-08-31 18:22
Well changes in this issue are not the root cause of my problem - it just exposes a broken implementation done long time ago. I'll examine more and open a new bug.
History
Date User Action Args
2016-08-31 18:22:20yan12125setmessages: + msg274056
2016-08-31 17:23:32dokosetmessages: + msg274042
2016-08-30 08:20:21yan12125setfiles: + issue23968-android-regression.patch

messages: + msg273904
2016-08-30 08:05:51yan12125setmessages: + msg273903
2016-08-29 18:04:43dokosetstatus: open -> closed
resolution: fixed
messages: + msg273876
2016-08-29 18:03:35python-devsetmessages: + msg273875
2016-08-20 04:18:40yan12125setmessages: + msg273185
2016-08-20 01:56:34dokosetmessages: + msg273182
2016-08-20 01:22:11vstinnersetpriority: critical -> release blocker
nosy: + vstinner
messages: + msg273179

2016-08-19 15:52:55zach.warelinkissue27798 superseder
2016-08-15 18:41:56python-devsetmessages: + msg272790
2016-08-13 08:22:01yan12125setnosy: + yan12125
2016-07-30 15:08:49koobssetnosy: + koobs
2016-07-30 15:06:02xdegayesetmessages: + msg271693
2016-07-30 15:00:46ned.deilysetmessages: + msg271692
2016-07-30 14:54:43xdegayesetmessages: + msg271691
2016-07-30 14:50:42xdegayesetnosy: + zach.ware
messages: + msg271690
2016-07-29 16:26:50xdegayesetmessages: + msg271636
2016-07-29 10:28:10xdegayesetnosy: + thomas-petazzoni
2016-06-27 08:28:25xdegayesetmessages: + msg269360
2016-06-25 14:55:34xdegayesetmessages: + msg269240
2016-06-25 13:24:34xdegayesetmessages: + msg269237
2016-06-25 12:55:48xdegayelinkissue26865 dependencies
2016-06-18 20:50:56ned.deilysetpriority: normal -> critical
stage: needs patch
versions: + Python 3.6, - Python 3.5
2016-06-18 20:47:37ned.deilysetmessages: + msg268833
2016-06-18 19:59:23python-devsetmessages: + msg268829
2016-06-14 07:22:55python-devsetmessages: + msg268536
2016-06-14 06:57:27dokosetmessages: + msg268530
2016-06-14 06:55:33python-devsetnosy: + python-dev
messages: + msg268528
2016-06-05 00:34:37ned.deilysetnosy: + ned.deily
2016-06-05 00:22:38ncoghlansetnosy: + xdegaye
messages: + msg267352
2016-06-05 00:06:33freakboy3742setnosy: + freakboy3742
2016-06-05 00:00:15ncoghlanlinkissue18083 dependencies
2016-06-04 23:59:43ncoghlanlinkissue22724 dependencies
2016-06-04 23:13:28dokosetfiles: + platdir2.diff
keywords: + patch
messages: + msg267342
2015-08-11 18:48:32wscullinsetnosy: + wscullin
messages: + msg248423
2015-06-01 18:24:20Arfreversetnosy: + Arfrever
messages: + msg244605
2015-05-29 20:09:36barrysetnosy: + barry
messages: + msg244411
2015-05-29 09:14:02sYnfosetmessages: + msg244362
2015-05-28 09:07:54sYnfosetnosy: + sYnfo
2015-05-23 08:17:59ncoghlansetnosy: + larry, rkuska
messages: + msg243896
2015-05-23 00:26:10dokosetkeywords: + needs review, - patch

messages: + msg243874
2015-04-15 19:04:07dokocreate