classification
Title: Remove the concept of platform-specific directories
Type: enhancement Stage: resolved
Components: Build, Cross-Build Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Alex.Willmer, doko, ned.deily, python-dev, vstinner, xdegaye, yan12125, zach.ware
Priority: normal Keywords: patch

Created on 2016-09-09 17:58 by zach.ware, last changed 2018-01-24 16:20 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
remove_plat.diff zach.ware, 2016-09-09 17:58 review
get_sysconfigdata_name.diff xdegaye, 2016-09-10 13:51 review
sysconfigdata_env_var.diff xdegaye, 2016-09-10 18:12 review
sysconfigdata_ABIFLAGS.patch xdegaye, 2016-10-26 19:05 review
Pull Requests
URL Status Linked Edit
PR 5289 merged vstinner, 2018-01-24 00:26
Messages (35)
msg275360 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-09 17:58
As a follow-on to #28027, the attached patch removes the concept of platform-specific directories (Lib/plat-darwin, Lib/plat-x86_64-linux-gnu, etc).  I believe this should be fine for cross-builds (at least as fine as the previous solution for _sysconfigdata), but I'd like someone to test it for me since I have no experience with cross-builds.
msg275363 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-09 18:01
You might also remove  Lib/_sysconfigdata.py from .gitignore.
msg275427 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-09-09 20:35
not sure if I fully understand:

 - The patch encodes the multiarch triplet into the _sysconfigdata
   name and installs it into the standard lib dir.
 - It removes the PLATDIR macro
 - what happens to the constants modules which are currently
   found in the platdir? These vary across architectures and
   should not be install in the standard lib dir.
msg275429 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-09 20:36
Those were removed in #28027, by BDFL decree.
msg275430 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-09-09 20:40
this comes as a surprise. I can't remember that discussion and decision.
msg275433 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-09 20:47
Sorry, it happened at the sprint at Instagram.  Do you have a use for those modules that we need to reconsider?
msg275436 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-09 20:50
> this comes as a surprise. I can't remember that discussion and decision.

Removing these modules is not a new idea: I just posted a message to the issue #28027 to complete the history of this change.
msg275438 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-09-09 20:59
The removal is not following the guidelines to first deprecate these for a release, and then to remove them.  I know that at least the RTLD_* constants are used in a number of places, which will just stop working.

I'm not against removing these in 3.7, however they should be deprecated in 3.6 first. There is no harm done with this approach.
msg275442 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-09 21:06
Matthias Klose added the comment:
> The removal is not following the guidelines to first deprecate these for a release, and then to remove them.  I know that at least the RTLD_* constants are used in a number of places, which will just stop working.

Please continue the discussion in the issue #28027, reopen it if you want.
msg275475 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-09 22:29
Xavier: Could you please make sure this doesn't break anything for Android/cross-builds?
msg275477 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-09-09 22:35
Sure, I was already planning to do it :)
msg275518 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-10 01:00
New changeset 90396ec9a2f8 by Zachary Ware in branch 'default':
Issue #28046: Remove platform-specific directories from sys.path
https://hg.python.org/cpython/rev/90396ec9a2f8
msg275519 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-10 01:01
Xavier: I'm still interested in hearing whether this breaks anything for you, but I went ahead and pushed it.
msg275532 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-10 01:36
New changeset 2bfe63a3eb5c by Zachary Ware in branch 'default':
Issue #28046: Fix distutils
https://hg.python.org/cpython/rev/2bfe63a3eb5c
msg275592 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-10 07:36
Seems after this changeset Python is broken:

shell@ASUS_Z00E_2:/data/local/tmp/python3 $ python3.6
Failed to import the site module
Traceback (most recent call last):
  File "/data/local/tmp/python3/lib/python3.6/site.py", line 549, in <module>
    main()
  File "/data/local/tmp/python3/lib/python3.6/site.py", line 536, in main
    known_paths = addusersitepackages(known_paths)
  File "/data/local/tmp/python3/lib/python3.6/site.py", line 281, in addusersitepackages
    user_site = getusersitepackages()
  File "/data/local/tmp/python3/lib/python3.6/site.py", line 257, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/data/local/tmp/python3/lib/python3.6/site.py", line 247, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/data/local/tmp/python3/lib/python3.6/sysconfig.py", line 600, in get_config_var
    return get_config_vars().get(name)
  File "/data/local/tmp/python3/lib/python3.6/sysconfig.py", line 549, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/data/local/tmp/python3/lib/python3.6/sysconfig.py", line 420, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata_m_linux_aarch64-linux-android'

sysconfig data is installed as lib-dynload/_sysconfigdata_m_linux_x86_64-linux-gnu.py. It should be at lib-dynload/_sysconfigdata_m_linux_aarch64-linux-android.py?

PS: I'm building for Android AArch64 on Linux x86_64
msg275599 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-10 08:19
Got it. When calling `python -m sysconfig --generate-posix-vars`, the host (Linux x86_64) Python is used, so _get_sysconfigdata_name() returns the name for Linux instead of for Android. Maybe _get_sysconfigdata_name() should check whether it's used for cross-compiling or not.
msg275635 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-09-10 13:51
Confirming the problem reported by Chi Hsuan Yen. The attached patch fixes this.

Another problem is that the shared libraries names of the extension modules are now suffixed with the wrong triplet, i.e. with the build system triplet instead of the target host triplet. So they cannot be imported. Not sure where this regression has been introduced.
msg275638 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-10 14:06
Xavier, that change looks good to me, please commit it.
msg275642 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-10 14:20
New changeset 1d71ce4531ac by Xavier de Gaye in branch 'default':
Issue #28046: Fix get_sysconfigdata_name().
https://hg.python.org/cpython/rev/1d71ce4531ac
msg275643 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-10 14:21
With this patch no extensions in Modules/ can be built. Let me check whether there's a bug in my build script.
msg275650 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-10 16:11
Hmm things are quite complicated. Brief: the build is broken for out-of-source cross-compiling if the host Python is an in-source build.

(Below $build_dir refers to the directory that invokes $source_dir/configure)

In an out-of-source build, setup.py relies on sysconfig.get_config_var('srcdir') to get correct filenames. In _init_posix(), _get_sysconfigdata_name() still returns an incorrect name (for example, _sysconfigdata_m_linux_x86_64-linux-gnu, on Linux x86_64). There's no _sysconfigdata_m_linux_x86_64-linux-gnu.py in $build_dir/build/lib.linux-aarch64-3.6, so $PYTHON_FOR_BUILD imports the one from the host python, which has srcdir == '.' As a result, build_ext can't find source files:

building 'xxlimited' extension
/home/yen/Projects/python3-android/clang-bin/cc -fPIC -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -g -fvar-tracking-assignments -g -fvar-tracking-assignments -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -g -fvar-tracking-assignments -g -fvar-tracking-assignments -std=c99 -DPy_LIMITED_API=0x03050000 -I./Include -I. -IInclude -I/usr/include -I/usr/local/include -I/home/yen/Projects/python3-android/src/cpython/build-target/Include -I/home/yen/Projects/python3-android/src/cpython/build-target -c xxlimited.c -o build/temp.linux-aarch64-3.6/xxlimited.o
clang: error: unknown argument: '-fvar-tracking-assignments'
clang: error: unknown argument: '-fvar-tracking-assignments'
clang: error: unknown argument: '-fvar-tracking-assignments'
clang: error: unknown argument: '-fvar-tracking-assignments'
clang: error: no such file or directory: 'xxlimited.c'
clang: error: no input files

Ironically, before 1d71ce4531ac, $build_dir/build/lib.linux-aarch64-3.6 has _sysconfigdata_m_linux_x86_64-linux-gnu.py, so building is OK.

Again, I don't know to fix it :(

[1] https://hg.python.org/cpython/file/1d71ce4531ac/setup.py#l218
msg275664 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-09-10 18:12
With the attached patch, the python test suite runs smoothly on android and without errors on linux. On android, there are two new failed test cases in test_sysconfig that I will look into later (in the frame of this issue, I guess ?), but otherwise about the same tests fail that were failing before.

Here are some details about the fix:

> Another problem is that the shared libraries names of the extension modules are now suffixed with the wrong triplet.

This is because _init_posix() in Lib/distutils/sysconfig.py still imports the native python sysconfigdata module instead of the newly built one for the target host.

After _init_posix() is fixed, another problem occurs: extension modules are now built by including /usr/include from the native system and this leads to some unresolved dlopen references at run time on android. The reason is that add_gcc_paths() in setup.py calls sysconfig.get_config_var('CC') and this returns the native compiler instead of the cross-compiler.

So a solution is to set the _SYSCONFIGDATA_NAME environment variable in PYTHON_FOR_BUILD.
msg275667 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-10 18:23
LGTM, thank you Xavier!
msg275677 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-10 19:12
Much thanks! It's now building fine and running fine with my configuration. The test suite is still running and I believe there won't be more surprise than failed tests.
msg275711 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-09-10 23:06
One of the test_sysconfig failed test is caused by the changes made in issue 27917. I will fix it there. The other one is caused by a change in my android build setup that does not install anymore include/python3.6m/pyconfig.h, this is fixed now and the test does not fail anymore.

Pending code reviews, I will commit the patch after changing the env var name from _SYSCONFIGDATA_NAME to _PYTHON_SYSCONFIGDATA_NAME.
msg275845 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-11 20:22
New changeset 04b679dd11eb by Xavier de Gaye in branch 'default':
Issue #28046: get_sysconfigdata_name() uses the _PYTHON_SYSCONFIGDATA_NAME
https://hg.python.org/cpython/rev/04b679dd11eb
msg275846 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-09-11 20:25
Thanks for your help in testing these changes Chi Hsuan Yen.
msg275850 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-11 20:32
No No, it's me who should say thank you. Thanks for all efforts on maintaining the Android port of CPython!
msg276245 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-09-13 10:54
I don't like the _PYTHON_SYSCONFIGDATA_NAME hack.  This should be based on the target, not on a name for just a particular file.  Following up on that in #28125.
msg279508 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-26 19:05
'make install' fails to remove the sysconfigdata module from lib-dynload and prints now instead:

    rm: cannot remove '/path/to/install/lib/python3.7/lib-dynload/_sysconfigdata_m.py': No such file or directory

The patch fixes this. It also removes a useless and now incorrect line that was meant to echo the previously executed command.
msg279592 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-28 09:41
New changeset 86577b7100a4 by Xavier de Gaye in branch '3.6':
Issue #28046: Fix the removal of the sysconfigdata module
https://hg.python.org/cpython/rev/86577b7100a4

New changeset 28205c4cf20b by Xavier de Gaye in branch 'default':
Issue #28046: Merge with 3.6.
https://hg.python.org/cpython/rev/28205c4cf20b
msg303426 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-09-30 20:19
Are there any remaining outstanding issues here?
msg310539 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-24 00:27
The cleanup is not complete, so I reopen the issue: https://github.com/python/cpython/pull/5289
msg310605 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-24 16:02
New changeset 5de15f1d5ff09085620f63f0597d0920d75be719 by Victor Stinner in branch 'master':
bpo-28046: Remove MACHDEPPATH from Modules/Setup.dist (#5289)
https://github.com/python/cpython/commit/5de15f1d5ff09085620f63f0597d0920d75be719
msg310608 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-24 16:20
Fixed. I close again the issue. I don't want to backport this change, it doesn't hurt Python 3.6.
History
Date User Action Args
2018-01-24 16:20:41vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg310608
2018-01-24 16:02:48vstinnersetmessages: + msg310605
2018-01-24 00:27:40vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg310539
2018-01-24 00:26:50vstinnersetpull_requests: + pull_request5135
2017-11-26 20:27:58zach.waresetstatus: pending -> closed
resolution: fixed
stage: commit review -> resolved
2017-09-30 20:19:28zach.waresetstatus: open -> pending
assignee: zach.ware ->
messages: + msg303426
2016-10-28 09:41:35python-devsetmessages: + msg279592
2016-10-26 19:05:40xdegayesetfiles: + sysconfigdata_ABIFLAGS.patch

messages: + msg279508
versions: + Python 3.7
2016-09-13 10:54:46dokosetmessages: + msg276245
2016-09-11 20:32:37yan12125setmessages: + msg275850
2016-09-11 20:25:23xdegayesetmessages: + msg275846
2016-09-11 20:22:57python-devsetmessages: + msg275845
2016-09-10 23:06:15xdegayesetmessages: + msg275711
stage: patch review -> commit review
2016-09-10 19:12:01yan12125setmessages: + msg275677
2016-09-10 18:23:41zach.waresetmessages: + msg275667
2016-09-10 18:12:37xdegayesetfiles: + sysconfigdata_env_var.diff

messages: + msg275664
2016-09-10 16:11:58yan12125setmessages: + msg275650
2016-09-10 14:21:14yan12125setmessages: + msg275643
2016-09-10 14:20:32python-devsetmessages: + msg275642
2016-09-10 14:06:21zach.waresetmessages: + msg275638
2016-09-10 13:51:23xdegayesetfiles: + get_sysconfigdata_name.diff

messages: + msg275635
2016-09-10 08:19:40yan12125setmessages: + msg275599
2016-09-10 07:36:12yan12125setnosy: + yan12125
messages: + msg275592
2016-09-10 01:36:42python-devsetmessages: + msg275532
2016-09-10 01:01:38zach.waresetmessages: + msg275519
2016-09-10 01:00:06python-devsetnosy: + python-dev
messages: + msg275518
2016-09-09 22:35:07xdegayesetmessages: + msg275477
2016-09-09 22:29:11zach.waresetmessages: + msg275475
2016-09-09 21:06:34vstinnersetmessages: + msg275442
2016-09-09 20:59:35dokosetmessages: + msg275438
2016-09-09 20:50:47vstinnersetmessages: + msg275436
2016-09-09 20:47:42zach.waresetmessages: + msg275433
2016-09-09 20:40:19dokosetmessages: + msg275430
2016-09-09 20:36:47zach.waresetmessages: + msg275429
2016-09-09 20:35:14dokosetmessages: + msg275427
2016-09-09 18:01:06vstinnersetnosy: + vstinner
messages: + msg275363
2016-09-09 17:58:33zach.warecreate