classification
Title: 'make install' fails when the configure 'prefix' is '/' and DESTDIR is used
Type: behavior Stage: patch review
Components: Cross-Build Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alex.Willmer, martin.panter, pitrou
Priority: normal Keywords: patch

Created on 2017-08-03 17:40 by xdegaye, last changed 2019-12-10 12:01 by xdegaye.

Pull Requests
URL Status Linked Edit
PR 4172 closed xdegaye, 2017-10-30 10:36
Messages (5)
msg299718 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-08-03 17:40
The error messages:
    running install_lib
    creating /lib/python3.7
    error: could not create '/lib/python3.7': Permission denied
    make[1]: *** [Makefile:1449: sharedinstall] Error 1

The command that triggers this failure:
    _PYTHON_PROJECT_BASE=/tmp/android-makesetup/build/python3.7-android-21-armv7 _PYTHON_HOST_PLATFORM=linux-arm PYTHONPATH=/tmp/android-makesetup/build/python3.7-android-21-armv7/build/lib.linux-arm-3.7:/src/python/master/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_ python /src/python/master/setup.py install \
      --prefix=/ \
      --install-scripts=//bin \
      --install-platlib=//lib/python3.7/lib-dynload \
      --root=/tmp/android-makesetup/build/python3.7-install-android-21-armv7/

Note how some paths in the setup.py options start with two slashes.

The problem is that change_root() in Lib/distutils/util.py cannot concatenate 'root' with 'pathname' as 'pathname' starts with two slashes and therefore cannot be made relative by just removing one slash.  The behavior of change_root() is correct (although it could be made more robust by calling os.path.normpath() first before removing the first slash) because according to POSIX a pathname starting with two slashes is implementation defined (but not when starting with three or more slashes).  And normpath respects this rule:
    >>> from os.path import normpath
    >>> normpath('//tmp')
    '//tmp'
    >>> normpath('///tmp')
    '/tmp'
However for most posix systems (possibly except Cygwin ?), a path starting with two slashes is the same path as the path starting with one slash.

The problem lies with the Makefile. A workaround for the user is to set the --exec-prefix option to '/./' instead of '/' when running configure and this fixes the problem, the installation is successfull. So a fix could be to swap '/' for '/./' in configure itself.
Maybe there is a better solution.

References:
issue 1676135:
    The changes made by this issue have excluded the case of '/' as there was a concern at that time that the sys.prefix value would be wrong.  I have checked that sys.prefix is correct when '/' is changed to '/./' (on configure command line). This is because the determination of sys.prefix made by Lib/site.py is quite different now from what it was 10 years ago.

issue 9674:
    This is exactly the same issue as this one.  I have preferred to open a new issue here because the discussion on issue 9674 is focused on making changes to distutils while the problem is actually in the posixly incorrect use of paths starting with double slashes in the Makefile.
msg299779 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-08-05 13:08
In the subprocess test named test_executable_without_cwd and when the test is run on the installed Python, argv[0] is not the python executable and calculate_path() in Modules/getpath.c, as a last resort, searches the directories pointed to by the preprocessor variables PREFIX and EXEC_PREFIX defined in the Makefile. So this test indeed would have failed with the (rejected) proposition made in issue 1676135 to substitute the prefix '/' for '' in the configure script.  But the test would succeed when the substitution replaces '/' with '/./' as proposed here (after Python has effectively been manually installed on '/' of course).

Actually using '/.' instead of '/./' is better since the configure script will remove the trailing slash anyway. In both cases the value of prefix in the sysconfig module is '/.'.
msg305218 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-10-30 10:48
To test PR 4172 on linux one needs a chroot. The following test is run on archlinux using systemd-nspawn (a chroot on steroids, see https://wiki.archlinux.org/index.php/Systemd-nspawn).

1) Setup systemd-nspawn by installing arch-install-scripts and by using pacstrap to install the archlinux 'base' package group in the /tmp/chroot container, this installs about 600M of archlinux packages in the chroot:
  $ mkdir /tmp/chroot
  $ pacman -S arch-install-scripts
  # pacstrap -i -c -d /tmp/chroot base --ignore linux

2) Build and install Python in the /tmp/python-build DESTDIR directory:
  $ mkdir /tmp/python-build
  $ ./configure --prefix=/ --without-ensurepip && make
  $ make DESTDIR=/tmp/python-build install

3) Copy the Python distribution to the chroot directory (note the trailing slash in the source directory):
  # rsync -av --keep-dirlinks /tmp/python-build/ /tmp/chroot

4) Boot into the chroot container and run python:
  # systemd-nspawn -b -D /tmp/chroot
  ...
  [  OK  ] Started Console Getty.
  [  OK  ] Reached target Login Prompts.
  [  OK  ] Started Login Service.
  [  OK  ] Reached target Multi-User System.
  [  OK  ] Reached target Graphical Interface.
  [  OK  ] Started Rotate log files.

  Arch Linux 4.13.4-1-ARCH (console)

  chroot login: root
  Last login: Mon Oct 30 10:54:11 on console
  [root@chroot ~]# python3.7
  Python 3.7.0a2+ (heads/bpo-31114:ffa5bff252, Oct 30 2017, 11:24:54)
  [GCC 7.2.0] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import abc, _bisect, sysconfig, sys
  >>> sys.prefix
  '/usr'
  >>> sysconfig.get_config_var('prefix')
  '/.'
  >>> sys.exit(0)
  [root@chroot ~]# poweroff
  ...
  [  OK  ] Reached target Shutdown.
  Container chroot has been shut down.
msg305221 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-10-30 11:06
Clarification about the test results:
On archlinux /bin is a symlink to /usr/bin, so on archlinux the value of sys.prefix is '/usr' instead of the expected '/'. This is because in Modules/getpath.c, 'argv0_path' is the directory of the link target when the executable location is a symbolic link.
msg305248 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-10-30 19:56
Actually, sys.prefix is '/usr' because '/usr/bin' is in PATH while '/bin' is not in PATH. After adding '/bin' to the start of PATH, then sys.prefix becomes '/.' when Python is run as 'python3'. The confusion comes from the '/bin' symlink (archlinux is not a perfect choice for those tests). Whatever the value of PATH, '/bin/python3' has also '/.' as its sys.prefix.

So Modules/getpath.c fails to get '/' as the prefix in step 3 of the algorithm described at the top of the source code and uses '/.' (i.e. PREFIX) instead. This is caused by its reduce() function not handling the corner case where Python is installed on '/'. I will update the PR shortly.
History
Date User Action Args
2019-12-10 12:01:16xdegayesetnosy: - xdegaye
2017-10-30 19:56:20xdegayesetmessages: + msg305248
2017-10-30 11:06:41xdegayesetmessages: + msg305221
2017-10-30 10:48:44xdegayesetnosy: + pitrou, martin.panter
messages: + msg305218
2017-10-30 10:36:52xdegayesetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request4141
2017-10-30 09:37:33xdegayesetnosy: + Alex.Willmer
title: 'make install' fails when exec_prefix is '/' and DESTDIR not empty -> 'make install' fails when the configure 'prefix' is '/' and DESTDIR is used
versions: - Python 3.5
components: + Cross-Build, - Build
stage: needs patch
2017-08-05 13:08:49xdegayesetmessages: + msg299779
2017-08-03 17:40:40xdegayecreate