classification
Title: CPython may fail to build in the presence of a ~/.pydistutils.cfg
Type: Stage: resolved
Components: Build, Distutils Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: dstufft, eric.araujo, iritkatriel
Priority: normal Keywords:

Created on 2018-08-12 19:32 by Antony.Lee, last changed 2021-06-23 11:09 by Antony.Lee.

Messages (3)
msg323452 - (view) Author: Antony Lee (Antony.Lee) * Date: 2018-08-12 19:32
I have a ~/.pydistutils.cfg with the following contents:

    [build_ext]
    force = true
    inplace = true

(--force is useful because sometimes just comparing timestamps is insufficient to know whether a package needs to be rebuilt, e.g. in the presence of external dependencies -- and I use ccache anyways to avoid paying for excessive rebuilds; --inplace is useful as I have quite a few packages with extension modules that are editably-installed).

With this ~/.pydistutils.cfg, cpython fails to build.  For example, having checked out v3.7.0 from a git clone (otherwise clean git repo, per `git clean -xfd`),

    mkdir build && cd build && ../configure && make

ultimately results in

    <elided>

    gcc -pthread -shared build/temp.linux-x86_64-3.7/home/antony/src/extern/cpython/Modules/_ctypes/_ctypes.o build/temp.linux-x86_64-3.7/home/antony/src/extern/cpython/Modules/_ctypes/callbacks.o build/temp.linux-x86_64-3.7/home/antony/src/extern/cpython/Modules/_ctypes/callproc.o build/temp.linux-x86_64-3.7/home/antony/src/extern/cpython/Modules/_ctypes/stgdict.o build/temp.linux-x86_64-3.7/home/antony/src/extern/cpython/Modules/_ctypes/cfield.o -L/usr/local/lib -lffi -ldl -o /home/antony/src/extern/cpython/build/_ctypes.cpython-37m-x86_64-linux-gnu.so
    *** WARNING: renaming "_struct" since importing it failed: build/lib.linux-x86_64-3.7/_struct.cpython-37m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory
    Traceback (most recent call last):
    File "../setup.py", line 442, in check_extension_import
        importlib._bootstrap._load(spec)
    File "<frozen importlib._bootstrap>", line 696, in _load
    File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 583, in module_from_spec
    File "<frozen importlib._bootstrap_external>", line 1043, in create_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    ImportError: build/lib.linux-x86_64-3.7/_struct.cpython-37m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "../setup.py", line 2363, in <module>
        main()
    File "../setup.py", line 2358, in main
        "Tools/scripts/2to3", "Tools/scripts/pyvenv"]
    File "/home/antony/src/extern/cpython/Lib/distutils/core.py", line 148, in setup
        dist.run_commands()
    File "/home/antony/src/extern/cpython/Lib/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
    File "/home/antony/src/extern/cpython/Lib/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
    File "/home/antony/src/extern/cpython/Lib/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
    File "/home/antony/src/extern/cpython/Lib/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
    File "/home/antony/src/extern/cpython/Lib/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
    File "/home/antony/src/extern/cpython/Lib/distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
    File "../setup.py", line 308, in build_extensions
        self.check_extension_import(ext)
    File "../setup.py", line 447, in check_extension_import
        assert not self.inplace
    AssertionError
    make: *** [Makefile:618: sharedmods] Error 1

Removing the ~/.pydistutils.cfg fixes the issue and leads to a successful build.

I think(?) CPython's build system should essentially make sure that distutils behaves as if `--no-user-cfg` was in effect.  Or at least the limitation should be documented, but it's not very practical to have to temporarily rename an existing ~/.pydistutils.cfg whenever building CPython.  See also https://bugs.python.org/issue9309, perhaps.
msg396370 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-06-22 22:20
Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools
msg396371 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2021-06-22 22:27
This one is about Python’s build itself, that still uses parts of distutils, so probably should stay open if reproducible.
History
Date User Action Args
2021-06-23 11:09:04Antony.Leesetnosy: - Antony.Lee
2021-06-22 22:30:59iritkatrielsetstatus: closed -> open
resolution: out of date ->
2021-06-22 22:27:04eric.araujosetmessages: + msg396371
2021-06-22 22:20:43iritkatrielsetstatus: open -> closed

components: + Distutils

nosy: + iritkatriel, eric.araujo, dstufft
messages: + msg396370
resolution: out of date
stage: resolved
2018-08-12 19:32:17Antony.Leecreate