classification
Title: pip cannot build extensions for debug Python
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Ivan.Pozdeev, dstufft, eric.araujo, steve.dower
Priority: normal Keywords:

Created on 2018-05-24 17:10 by Ivan.Pozdeev, last changed 2021-02-03 18:19 by steve.dower. This issue is now closed.

Messages (3)
msg317589 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2018-05-24 17:10
(the output is for 2.7 head; for master, it's the same save for paths, library versions and DeprecationWarning's)

>python.bat -m ensurepip
Running Debug|x64 interpreter...
Looking in links: c:\users\sasha\appdata\local\temp\tmpiysoby
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-10.0.1 setuptools-39.0.1
>python.bat -m pip install scandir
Running Debug|x64 interpreter...
Collecting scandir
  Using cached https://files.pythonhosted.org/packages/13/bb/e541b74230bbf7a20a3949a2ee6631be299378a784f5445aa5d0047c192b/scandir-1.7.tar.gz
Installing collected packages: scandir
  Running setup.py install for scandir ... error
    Complete output from command C:\Users\Sasha\Documents\cpython\PCBuild\amd64\python_d.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\sasha\\appdata\\local\\temp\\pip-install-xafnbn\\scandir\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\sasha\appdata\local\temp\pip-record-gbe0kk\install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-2.7-pydebug
    copying scandir.py -> build\lib.win-amd64-2.7-pydebug
    running build_ext
    building '_scandir' extension
    creating build\temp.win-amd64-2.7-pydebug
    creating build\temp.win-amd64-2.7-pydebug\Release
    c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Users\Sasha\Documents\cpython\include -IC:\Users\Sasha\Documents\cpython\PC /Tc_scandir.c /Fobuild\temp.win-amd64-2.7-pydebug\Release\_scandir.obj
    _scandir.c
    c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Users\Sasha\Documents\cpython\libs /LIBPATH:C:\Users\Sasha\Documents\cpython\PCbuild\amd64 /LIBPATH:C:\Users\Sasha\Documents\cpython\PC\VS9.0\amd64 /EXPORT:init_scandir build\temp.win-amd64-2.7-pydebug\Release\_scandir.obj /OUT:build\lib.win-amd64-2.7-pydebug\_scandir.pyd /IMPLIB:build\temp.win-amd64-2.7-pydebug\Release\_scandir.lib /MANIFESTFILE:build\temp.win-amd64-2.7-pydebug\Release\_scandir.pyd.manifest
    LINK : fatal error LNK1104: cannot open file 'python27.lib'
    error: command 'c:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\BIN\\amd64\\link.exe' failed with exit status 1104

    ----------------------------------------
Command "C:\Users\Sasha\Documents\cpython\PCBuild\amd64\python_d.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\sasha\\appdata\\local\\temp\\pip-install-xafnbn\\scandir\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\sasha\appdata\local\temp\pip-record-gbe0kk\install-record.txt --single-version-externally-managed --compile" failed with error code
1 in c:\users\sasha\appdata\local\temp\pip-install-xafnbn\scandir\



The cause is that distutils' `build_ext` looks at the --debug switch instead of the running Python to decide which Python runtime to link against.

A workaround is thus to run setup.py by hand with the --debug switch.


Suggested fix is to split compiler options into optimization options and linking options and:

* for debug Python, build without optimizations and against debug C runtime + Python runtime regardless of --debug
* for release Python,
    * build without optimizations but against release C runtime + Python runtime with --debug


Rationale:

* It doesn't make sense to try to build against a different runtime than the running Python -- this will always fail because include/lib paths, preprocessor directives and such that build_ext generates is for the running Python.
* Must build against the same C runtime that Python uses in any case since using two different runtimes will break things, especially in 2.7 that relies on stdio.
* Building an extension without optimizations will allow to diagnose problems in it. Debug facilities of the runtime like malloc tracking and probably tracing will not work, but at least one will see all the variables and call hierarchy.
msg317600 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2018-05-24 18:24
>one will see all the variables and call hierarchy

and be able to step through the code. With optimizations, the current line jumps unpredictably, and some lines can't be visited because they're optimized away.
msg386336 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-03 18:19
Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools
History
Date User Action Args
2021-02-03 18:19:27steve.dowersetstatus: open -> closed

nosy: + steve.dower
messages: + msg386336

resolution: out of date
stage: resolved
2018-05-24 18:24:32Ivan.Pozdeevsetmessages: + msg317600
2018-05-24 17:10:47Ivan.Pozdeevcreate