classification
Title: distutils build_ext.get_outputs returns wrong result
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: ajaksu2, alexis, benjamin.peterson, eric.araujo, kxroberto, rpetrov, steve.dower, tarek
Priority: normal Keywords: patch

Created on 2009-05-09 13:29 by kxroberto, last changed 2021-02-03 18:10 by steve.dower. This issue is now closed.

Files
File name Uploaded Description Edit
ext_filename.patch kxroberto, 2009-05-09 13:29
build_ext._compiler.diff ajaksu2, 2009-05-12 18:46 Revert build_ext.compiler -> build_ext._compiler review
issue5977-w32.patch rpetrov, 2009-05-14 20:11
Messages (16)
msg87494 - (view) Author: kxroberto (kxroberto) Date: 2009-05-09 13:29
with --inplace etc. build_ext.get_outputs returns wrong extension paths;
tries to computes them out of the air *again* and ..

Tools which need to further know the produced module file, e.g.
pyximport (Cython) do crash then ...

patch see attachment: stores and uses the real path in use.
writes a new attribute ext_filename to Extension object - should it be
underscored _ext_filename or ... ?
msg87495 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-05-09 13:36
can you provide a test case that demonstrates the problem ?
msg87506 - (view) Author: kxroberto (kxroberto) Date: 2009-05-09 19:15
>>> test_build_ext: The test must be run in a python build dir

don't have a build setup currently. maybe in future.

One can yet see in build_ext.py:

after "if self.inplace:" (line485)
there are 2 different computations for ext_filename

while computation in get_outputs() is not aware of self.inplace (line447
msg87524 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-05-10 12:26
I was talking about your use case (pyximport (Cython) do crash then ?)

If I can have a real example, with the output and the expected one,
I will build a test with it in Distutils before I fix it
msg87525 - (view) Author: kxroberto (kxroberto) Date: 2009-05-10 14:31
its with any .pyx (Cython) module , when after pyximport.install() and
injection of --inplace option a .pyx module is imported.

the relevant lines in pyxbuild.py are

        dist.run_commands()
        return dist.get_command_obj("build_ext").get_outputs()[0]

which use the buggy "get_outputs" and shall return the full path of
built module to pyximport.py :

    so_path = pyxbuild.pyx_to_dll(pyxfilename, extension_mod,
                                  build_in_temp=build_in_temp,
                                  pyxbuild_dir=pyxbuild_dir,
                                  setup_args=sargs )
    assert os.path.exists(so_path), "Cannot find: %s" % so_path

=> crash with "Cannot find..." before pyximport.load_module goes to
import it.

-

A stripped down test case should perhaps 'build_ext' any arbitrary
extension module with --inplace ( a option of base command 'build' ) and
something like ...

dist.get_command_obj("build_ext").inplace=1
dist.run_commands()
so_path = dist.get_command_obj("build_ext").get_outputs()[0]
assert os.path.isfile(so_path) and os.path.dirname(so_path) in ('','.')

... will produce a invalid so_path: not pointing to actual locally
in-place built xy.pyd/.so, but to a non-existing or old file in the
build folders
msg87643 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-05-12 17:17
done in r72585 and propagated in 2.6, 3.0, 3.1 branches

Thank you !
msg87654 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2009-05-12 18:46
Tarek,
It looks like all buildbots are red because setup.py doesn't work with
this change:

running build
running build_ext
Traceback (most recent call last):
  File "./setup.py", line 1896, in <module>
    main()
  File "./setup.py", line 1891, in main
    'Lib/smtpd.py']
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/core.py",
line 149, in setup
    dist.run_commands()
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/dist.py",
line 974, in run_commands
    self.run_command(cmd)
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/dist.py",
line 994, in run_command
    cmd_obj.run()
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/command/build.py",
line 132, in run
    self.run_command(cmd_name)
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/cmd.py",
line 326, in run_command
    self.distribution.run_command(command)
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/dist.py",
line 994, in run_command
    cmd_obj.run()
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/command/build_ext.py",
line 348, in run
    self.build_extensions()
  File "./setup.py", line 103, in build_extensions
    missing = self.detect_modules()
  File "./setup.py", line 302, in detect_modules
    add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib')
AttributeError: 'NoneType' object has no attribute 'library_dirs'
[31295 refs]
*** Error code 1
The following command caused the error:
case $MAKEFLAGS in \
*s*)  CC='gcc' LDSHARED='gcc -shared' OPT='-g -Wall -Wstrict-prototypes'
./python -E ./setup.py -q build;; \
*)  CC='gcc' LDSHARED='gcc -shared' OPT='-g -Wall -Wstrict-prototypes'
./python -E ./setup.py build;; \
esac
make: Fatal error: Command failed for target `sharedmods'
program finished with exit code 1
msg87662 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-05-12 22:09
Sorry about that, I was travelling and didn't see that the buildbots
turned red. (I ran the test but didn't build python again)

I have changed that because build_ext.compiler is a string option, e.g.
the compiler type. Turning it into a compiler instance when the command
is run twice it breaks  because it is used as a string argument to
create a compiler.

Maybe a solution would be to rename the option name into "compiler_type"
and keep build_ext.compiler as a compiler instance.
msg87663 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2009-05-12 23:12
Tarek,
Sorry about breaking the compiler option, I was aware that passing None
there would have that risk.

I think anything that breaks people's setup.py shouldn't go into
release26-maint without a really important motivation.

Is there a way to keep self.compiler as an attribute and make
new_compiler work correctly with it?
msg87669 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-05-13 07:55
new_compiler takes "compiler" as a compiler name. It would make much
sense to make it accept a compiler instance.

now for people setup.py, I wasn't aware that people are using
build_ext.compiler as a compiler, besides Python itself.

IMHO we should rename the compiler option to "compiler_type" then
provide an API build_ext.get_compiler() then fix Python setup.py file.

Then this should not go into 2.6 but in 2.7, and we should put back the
option in new_compiler like it was (it's wrong but at least it works if
you run the command just once, which is the most common case)
msg87670 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-05-13 08:12
typo: 

It would make much... -> It would not make much...
msg87759 - (view) Author: Roumen Petrov (rpetrov) * Date: 2009-05-14 20:11
Proposed patch may fix another windows failure as after recent commit
"unresolved external symbol initfoo" seems to me fixed.
msg87761 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-05-14 20:27
Roumen, the problem you mentioned was existing because of the
empty c file in test_get_outputs. I have fixed it yesterday by creating a
simple c file containing a minimal code:

void inifoo(void) {};

can you check it works for you ?
msg143918 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-12 16:21
The fix you committed uses an absolute path for the C sources, which is invalid: sources should contain relative paths only.
msg213606 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-03-15 01:06
This bug is quite bad, and was reported as far back as 2005 on distutils-sig.  If one runs “python setup.py build_ext --inplace build”, then for example runs tests, and then “python setup.py install” runs build again, which runs build_py (.py files are already in the build directory, do nothing) then build_ext (.so files are not in the build directory: build them), so extensions modules are installed.  But if one has “[build_ext] inplace=1” in the setup.cfg, then the .so files will not be recompiled and not installed.

It would be interesting to test if bdist_wheel is affected, as bdist commands use the install command under the hood.
msg386281 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-03 18:10
Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools
History
Date User Action Args
2021-02-03 18:10:47steve.dowersetstatus: open -> closed

nosy: + steve.dower
messages: + msg386281

resolution: out of date
stage: patch review -> resolved
2014-03-15 01:06:19eric.araujosetassignee: tarek -> eric.araujo
messages: + msg213606
components: - Distutils2
versions: + Python 3.4, - 3rd party, Python 3.2
2011-09-12 16:24:19eric.araujolinkissue9322 dependencies
2011-09-12 16:21:31eric.araujosetmessages: + msg143918
versions: - Python 3.1
2011-03-21 01:40:43eric.araujosetnosy: + eric.araujo, alexis
title: distutils build_ext.get_outputs returns wrong result (patch) -> distutils build_ext.get_outputs returns wrong result
versions: + 3rd party, Python 3.2, Python 3.3, - Python 2.6, Python 3.0
components: + Distutils2
type: crash -> behavior
stage: patch review
2009-05-14 20:27:20tareksetmessages: + msg87761
2009-05-14 20:11:26rpetrovsetfiles: + issue5977-w32.patch
nosy: + rpetrov
messages: + msg87759

2009-05-13 08:12:46tareksetmessages: + msg87670
2009-05-13 07:55:54tareksetmessages: + msg87669
2009-05-12 23:12:51ajaksu2setpriority: critical -> normal

messages: + msg87663
2009-05-12 22:09:18tareksetnosy: + benjamin.peterson
messages: + msg87662
2009-05-12 18:46:08ajaksu2setstatus: closed -> open
files: + build_ext._compiler.diff

nosy: + ajaksu2
messages: + msg87654

priority: critical
2009-05-12 17:17:20tareksetstatus: open -> closed

messages: + msg87643
versions: + Python 2.6, Python 3.0
2009-05-10 14:31:31kxrobertosetmessages: + msg87525
2009-05-10 12:26:47tareksetmessages: + msg87524
2009-05-09 19:15:23kxrobertosetmessages: + msg87506
2009-05-09 13:36:39tareksetmessages: + msg87495
versions: + Python 3.1, Python 2.7, - Python 2.6
2009-05-09 13:29:48kxrobertocreate