with --inplace etc. build_ext.get_outputs returns wrong extension paths;
tries to computes them out of the air *again* and ..

Tools which need to further know the produced module file, e.g.
pyximport (Cython) do crash then ...

patch see attachment: stores and uses the real path in use.
writes a new attribute ext_filename to Extension object - should it be
underscored _ext_filename or ... ?

can you provide a test case that demonstrates the problem ?

>>> test_build_ext: The test must be run in a python build dir

don't have a build setup currently. maybe in future.

One can yet see in build_ext.py:

after "if self.inplace:" (line485)
there are 2 different computations for ext_filename

while computation in get_outputs() is not aware of self.inplace (line447

I was talking about your use case (pyximport (Cython) do crash then ?)

If I can have a real example, with the output and the expected one,
I will build a test with it in Distutils before I fix it

its with any .pyx (Cython) module , when after pyximport.install() and
injection of --inplace option a .pyx module is imported.

the relevant lines in pyxbuild.py are

        dist.run_commands()
        return dist.get_command_obj("build_ext").get_outputs()[0]

which use the buggy "get_outputs" and shall return the full path of
built module to pyximport.py :

    so_path = pyxbuild.pyx_to_dll(pyxfilename, extension_mod,
                                  build_in_temp=build_in_temp,
                                  pyxbuild_dir=pyxbuild_dir,
                                  setup_args=sargs )
    assert os.path.exists(so_path), "Cannot find: %s" % so_path

=> crash with "Cannot find..." before pyximport.load_module goes to
import it.

-

A stripped down test case should perhaps 'build_ext' any arbitrary
extension module with --inplace ( a option of base command 'build' ) and
something like ...

dist.get_command_obj("build_ext").inplace=1
dist.run_commands()
so_path = dist.get_command_obj("build_ext").get_outputs()[0]
assert os.path.isfile(so_path) and os.path.dirname(so_path) in ('','.')

... will produce a invalid so_path: not pointing to actual locally
in-place built xy.pyd/.so, but to a non-existing or old file in the
build folders

done in r72585 and propagated in 2.6, 3.0, 3.1 branches

Thank you !

Tarek,
It looks like all buildbots are red because setup.py doesn't work with
this change:

running build
running build_ext
Traceback (most recent call last):
  File "./setup.py", line 1896, in <module>
    main()
  File "./setup.py", line 1891, in main
    'Lib/smtpd.py']
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/core.py",
line 149, in setup
    dist.run_commands()
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/dist.py",
line 974, in run_commands
    self.run_command(cmd)
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/dist.py",
line 994, in run_command
    cmd_obj.run()
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/command/build.py",
line 132, in run
    self.run_command(cmd_name)
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/cmd.py",
line 326, in run_command
    self.distribution.run_command(command)
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/dist.py",
line 994, in run_command
    cmd_obj.run()
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/command/build_ext.py",
line 348, in run
    self.build_extensions()
  File "./setup.py", line 103, in build_extensions
    missing = self.detect_modules()
  File "./setup.py", line 302, in detect_modules
    add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib')
AttributeError: 'NoneType' object has no attribute 'library_dirs'
[31295 refs]
*** Error code 1
The following command caused the error:
case $MAKEFLAGS in \
*s*)  CC='gcc' LDSHARED='gcc -shared' OPT='-g -Wall -Wstrict-prototypes'
./python -E ./setup.py -q build;; \
*)  CC='gcc' LDSHARED='gcc -shared' OPT='-g -Wall -Wstrict-prototypes'
./python -E ./setup.py build;; \
esac
make: Fatal error: Command failed for target `sharedmods'
program finished with exit code 1

Sorry about that, I was travelling and didn't see that the buildbots
turned red. (I ran the test but didn't build python again)

I have changed that because build_ext.compiler is a string option, e.g.
the compiler type. Turning it into a compiler instance when the command
is run twice it breaks  because it is used as a string argument to
create a compiler.

Maybe a solution would be to rename the option name into "compiler_type"
and keep build_ext.compiler as a compiler instance.

Tarek,
Sorry about breaking the compiler option, I was aware that passing None
there would have that risk.

I think anything that breaks people's setup.py shouldn't go into
release26-maint without a really important motivation.

Is there a way to keep self.compiler as an attribute and make
new_compiler work correctly with it?

new_compiler takes "compiler" as a compiler name. It would make much
sense to make it accept a compiler instance.

now for people setup.py, I wasn't aware that people are using
build_ext.compiler as a compiler, besides Python itself.

IMHO we should rename the compiler option to "compiler_type" then
provide an API build_ext.get_compiler() then fix Python setup.py file.

Then this should not go into 2.6 but in 2.7, and we should put back the
option in new_compiler like it was (it's wrong but at least it works if
you run the command just once, which is the most common case)

typo: 

It would make much... -> It would not make much...

Proposed patch may fix another windows failure as after recent commit
"unresolved external symbol initfoo" seems to me fixed.

Roumen, the problem you mentioned was existing because of the
empty c file in test_get_outputs. I have fixed it yesterday by creating a
simple c file containing a minimal code:

void inifoo(void) {};

can you check it works for you ?

The fix you committed uses an absolute path for the C sources, which is invalid: sources should contain relative paths only.

This bug is quite bad, and was reported as far back as 2005 on distutils-sig.  If one runs “python setup.py build_ext --inplace build”, then for example runs tests, and then “python setup.py install” runs build again, which runs build_py (.py files are already in the build directory, do nothing) then build_ext (.so files are not in the build directory: build them), so extensions modules are installed.  But if one has “[build_ext] inplace=1” in the setup.cfg, then the .so files will not be recompiled and not installed.

It would be interesting to test if bdist_wheel is affected, as bdist commands use the install command under the hood.

Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools