Building in a separate directory got broken at some point. The code is
hairy but it looks like the source of the problem was a lame
sysconfig.get_python_inc() function. The attached patches fix things
and hopefully do not introduce any new bugs.

get_python_inc() and test case is addressed in bpo-4070 as well.

[Roumen Petrov]

Modification for test_build_ext.py from above mentioned issue break
non-posix builds.

Without that change the test fails on Unix platforms when the build is
not done inside the source directory. I guess the problem must be the
change from (essentially):

def _get_source_filename():
    return os.path.join(sysconfig.project_base, 'Modules', 'xxmodule.c')

to

def _get_source_filename():
    srcdir = sysconfig.get_config_var('srcdir')
    return os.path.join(srcdir, 'Modules', 'xxmodule.c')

On POSIX, project_base is just the directory containing the Python
executable. I think the right thing to do is to fix the srcdir var on
non-POSIX systems. I don't have a Windows machine to test on. Could
someone test the get_python_inc2.patch with the non_posix_srcdir.patch
applied on top of it?

I committed my proposed changes in several chunks, ending with r69305.
I think building in a separate directory again works and that non-POSIX
platforms are not adversely affected by this change.

May be not related to the commit but after clean make fail to build a
number of modules, as example _bisect. Second run of make build all left.

The value of srcdir (from Makefile) is ".." without quotes.

About cleanup: one use of '(srcdir,)==...' left in setup.py .

The "clean build' mean missing subdirectory build in <builddir> . Now
module objects go in build/Modules/. I guess that second make succeed as
libffi create build directory.

Roumen, thanks for reporting the bug. It seems that distutils requires
an absolute path (although I didn't waste time digging too deep into the
issue).

still seen on the 2.6 branch. applying r69374 on the branch doesn't fix
it there.

On Thu, Feb 12, 2009 at 01:32:37PM +0000, Matthias Klose wrote:

still seen on the 2.6 branch. applying r69374 on the branch doesn't fix
it there.

The fix is spread over a number of commits: r69374, r69322, r69315,
r69305, r69304, r69303, r69302. It could be backported but changing
the build system is always fraught with portability traps and that's
why it ends up turning to crud over time.

Obviously most core Python developers don't build in a separate
directory otherwise this feature won't have been broken for so long.

I'm having trouble with this as well :( It's pretty much a blocker for
integrating distutils-based extension builds with an autotools library
build, because the "make distcheck" target explicitly does the build in
a subtree of the source directory, and write-protects the srcdir during
the build.

Effectively the source is referred to in my setup.py as living in
"../../pyext", which translates into distutils attempting to build the
extension in "build/temp.linux-i686-2.5/../../pyext/", which is the
write-protected source dir. This would work okay if the directory path
to the source file were ignored when constructing the output path: would
the proposed patch(es) fix this?

One additional commit change srcdir = os.path.normpath(srcdir) to
srcdir = os.path.abspath(srcdir) and this should not create relative
path. In this case I expect build in directory
build/temp.<PLATFORM_VERSION>/<ABS_SRC_PATH>.

Andi, did above change work for you.

current status with 2.7 alpha3:

FAIL: test_get_python_inc (distutils.tests.test_sysconfig.SysconfigTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7~a3/Lib/distutils/tests/test_sysconfig.py", line 47, in test_get_python_inc
    self.assertTrue(os.path.isfile(python_h), python_h)
AssertionError: /home/packages/python/2.7/python2.7-2.7~a3/build-static/Include/Python.h

unsure about this one:
FAIL: test_finalize_options (distutils.tests.test_build.BuildTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7~a3/Lib/distutils/tests/test_build.py", line 24, in test_finalize_options
    self.assertEquals(cmd.build_purelib, wanted)
AssertionError: 'build/lib.linux-i686-2.7' != 'build/lib'

I believe the latest problem reported has been fixed by SVN rev 80649.

I think so too.

For the record, I’m not clear about what is fixed or not in each branch.