classification
Title: can't build extensions with non-default ldflags (e.g. -m32)
Type: behavior Stage: resolved
Components: Distutils, Tests Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: tarek Nosy List: Arfrever, TheoLandreville, benjamin.peterson, brett.cannon, eric.araujo, georg.brandl, jyasskin, loewis, lukasz.langa, pitrou, tarek
Priority: release blocker Keywords: patch

Created on 2010-07-31 15:55 by pitrou, last changed 2011-02-15 19:03 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
m32.patch pitrou, 2010-08-21 17:42
m32-2.patch pitrou, 2010-09-05 11:53
Messages (23)
msg112163 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-07-31 15:55
When I use the following configuration (in order to build a 32-bit Python a 64-bit OS):
  ./configure --with-computed-gotos CFLAGS=-m32 LDFLAGS=-m32

I then get two failures in test_distutils:

test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase) ... /tmp/tmpssz80G/tmp/tmpssz80G/xxmodule.o: could not read symbols: File in wrong format
collect2: ld a retourné 1 code d'état d'exécution
ERROR
[...]
test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase) ... /tmp/tmphxwa1S/tempt/tmp/tmpuD17us/foo.o: could not read symbols: File in wrong format
collect2: ld a retourné 1 code d'état d'exécution
ERROR
[...]

======================================================================
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antoine/py3k/m32/Lib/distutils/unixccompiler.py", line 254, in link
    self.spawn(linker + ld_args)
  File "/home/antoine/py3k/m32/Lib/distutils/ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/home/antoine/py3k/m32/Lib/distutils/spawn.py", line 34, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/home/antoine/py3k/m32/Lib/distutils/spawn.py", line 138, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/antoine/py3k/m32/Lib/distutils/tests/test_build_ext.py", line 65, in test_build_ext
    cmd.run()
  File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 456, in build_extensions
    self.build_extension(ext)
  File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 543, in build_extension
    target_lang=language)
  File "/home/antoine/py3k/m32/Lib/distutils/ccompiler.py", line 719, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/home/antoine/py3k/m32/Lib/distutils/unixccompiler.py", line 256, in link
    raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1

======================================================================
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antoine/py3k/m32/Lib/distutils/unixccompiler.py", line 254, in link
    self.spawn(linker + ld_args)
  File "/home/antoine/py3k/m32/Lib/distutils/ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/home/antoine/py3k/m32/Lib/distutils/spawn.py", line 34, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/home/antoine/py3k/m32/Lib/distutils/spawn.py", line 138, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/antoine/py3k/m32/Lib/distutils/tests/test_build_ext.py", line 321, in test_get_outputs
    cmd.run()
  File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 456, in build_extensions
    self.build_extension(ext)
  File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 543, in build_extension
    target_lang=language)
  File "/home/antoine/py3k/m32/Lib/distutils/ccompiler.py", line 719, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/home/antoine/py3k/m32/Lib/distutils/unixccompiler.py", line 256, in link
    raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1
msg114525 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-21 17:42
It turns out, quite expectedly, that distutils doesn't reuse the various LDFLAGS recorded by sysconfig (why are there three of them?), but instead only uses LDSHARED:

>>> sysconfig.get_config_var('LDSHARED')
'gcc -pthread -shared'
>>> sysconfig.get_config_var('LDFLAGS')
'-m32 '
>>> sysconfig.get_config_var('PY_LDFLAGS')
'-m32'
>>> sysconfig.get_config_var('CONFIGURE_LDFLAGS')
'-m32'

I would suggest bumping this to critical or even release blocker, since building of extensions is broken when using non-default linker flags.
msg114527 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-21 17:42
Following patch works here. I've added XXX tags in strange places.
msg114530 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-21 17:47
Ok, there is also a failure in test_sysconfig (the patch makes no difference):

======================================================================
FAIL: test_ldshared_value (test.test_sysconfig.TestSysConfig)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antoine/py3k/m32/Lib/test/test_sysconfig.py", line 285, in test_ldshared_value
    self.assertIn(ldflags, ldshared)
AssertionError: '-m32 ' not found in 'gcc -pthread -shared'
msg114532 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-21 17:49
It appears that 2.7 has the same issue, although fewer tests fail (apparently not all 3.2 tests have been backported).
msg114582 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-08-21 22:35
May be related to #4010 and #9047

The bug report that triggered the edits to configure is #9189
msg115634 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-09-05 08:12
Patch looks like it at least can't make things worse to me.  linker_exe should probably also get LDFLAGS (but not LDSHARED).
msg115638 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-09-05 08:16
It is however not important enough to block 3.2a2.
msg115642 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-05 11:53
Other patch, fixes all failures.
msg115661 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-09-05 17:48
I agree with Georg: The patch seems not to make things worse, and if it makes the tests pass, I think it should be committed. Antoine, you decide if you want to commit it now or wait for Tarek’s review.
msg115662 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-05 17:52
> I agree with Georg: The patch seems not to make things worse, and if
> it makes the tests pass, I think it should be committed. Antoine, you
> decide if you want to commit it now or wait for Tarek’s review.

Well, ideally, someone should have knowledge of what is precisely needed
for these configuration variables. Otherwise the overall robustness of
distutils and friends won't really progress.
msg115663 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-09-05 18:01
I agree. This expert is not me, and my reply was only pragmatic.

Martin: This is a bug about LDSHARED vs. LDFLAGS vs. PY_LDFLAGS vs. CONFIGURE_LDFLAGS. Could you maybe shed some light about the difference between these configure variables and review the small patch made by Antoine? Thanks.
msg118326 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-10 09:37
Committed to 3.x in r85353.
msg118329 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-10 09:55
Backported to 2.7 in r85358.
msg118363 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-11 14:27
There are now failures on the FreeBSD 7.2 buildbot:

======================================================================
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/unixccompiler.py", line 254, in link
    self.spawn(linker + ld_args)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/spawn.py", line 34, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/spawn.py", line 138, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/tests/test_build_ext.py", line 65, in test_build_ext
    cmd.run()
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 456, in build_extensions
    self.build_extension(ext)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 543, in build_extension
    target_lang=language)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/ccompiler.py", line 719, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/unixccompiler.py", line 256, in link
    raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1

======================================================================
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/unixccompiler.py", line 254, in link
    self.spawn(linker + ld_args)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/spawn.py", line 34, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/spawn.py", line 138, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/tests/test_build_ext.py", line 321, in test_get_outputs
    cmd.run()
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 456, in build_extensions
    self.build_extension(ext)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 543, in build_extension
    target_lang=language)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/ccompiler.py", line 719, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/unixccompiler.py", line 256, in link
    raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1


Of course since distutils doesn't tell us what the gcc error message is, it's hard to do anything.
msg118364 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-11 14:31
Apparently, the error message is:

gcc: ${LDFLAGS}: No such file or directory
msg118366 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-10-11 14:34
Notice that the error shows further up in the output, at the point where the test was actually run:

gcc: ${LDFLAGS}: No such file or directory
msg118369 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-11 15:34
The cause for the failure seems to be a bug in the dreaded _parse_makefile(). When you give it a Makefile such as:

LDFLAGS=-foo
PY_LDFLAGS=-bar
LDSHARED=$(CC) -shared ${LDFLAGS} $(PY_LDFLAGS)

It outputs the following variables:

{'CC': '',
 'LDFLAGS': '-foo',
 'LDSHARED': '-shared ${LDFLAGS} -bar',
 'PY_LDFLAGS': '-bar'}

That is, ${LDFLAGS} isn't expanded.

However, since the Makefile now explicitly appends PY_LDFLAGS to LDSHARED, we could also stop doing so in ./configure.
msg118551 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-13 17:27
> However, since the Makefile now explicitly appends PY_LDFLAGS to
> LDSHARED, we could also stop doing so in ./configure.

I've committed a patch which does just this (in r85422). Hopefully it
won't break anything.
msg119518 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2010-10-24 15:23
These changes cause some regressions:
- distutils builds third-party extensions with LDFLAGS, which were used to build CPython. (This is more important regression.)
  This problem concerns Python 2.7 and 3.2.
- distutils builds third-party extensions with LDFLAGS environment variable passed twice to compiler (distutils uses LDSHARED, which now contains $(LDFLAGS), and uses LDFLAGS).
  This problem concerns Python 3.2.

Example for 3.2:
x86_64-pc-linux-gnu-gcc -pthread -shared ${LDFLAGS_from_CPython} ${LDFLAGS_from_current_environment} ${LDFLAGS_from_current_environment}  ${CFLAGS} ${object_files} -L/usr/lib64 -lpython3.2mu -o ${extension_module}
msg119519 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-24 15:42
> These changes cause some regressions:
> - distutils builds third-party extensions with LDFLAGS, which were
> used to build CPython. (This is more important regression.)
>   This problem concerns Python 2.7 and 3.2.

Well, is this a problem? This sounds like the expected behaviour to me.
Especially if Python was packaged was someone else and you don't want
the user to manually set LDFLAGS each time they run a setup.py.
msg128614 - (view) Author: Landreville (TheoLandreville) Date: 2011-02-15 18:58
> I've committed a patch which does just this (in r85422). Hopefully it
> won't break anything.

Could I convince you to patch 2.7 as well?
msg128617 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-15 19:03
> > I've committed a patch which does just this (in r85422). Hopefully it
> > won't break anything.
> 
> Could I convince you to patch 2.7 as well?

It was backported in r86848. This means it should be available in 2.7.2.
History
Date User Action Args
2011-02-15 19:03:31pitrousetnosy: loewis, brett.cannon, georg.brandl, pitrou, jyasskin, benjamin.peterson, tarek, eric.araujo, Arfrever, lukasz.langa, TheoLandreville
messages: + msg128617
2011-02-15 18:58:54TheoLandrevillesetnosy: + TheoLandreville, benjamin.peterson
messages: + msg128614
2010-10-24 15:42:55pitrousetmessages: + msg119519
2010-10-24 15:23:47Arfreversetnosy: + Arfrever
messages: + msg119518
2010-10-18 17:37:40pitrousetstatus: open -> closed
2010-10-13 17:27:36pitrousetmessages: + msg118551
2010-10-11 15:34:44pitrousetmessages: + msg118369
2010-10-11 14:34:48loewissetmessages: + msg118366
2010-10-11 14:31:10pitrousetmessages: + msg118364
2010-10-11 14:27:40pitrousetstatus: pending -> open

messages: + msg118363
2010-10-10 09:55:38pitrousetstatus: open -> pending
resolution: fixed
messages: + msg118329

stage: needs patch -> resolved
2010-10-10 09:37:54pitrousetmessages: + msg118326
2010-09-29 16:23:49brett.cannonsetnosy: + brett.cannon
2010-09-29 15:21:55pitroulinkissue9963 superseder
2010-09-06 08:26:23georg.brandlsetpriority: deferred blocker -> release blocker
2010-09-05 18:01:53eric.araujosetnosy: + loewis
messages: + msg115663
2010-09-05 17:52:59pitrousetmessages: + msg115662
2010-09-05 17:48:02eric.araujosetmessages: + msg115661
2010-09-05 11:53:43pitrousetfiles: + m32-2.patch

messages: + msg115642
2010-09-05 11:39:33lukasz.langasetmessages: - msg115640
2010-09-05 11:37:34lukasz.langasetnosy: + lukasz.langa
messages: + msg115640
2010-09-05 08:16:31georg.brandlsetpriority: release blocker -> deferred blocker

messages: + msg115638
2010-09-05 08:12:57georg.brandlsetnosy: + georg.brandl
messages: + msg115634
2010-08-21 22:35:18eric.araujosetmessages: + msg114582
2010-08-21 17:50:29pitrousetpriority: high -> release blocker
title: test_distutils failure with -m32 -> can't build extensions with non-default ldflags (e.g. -m32)
2010-08-21 17:49:49pitrousetmessages: + msg114532
versions: + Python 2.7
2010-08-21 17:47:41pitrousetmessages: + msg114530
2010-08-21 17:42:39pitrousetfiles: + m32.patch
keywords: + patch
messages: + msg114527
2010-08-21 17:42:00pitrousetnosy: + eric.araujo
messages: + msg114525
2010-07-31 15:55:11pitroucreate