classification
Title: "Out of tree" build fails on AIX
Type: compile error Stage: resolved
Components: Build Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: David.Edelsohn, Tristan.Carel, eric.araujo, haubi, martin.panter, pedz, python-dev, sable
Priority: normal Keywords: patch

Created on 2010-12-09 03:35 by pedz, last changed 2018-01-02 15:40 by David.Edelsohn. This issue is now closed.

Files
File name Uploaded Description Edit
aix-fix-out-of-source.diff Tristan.Carel, 2011-03-31 11:15 Patch to apply on 2.7 branch review
issue10656-out-of-source-build-on-aix.patch haubi, 2014-06-04 06:20 patch for 3.4 review
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Repositories containing patches
http://hg.code.sf.net/p/prefix-launcher/cpython#issue10656
Messages (10)
msg123668 - (view) Author: Perry Smith (pedz) Date: 2010-12-09 03:35
I do "out of tree" builds (I think that is the right term).  I make an
empty directory, cd into it, and then do: 

<path to src directory>/configure

followed by make.  This works on most open source pages.  It appears to
be very close to working with Python 2.7.1 but the problem is that it assumes ld_so_aix is in the source tree but it is in the build tree.

I did an "in tree" build using basically just ./configure && make and so far (I'm still testing) that has worked fine.
msg128275 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-10 10:15
There are some unit tests concerning distutils which are currently failing on AIX and may be an illustration of the same problem:

======================================================================
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 254, in link
    self.spawn(linker + ld_args)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 911, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 34, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 138, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command './Modules/ld_so_aix' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/tests/test_build_ext.py", line 85, in test_build_ext
    cmd.run()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 456, in build_extensions
    self.build_extension(ext)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 543, in build_extension
    target_lang=language)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 719, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 256, in link
    raise LinkError(msg)
distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

======================================================================
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 254, in link
    self.spawn(linker + ld_args)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 911, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 34, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 138, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command './Modules/ld_so_aix' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/tests/test_build_ext.py", line 343, in test_get_outputs
    cmd.run()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 456, in build_extensions
    self.build_extension(ext)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 543, in build_extension
    target_lang=language)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 719, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 256, in link
    raise LinkError(msg)
distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

I will take a look.
msg128315 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-10 17:12
I tried with trunk (python 3.2) on AIX:
./build/configure --with-pydebug --without-computed-gotos
make
and that worked correctly.

Not sure with python 2.7.
msg132649 - (view) Author: Tristan Carel (Tristan.Carel) Date: 2011-03-31 11:15
I have experienced the same problem under ppc64 aix 6.1.2.0 while compiling Python 2.7.1

$ cd Python-2.7.1
$ mkdir __build
$ cd __build
$ ../configure --with-gcc=xlc_r --with-cxx-main=xlC_r --with-threads --disable-ipv6
$ /usr/linux/bin/make
[...]
../Modules/ld_so_aix xlc_r -bI:Modules/python.exp -L.. -L/opt/freeware/lib -L/opt/freeware/lib build/temp.aix-6.1-2.7/data/tcarel/ngbuilder/python/2.7.1/Python-2.7.1/Modules/_struct.o -L/opt/freeware/lib -L/usr/local/lib -o build/lib.aix-6.1-2.7/_struct.so
unable to execute ../Modules/ld_so_aix: No such file or directory

There are references to $(srcdir)/Modules/ld_so_aix in both configure.in and Makefile.pre.in whereas ld_so_aix is created in build tree.

==================================================
diff -r f4e6e18cae4b Makefile.pre.in
--- a/Makefile.pre.in	Tue Mar 29 17:28:25 2011 -0700
+++ b/Makefile.pre.in	Thu Mar 31 13:05:45 2011 +0200
@@ -1028,7 +1028,7 @@
 		$(INSTALL_SCRIPT) $(srcdir)/Modules/makexp_aix	\
 				$(DESTDIR)$(LIBPL)/makexp_aix;		\
 		echo "$(LIBPL)/makexp_aix";			\
-		$(INSTALL_SCRIPT) $(srcdir)/Modules/ld_so_aix	\
+		$(INSTALL_SCRIPT) Modules/ld_so_aix	\
 				$(DESTDIR)$(LIBPL)/ld_so_aix;		\
 		echo "$(LIBPL)/ld_so_aix";			\
 		echo; echo "See Misc/AIX-NOTES for details.";	\
diff -r f4e6e18cae4b configure.in
--- a/configure.in	Tue Mar 29 17:28:25 2011 -0700
+++ b/configure.in	Thu Mar 31 13:05:45 2011 +0200
@@ -1778,7 +1778,7 @@
 then
 	case $ac_sys_system/$ac_sys_release in
 	AIX*)
-		BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) -bI:\$(srcdir)/Modules/python.exp"
+		BLDSHARED="Modules/ld_so_aix \$(CC) -bI:Modules/python.exp -L\$(srcdir)"
 		LDSHARED="\$(BINLIBDEST)/config/ld_so_aix \$(CC) -bI:\$(BINLIBDEST)/config/python.exp"
 		;;
 	BeOS*)
==================================================

After fixing these 2 files, Modules/ld_so_aix is found and properly executed by compilation process, but then there is a new issue:

Modules/ld_so_aix xlc_r -bI:Modules/python.exp -L.. -L/opt/freeware/lib -L/opt/freeware/lib build/temp.aix-6.1-2.7/data/tcarel/Python-2.7.1/Modules/_struct.o -L/opt/freeware/lib -L/usr/local/lib -o build/lib.aix-6.1-2.7/_struct.so
Modules/ld_so_aix[183]: Modules/makexp_aix:  not found.

This time, makexp_aix is expected to be is the same directory than ld_so_aix whereas makeexp_aix remains in the source tree.

Here is a fix:

==================================================
diff -r f4e6e18cae4b Modules/ld_so_aix.in
--- a/Modules/ld_so_aix.in	Tue Mar 29 17:28:25 2011 -0700
+++ b/Modules/ld_so_aix.in	Thu Mar 31 13:01:57 2011 +0200
@@ -69,7 +69,10 @@
   echo $usage; exit 2
 fi
 
-makexp=`dirname $0`/makexp_aix
+makexp=`dirname $0`/@srcdir@/makexp_aix
+if test ! -e "$makexp" ; then
+  makeexp=makexp_aix
+fi
 
 # Check for existence of compiler.
 CC=$1; shift
==================================================

The first value fix the issue while building Python. The second will use PATH to find makeexp_aix as explained in section `SHARED LIBRARY SUPPORT' in AIX-NOTES (for building shared extension modules).

You can find a patch for 2.7 branch in attachment.

Regards
-- 
Tristan Carel
Exalead
msg219740 - (view) Author: Michael Haubenwallner (haubi) * Date: 2014-06-04 06:20
Basically the same as Tristan's patch, with a little improvement to not rely on PATH to find makexp_aix within ld_so_aix.

Thanks!
msg279474 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-10-26 01:05
The idea of the patches makes sense to me.

Embedding the name of whatever directory Python was built in into the installed version of Modules/ld_so_aix.in does not seem very clean to me. Is there another way of doing this? Perhaps a file copy or symlink?
msg279475 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-10-26 01:14
Actually, why do you remove $(srcdir) for Modules/python.exp? This code was added in r88426; see <https://bugs.python.org/issue941346#msg128586>.
msg279526 - (view) Author: Michael Haubenwallner (haubi) * Date: 2016-10-27 06:32
Modules/python.exp is generated at build-time, thus does not belong to $(srcdir).
msg281196 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-11-19 03:18
Okay I will assume that $(srcdir) reference was just due to a mistake or bad cargo-culting then.

Will commit this soon unless anyone has a good idea to avoid embedding @abs_srcdir@.
msg281307 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-11-20 23:24
New changeset 48526666321a by Martin Panter in branch '3.5':
Issue #10656: Fix out-of-tree building on AIX
https://hg.python.org/cpython/rev/48526666321a

New changeset 76d1f8001e27 by Martin Panter in branch '3.6':
Issue #10656: Merge AIX build fix from 3.5
https://hg.python.org/cpython/rev/76d1f8001e27

New changeset 180f046b597e by Martin Panter in branch 'default':
Issue #10656: Merge AIX build fix from 3.6
https://hg.python.org/cpython/rev/180f046b597e

New changeset ca46883fc5cf by Martin Panter in branch '2.7':
Issue #10656: Fix out-of-tree building on AIX
https://hg.python.org/cpython/rev/ca46883fc5cf
History
Date User Action Args
2018-01-02 15:40:35David.Edelsohnsetnosy: + David.Edelsohn
2017-03-31 16:36:11dstufftsetpull_requests: + pull_request864
2016-11-21 03:31:23martin.pantersetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2016-11-20 23:24:12python-devsetnosy: + python-dev
messages: + msg281307
2016-11-19 03:18:58martin.pantersetmessages: + msg281196
stage: patch review -> commit review
2016-10-27 06:32:24haubisetmessages: + msg279526
2016-10-26 01:14:39martin.pantersetmessages: + msg279475
2016-10-26 01:05:00martin.pantersetversions: + Python 3.6, Python 3.7, - Python 3.4
nosy: + martin.panter

messages: + msg279474

stage: patch review
2014-06-06 11:26:33haubisethgrepos: + hgrepo252
2014-06-04 06:20:02haubisetfiles: + issue10656-out-of-source-build-on-aix.patch

messages: + msg219740
2014-06-04 06:14:55haubisetversions: + Python 3.4, Python 3.5
title: "Out of tree" build fails on AIX 5.3 -> "Out of tree" build fails on AIX
2014-06-04 06:14:30haubisethgrepos: - hgrepo246
2014-06-03 09:23:55haubisethgrepos: + hgrepo246
2014-06-02 15:00:20haubisetnosy: + haubi
2011-03-31 11:15:40Tristan.Carelsetfiles: + aix-fix-out-of-source.diff

nosy: + Tristan.Carel
messages: + msg132649

keywords: + patch
2011-02-10 17:12:45sablesetnosy: sable, eric.araujo, pedz
messages: + msg128315
2011-02-10 10:15:20sablesetnosy: sable, eric.araujo, pedz
messages: + msg128275
2010-12-15 15:49:22pitrousetnosy: + sable
2010-12-09 03:36:00eric.araujosetnosy: + eric.araujo
2010-12-09 03:35:03pedzcreate