classification
Title: AIX shared library extension modules installation broken
Type: behavior Stage: resolved
Components: Installation Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: David.Edelsohn, Michael.Felt, ericvw, martin.panter, pitrou, python-dev
Priority: normal Keywords:

Created on 2015-12-08 19:29 by David.Edelsohn, last changed 2018-09-18 15:08 by Michael.Felt. This issue is now closed.

Files
File name Uploaded Description Edit
Issue25825-patch1.txt David.Edelsohn, 2016-08-10 15:22 Patch for Makefile.pre.in review
Issue25825-patch2.txt David.Edelsohn, 2016-08-13 17:56 Patch to configure.ac for AIX LDSHARED review
Messages (20)
msg256123 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2015-12-08 19:29
AIX requires helper scripts to build Python shared extension modules.  The definitions and Makefile installation rules have bitrotted.

Makefile.pre.in:

      @if [ -s Programs/python.exp -a \

except python.exp is created in Modules/python.exp, not Programs.exp

               $(INSTALL_DATA) Programs/python.exp             \
                               $(DESTDIR)$(LIBPL)/python.exp;

Post-substitution Makefile.pre:

BINLIBDEST=     $(LIBDIR)/python$(VERSION)
LIBDEST=        $(SCRIPTDIR)/python$(VERSION)
LIBPL=          $(LIBDEST)/config-$(LDVERSION)

configure.ac:

        AIX*)
                BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) -bI:\$(srcdir)/Modules/python.exp"
                LDSHARED="\$(BINLIBDEST)/config/ld_so_aix \$(CC) -bI:\$(BINLIBDEST)/config/python.exp"

which is forever enshrined in _sysconfigdata.py

In other words, configure sets LDSHARED to $(BINLIBDEST)/config, which does not exist in current installations.  And Makefile installs the files in LIBPL, which is based on LIBDEST (prefix and exec_prefix could be different).

And the files are not installed, because the installation tests Programs/python.exp instead of Modules/python.exp.

Changing Makefile.pre.in to test Modules/python.exp is easy enough as a partial fix.

What is the preferred location to install the files so that configure.ac and Makefile.pre.in can be adjusted to match each other?
msg256136 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-12-09 04:38
It looks like the Programs/python.exp change was done accidentally in revision 88a532a31eb3 (Issue 18093), so we should be safe reverting the relevant lines back to Modules/python.exp.

The configure script currently sets

LIBPL='$(prefix)'"/lib/python${VERSION}/config-${LDVERSION}"
AC_SUBST(LIBPL)

However there is a contradictory comment in the makefile:

# This goes into $(exec_prefix)
LIBPL=  @LIBPL@

It looks like the comment came first. $(LIBPL) was changed from $(LIBDIR) [presumably using $(exec_prefix)] to use $(prefix) in r86731 (Issue 9807).

This isn’t my area of expertise, but my understanding is $(exec_prefix) is for architecture-specific files only. If python.exp is not architecture-specific, perhaps it should go in $(prefix), and the configure script (and makefile comment) needs fixing.
msg256155 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2015-12-09 13:59
$(prefix) and $(exec_prefix) result in the same path on AIX, so it does not matter in practice, although the semantics are different.

# Install prefix for architecture-dependent files
exec_prefix=    ${prefix}

python.exp is not architecture dependent, although it only is useful on AIX target.  It is essentially equivalent to a list of symbols with ELF global, non-hidden visibility.  It is less confusing if the list is co-located with the scripts that use it.

LIBPL is fine with me.  configure.ac and Makefile.pre.in must match.
msg261127 - (view) Author: Michael Felt (Michael.Felt) * Date: 2016-03-02 16:42
FYI: when build and src are the same directory, there is no error message. However, when src and build are in seperate directories (e.g.,

build = '.',
src = '../src/python-2.7.11'

the following message occurs MANY times:

unable to execute '../src/python-2.7.11/Modules/ld_so_aix': No such file or directory

during configure the file
./Modules/ld_so_aix is 'created' in the build area.

root@x064:[/data/prj/aixtools/python-2.7.11]ls -l Modules/ld*aix*
-rwxr-xr-x   1 root     system         6205 Mar 02 16:24 Modules/ld_so_aix

Just guessing, but a line such as:
BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) -bI:\$(srcdir)/Modules/python.exp"

using $(srcdir) is not correct.

Further, a find in $(srcdir) for ld_so_aix returns:
root@x064:[/data/prj/aixtools/python-2.7.11]ls -dl ../src/py*
drwxr-xr-x  17 199      1954           4096 May 23 2015  ../src/python-2.7.10
drwxr-xr-x  17 199      1954           4096 Dec 05 19:47 ../src/python-2.7.11

So, $(srcdir) seems to be wrong, by definition.

For now, I am copying $(srcdir) to $(builddir) to work-around this.
msg271417 - (view) Author: Michael Felt (Michael.Felt) * Date: 2016-07-26 22:05
hmm. needs patch. For what?

One comment says a change will be undone.

If a patch is expected for src/builddir issue:

a) I sinned - that is a new issue, and should be posted seperately.
b) i do not know autotools well enough to solve the srcdir and builddir problem.

rather than a patch i can on that is see if the same issue occurs with linux and post a new issue.

So, my question would be - was a rollback done - if yes, perhaps a comment to verify and then close this issue.
msg272336 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2016-08-10 15:22
Let's start with this patch to revert the change mentioned in msg256136 to look for python.exp in Modules -- the test, the source for the file, and the location to delete the file.

A follow-up patch will fix the data in _sysconfigdata.py.
msg272490 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-08-12 02:58
Patch1 looks fine to me, though I will have to rely on you people to verify that it does what it’s supposed to. Do you want me to commit it straight away, or wait for your follow-up patch?
msg272538 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2016-08-12 13:08
Yes, please apply Patch 1 that reverts the mistaken change of revision 88a532a31eb3 .  I want to work through this incrementally so that it's clear to reviewers.
msg272539 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-12 13:27
New changeset 58c8cae6c61a by Martin Panter in branch '3.5':
Issue #25825: Fix references to Modules/python.exp
https://hg.python.org/cpython/rev/58c8cae6c61a

New changeset 4d4b5b978b7e by Martin Panter in branch 'default':
Issue #25825: Merge AIX fix from 3.5
https://hg.python.org/cpython/rev/4d4b5b978b7e
msg272602 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2016-08-13 17:56
The second of two patches. This patch changes the definition of LDSHARED for AIX in configure to reference the matching installed location as defined in Makefile.pre.in by Patch1.  The definition from configure propagates into _sysconfigdata.py.

This change will (further) break test_distutils, but the testcase is wrong on AIX. And, for AIX, a correct test result clearly does not correspond to correct behavior.

The patch affects both configure.ac and configure because the corresponding change to configure is obvious. The person who installs the patch may regenerate configure, as appropriate.
msg273389 - (view) Author: Eric N. Vander Weele (ericvw) * Date: 2016-08-22 17:59
> Patch1 looks fine to me, though I will have to rely on you people to verify that it does what it’s supposed to. Do you want me to commit it straight away, or wait for your follow-up patch?

Independently, I have created a similar patch as well and can also verify that it does what it's suppose to do.

> The second of two patches. This patch changes the definition of LDSHARED for AIX in configure to reference the matching installed location as defined in Makefile.pre.in by Patch1. [...]

I can also confirm that Patch2 is necessary as well and does what it's suppose to do, having creating this patch independently as well.
msg273416 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-08-23 03:02
It looks like the change from plain config to config-<ABI> was made in Issue 9807 (r86731, 3.2). So the patch seems reasonable to me; it is just catching up with that change.

This bug was marked for 2.7 as well. Is there anything that needs to be done for 2.7?

How does patch 2 make the test_distutils situation worse? Is there anything that should be done to improve things first, or should that just be handled independently? I see that the 2.7 and 3.5 buildbots are currently failing with errors like

distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

while 3.6 is failing with

distutils.errors.LinkError: command '/usr/local/lib/python3.6/config/ld_so_aix' failed with exit status 1

Presumably the 3.6 failure may be helped by Patch 2, since it is complaining about the plain “config” path.
msg273417 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2016-08-23 03:11
> This bug was marked for 2.7 as well. Is there anything that needs to be done for 2.7?

It would be great if both patches were applied to 2.7 also.

> How does patch 2 make the test_distutils situation worse? Is there anything that should be done to improve things first, or should that just be handled independently? I see that the 2.7 and 3.5 buildbots are currently failing with errors like

distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

while 3.6 is failing with

distutils.errors.LinkError: command '/usr/local/lib/python3.6/config/ld_so_aix' failed with exit status 1

Presumably the 3.6 failure may be helped by Patch 2, since it is complaining about the plain “config” path.

Patch2 makes test_distutils worse because LDSHARED refers to a file that only exists when Python is installed.  If one tests in tree, the files don't exist.  This may be related to the existence of BLDSHARED, used to build modules in the source tree, which can be overridden separately -- otherwise it defaults to LDSHARED.  If the rest of the testsuite ever is cleaned up for AIX, one either could skip the parts of test_distutils that require the LDSHARED files or add special hooks in the test for AIX.  As you point out, the change doesn't make the current testsuite results situation any worse.
msg273425 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-08-23 08:56
Patch 1 does not apply to 2.7. The offending commit 88a532a31eb3 was not made in that branch.

Patch 2 may apply to 2.7, but I am not sure if it is needed or worthwhile. Again, the offending commit r86731 not made to 2.7. There is also a $(BINLIBDEST)/config reference for Be OS; surely if one is updated the other should too.

David: I don’t understand “Patch2 makes test_distutils worse because LDSHARED refers to a file that only exists when Python is installed.” At the moment, doesn’t LDSHARED refer to a file that never exists? So at least the test will be improved when Python is installed, right?

I looked into the test_distutils failures and here’s what I found:

* Revision c958678720fd assigns BLDSHARED → LDSHARED in distutils if the source tree is detected. This survived in the 3.5 branch, so may explain why the 3.5 buildbot failure mentions “./Modules/ld_so_aix”.

* In 2.7, revision fa69e891edf4 moved the assignment into sysconfig.py.

* Then be3b4aa2ad28 loads LDSHARED from _sysconfigdata.py, which is created by sysconfig.py with the alternative BLDSHARED → LDSHARED value. This explains “./Modules/ld_so_aix” in 2.7.

* In 3.6, c554194240fc (Issue 18235) reversed the assignment in sysconfig.py, which now assigns LDSHARED → BLDSHARED, and creates _sysconfigdata.py with this alternative value.

* Then 3fa8aebed636 loads LDSHARED directly from _sysconfigdata.py. In this case it is the original LDSHARED value, so the 3.6 buildbot failure refers to /usr/local/lib/python3.6/config/ld_so_aix.
msg273448 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2016-08-23 14:00
$(BINLIBDEST)/config is equivalent to $(LIBPL) in Python 2.7, so Python 2.7 should be okay.

Patch2 will not make test_distutils results worse, but the test results may not represent the true status of distutils on AIX if the matching Python version is not installed so that the test can find the files in the installed location.
msg273756 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-27 04:10
New changeset ca1ddd365f5f by Martin Panter in branch '3.5':
Issue #25825: Fix references to $(LIBPL) installation path on AIX
https://hg.python.org/cpython/rev/ca1ddd365f5f

New changeset 5a05c0eeefc3 by Martin Panter in branch 'default':
Issue #25825: Merge $(LIBPL) fix from 3.5
https://hg.python.org/cpython/rev/5a05c0eeefc3
msg273766 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-08-27 07:20
Okay, the second patch is committed to 3.5+. Is everything working now (on 2.7, 3.5, 3.6), or is there more to do?
msg274375 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2016-09-04 18:24
I believe that everything is functioning correctly.
msg325654 - (view) Author: Michael Felt (Michael.Felt) * Date: 2018-09-18 14:46
On 10/08/2016 17:22, David Edelsohn wrote:
> David Edelsohn added the comment:
Hi, Just thought I would mention that I have a branch with all the test
PR fixes I have made in the last two months.

a) would appreciate your testig them with gcc - to be sure all is good
there as well.
b) If all test also pass for you - maybe if you make a bit of noise
python may be a bit quicker to actually merge them.

Hope this helps (AIX :) )

Michael

Links:
https://github.com/python/cpython/pulls?q=is%3Apr+is%3Aopen+AIX+author%3Aaixtools
https://github.com/aixtools/cpython/tree/aix-pr
msg325656 - (view) Author: Michael Felt (Michael.Felt) * Date: 2018-09-18 15:08
On 18/09/2018 16:46, Michael Felt wrote:
> Michael Felt <aixtools@felt.demon.nl> added the comment:

Ignore this. If only I could remove stuff!

Good day all.
History
Date User Action Args
2018-09-18 15:08:46Michael.Feltsetmessages: + msg325656
2018-09-18 14:46:05Michael.Feltsetmessages: + msg325654
title: AIX shared library extension modules installation broken: wrong dir names -> AIX shared library extension modules installation broken
2017-01-13 21:30:15martin.pantersettitle: AIX shared library extension modules installation broken -> AIX shared library extension modules installation broken: wrong dir names
2016-09-13 13:36:48martin.panterlinkissue16189 superseder
2016-09-05 01:19:38martin.pantersetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: - Python 2.7
2016-09-04 18:24:48David.Edelsohnsetmessages: + msg274375
2016-08-27 07:20:19martin.pantersetmessages: + msg273766
2016-08-27 04:10:54python-devsetmessages: + msg273756
2016-08-23 14:00:09David.Edelsohnsetmessages: + msg273448
2016-08-23 08:56:15martin.pantersetmessages: + msg273425
2016-08-23 03:11:03David.Edelsohnsetmessages: + msg273417
2016-08-23 03:02:19martin.pantersetmessages: + msg273416
stage: needs patch -> patch review
2016-08-22 17:59:09ericvwsetmessages: + msg273389
2016-08-20 21:34:43ericvwsetnosy: + ericvw
2016-08-13 17:56:50David.Edelsohnsetfiles: + Issue25825-patch2.txt

messages: + msg272602
2016-08-12 13:27:39python-devsetnosy: + python-dev
messages: + msg272539
2016-08-12 13:08:46David.Edelsohnsetmessages: + msg272538
2016-08-12 02:58:01martin.pantersetmessages: + msg272490
2016-08-10 15:22:52David.Edelsohnsetfiles: + Issue25825-patch1.txt

messages: + msg272336
2016-07-26 22:05:54Michael.Feltsetmessages: + msg271417
2016-03-06 02:16:16martin.pantersetstage: needs patch
2016-03-02 16:42:41Michael.Feltsetnosy: + Michael.Felt
messages: + msg261127
2015-12-09 13:59:17David.Edelsohnsetmessages: + msg256155
2015-12-09 04:38:43martin.pantersetnosy: + martin.panter

messages: + msg256136
versions: - Python 3.2, Python 3.3, Python 3.4
2015-12-08 19:33:19David.Edelsohnsetnosy: + pitrou
2015-12-08 19:29:34David.Edelsohncreate