AIX requires helper scripts to build Python shared extension modules.  The definitions and Makefile installation rules have bitrotted.

Makefile.pre.in:

      @if [ -s Programs/python.exp -a \

except python.exp is created in Modules/python.exp, not Programs.exp

               $(INSTALL_DATA) Programs/python.exp             \
                               $(DESTDIR)$(LIBPL)/python.exp;

Post-substitution Makefile.pre:

BINLIBDEST=     $(LIBDIR)/python$(VERSION)
LIBDEST=        $(SCRIPTDIR)/python$(VERSION)
LIBPL=          $(LIBDEST)/config-$(LDVERSION)

configure.ac:

        AIX*)
                BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) -bI:\$(srcdir)/Modules/python.exp"
                LDSHARED="\$(BINLIBDEST)/config/ld_so_aix \$(CC) -bI:\$(BINLIBDEST)/config/python.exp"

which is forever enshrined in _sysconfigdata.py

In other words, configure sets LDSHARED to $(BINLIBDEST)/config, which does not exist in current installations.  And Makefile installs the files in LIBPL, which is based on LIBDEST (prefix and exec_prefix could be different).

And the files are not installed, because the installation tests Programs/python.exp instead of Modules/python.exp.

Changing Makefile.pre.in to test Modules/python.exp is easy enough as a partial fix.

What is the preferred location to install the files so that configure.ac and Makefile.pre.in can be adjusted to match each other?

It looks like the Programs/python.exp change was done accidentally in revision 88a532a31eb3 (Issue 18093), so we should be safe reverting the relevant lines back to Modules/python.exp.

The configure script currently sets

LIBPL='$(prefix)'"/lib/python${VERSION}/config-${LDVERSION}"
AC_SUBST(LIBPL)

However there is a contradictory comment in the makefile:

# This goes into $(exec_prefix)
LIBPL=  @LIBPL@

It looks like the comment came first. $(LIBPL) was changed from $(LIBDIR) [presumably using $(exec_prefix)] to use $(prefix) in r86731 (Issue 9807).

This isn’t my area of expertise, but my understanding is $(exec_prefix) is for architecture-specific files only. If python.exp is not architecture-specific, perhaps it should go in $(prefix), and the configure script (and makefile comment) needs fixing.

$(prefix) and $(exec_prefix) result in the same path on AIX, so it does not matter in practice, although the semantics are different.

# Install prefix for architecture-dependent files
exec_prefix=    ${prefix}

python.exp is not architecture dependent, although it only is useful on AIX target.  It is essentially equivalent to a list of symbols with ELF global, non-hidden visibility.  It is less confusing if the list is co-located with the scripts that use it.

LIBPL is fine with me.  configure.ac and Makefile.pre.in must match.

FYI: when build and src are the same directory, there is no error message. However, when src and build are in seperate directories (e.g.,

build = '.',
src = '../src/python-2.7.11'

the following message occurs MANY times:

unable to execute '../src/python-2.7.11/Modules/ld_so_aix': No such file or directory

during configure the file
./Modules/ld_so_aix is 'created' in the build area.

root@x064:[/data/prj/aixtools/python-2.7.11]ls -l Modules/ld*aix*
-rwxr-xr-x   1 root     system         6205 Mar 02 16:24 Modules/ld_so_aix

Just guessing, but a line such as:
BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) -bI:\$(srcdir)/Modules/python.exp"

using $(srcdir) is not correct.

Further, a find in $(srcdir) for ld_so_aix returns:
root@x064:[/data/prj/aixtools/python-2.7.11]ls -dl ../src/py*
drwxr-xr-x  17 199      1954           4096 May 23 2015  ../src/python-2.7.10
drwxr-xr-x  17 199      1954           4096 Dec 05 19:47 ../src/python-2.7.11

So, $(srcdir) seems to be wrong, by definition.

For now, I am copying $(srcdir) to $(builddir) to work-around this.

hmm. needs patch. For what?

One comment says a change will be undone.

If a patch is expected for src/builddir issue:

a) I sinned - that is a new issue, and should be posted seperately.
b) i do not know autotools well enough to solve the srcdir and builddir problem.

rather than a patch i can on that is see if the same issue occurs with linux and post a new issue.

So, my question would be - was a rollback done - if yes, perhaps a comment to verify and then close this issue.

Let's start with this patch to revert the change mentioned in msg256136 to look for python.exp in Modules -- the test, the source for the file, and the location to delete the file.

A follow-up patch will fix the data in _sysconfigdata.py.

Patch1 looks fine to me, though I will have to rely on you people to verify that it does what it’s supposed to. Do you want me to commit it straight away, or wait for your follow-up patch?

Yes, please apply Patch 1 that reverts the mistaken change of revision 88a532a31eb3 .  I want to work through this incrementally so that it's clear to reviewers.

New changeset 58c8cae6c61a by Martin Panter in branch '3.5':
Issue #25825: Fix references to Modules/python.exp
https://hg.python.org/cpython/rev/58c8cae6c61a

New changeset 4d4b5b978b7e by Martin Panter in branch 'default':
Issue #25825: Merge AIX fix from 3.5
https://hg.python.org/cpython/rev/4d4b5b978b7e

The second of two patches. This patch changes the definition of LDSHARED for AIX in configure to reference the matching installed location as defined in Makefile.pre.in by Patch1.  The definition from configure propagates into _sysconfigdata.py.

This change will (further) break test_distutils, but the testcase is wrong on AIX. And, for AIX, a correct test result clearly does not correspond to correct behavior.

The patch affects both configure.ac and configure because the corresponding change to configure is obvious. The person who installs the patch may regenerate configure, as appropriate.

> Patch1 looks fine to me, though I will have to rely on you people to verify that it does what it’s supposed to. Do you want me to commit it straight away, or wait for your follow-up patch?

Independently, I have created a similar patch as well and can also verify that it does what it's suppose to do.

> The second of two patches. This patch changes the definition of LDSHARED for AIX in configure to reference the matching installed location as defined in Makefile.pre.in by Patch1. [...]

I can also confirm that Patch2 is necessary as well and does what it's suppose to do, having creating this patch independently as well.

It looks like the change from plain config to config-<ABI> was made in Issue 9807 (r86731, 3.2). So the patch seems reasonable to me; it is just catching up with that change.

This bug was marked for 2.7 as well. Is there anything that needs to be done for 2.7?

How does patch 2 make the test_distutils situation worse? Is there anything that should be done to improve things first, or should that just be handled independently? I see that the 2.7 and 3.5 buildbots are currently failing with errors like

distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

while 3.6 is failing with

distutils.errors.LinkError: command '/usr/local/lib/python3.6/config/ld_so_aix' failed with exit status 1

Presumably the 3.6 failure may be helped by Patch 2, since it is complaining about the plain “config” path.

> This bug was marked for 2.7 as well. Is there anything that needs to be done for 2.7?

It would be great if both patches were applied to 2.7 also.

> How does patch 2 make the test_distutils situation worse? Is there anything that should be done to improve things first, or should that just be handled independently? I see that the 2.7 and 3.5 buildbots are currently failing with errors like

distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

while 3.6 is failing with

distutils.errors.LinkError: command '/usr/local/lib/python3.6/config/ld_so_aix' failed with exit status 1

Presumably the 3.6 failure may be helped by Patch 2, since it is complaining about the plain “config” path.

Patch2 makes test_distutils worse because LDSHARED refers to a file that only exists when Python is installed.  If one tests in tree, the files don't exist.  This may be related to the existence of BLDSHARED, used to build modules in the source tree, which can be overridden separately -- otherwise it defaults to LDSHARED.  If the rest of the testsuite ever is cleaned up for AIX, one either could skip the parts of test_distutils that require the LDSHARED files or add special hooks in the test for AIX.  As you point out, the change doesn't make the current testsuite results situation any worse.

Patch 1 does not apply to 2.7. The offending commit 88a532a31eb3 was not made in that branch.

Patch 2 may apply to 2.7, but I am not sure if it is needed or worthwhile. Again, the offending commit r86731 not made to 2.7. There is also a $(BINLIBDEST)/config reference for Be OS; surely if one is updated the other should too.

David: I don’t understand “Patch2 makes test_distutils worse because LDSHARED refers to a file that only exists when Python is installed.” At the moment, doesn’t LDSHARED refer to a file that never exists? So at least the test will be improved when Python is installed, right?

I looked into the test_distutils failures and here’s what I found:

* Revision c958678720fd assigns BLDSHARED → LDSHARED in distutils if the source tree is detected. This survived in the 3.5 branch, so may explain why the 3.5 buildbot failure mentions “./Modules/ld_so_aix”.

* In 2.7, revision fa69e891edf4 moved the assignment into sysconfig.py.

* Then be3b4aa2ad28 loads LDSHARED from _sysconfigdata.py, which is created by sysconfig.py with the alternative BLDSHARED → LDSHARED value. This explains “./Modules/ld_so_aix” in 2.7.

* In 3.6, c554194240fc (Issue 18235) reversed the assignment in sysconfig.py, which now assigns LDSHARED → BLDSHARED, and creates _sysconfigdata.py with this alternative value.

* Then 3fa8aebed636 loads LDSHARED directly from _sysconfigdata.py. In this case it is the original LDSHARED value, so the 3.6 buildbot failure refers to /usr/local/lib/python3.6/config/ld_so_aix.

$(BINLIBDEST)/config is equivalent to $(LIBPL) in Python 2.7, so Python 2.7 should be okay.

Patch2 will not make test_distutils results worse, but the test results may not represent the true status of distutils on AIX if the matching Python version is not installed so that the test can find the files in the installed location.

New changeset ca1ddd365f5f by Martin Panter in branch '3.5':
Issue #25825: Fix references to $(LIBPL) installation path on AIX
https://hg.python.org/cpython/rev/ca1ddd365f5f

New changeset 5a05c0eeefc3 by Martin Panter in branch 'default':
Issue #25825: Merge $(LIBPL) fix from 3.5
https://hg.python.org/cpython/rev/5a05c0eeefc3

Okay, the second patch is committed to 3.5+. Is everything working now (on 2.7, 3.5, 3.6), or is there more to do?

I believe that everything is functioning correctly.

On 10/08/2016 17:22, David Edelsohn wrote:
> David Edelsohn added the comment:
Hi, Just thought I would mention that I have a branch with all the test
PR fixes I have made in the last two months.

a) would appreciate your testig them with gcc - to be sure all is good
there as well.
b) If all test also pass for you - maybe if you make a bit of noise
python may be a bit quicker to actually merge them.

Hope this helps (AIX :) )

Michael

Links:
https://github.com/python/cpython/pulls?q=is%3Apr+is%3Aopen+AIX+author%3Aaixtools
https://github.com/aixtools/cpython/tree/aix-pr

On 18/09/2018 16:46, Michael Felt wrote:
> Michael Felt <aixtools@felt.demon.nl> added the comment:

Ignore this. If only I could remove stuff!

Good day all.