Since python-3.2, there is a race condition building in parallel on AIX:

Consider these Makefile(.pre.in) rules:

$(BUILDPYTHON): ...
  $(LINKCC) ... $(LINKFORSHARED) ...

Modules/_testembed: ...
  $(LINKCC) ... $(LINKFORSHARED) ...

Modules/_freeze_importlib: ...
  $(LINKCC) ...

On AIX, the variables get these values:

LINKCC = $(srcdir)/Modules/makexp_aix Modules/python.exp ...
LINKFORSHARED = -Wl,-bE:Modules/python.exp ...

Now $(BUILDPYTHON) and Modules/_testembed may run in parallel, causing Modules/python.exp to be created by two instances of makexp_aix eventually running at the same time.

Attached patch fixes this problem for cpython tip (doubt supporting AIX 4.1 and earlier still is necessary).

Thank you!

Wouldn't it be better if linking _testembed generated _testembed.exp instead of generating python.exp? I hope using $@.exp somehow could help. Hard-coding the name of the export file sounds like a flaw in the first place.

I'm unsure about the real purpose of _testembed, but given the name it does make sense to me to export the same symbols as $(BUILDPYTHON), thus reusing python.exp.

+1

Patch including configure update now.

I also have found this goes back since Python 2.7.

I have refreshed the patched for the tip of CPython.  What can I do to help push this forward?

I may be able to simplify the build on AIX by removing ld_so_aix and python.exp entirely.  Would this be a preferred solution if I am able to get something working?  If so, should I create a separate issue to track the change?

The original patch is a bit dated, do we still need the export symbol generation?

https://www.ibm.com/support/knowledgecenter/en/SSGH3R_16.1.0/com.ibm.xlcpp161.aix.doc/proguide/compiling_shared_aix.html

"Use the -G option to create a shared library from the generated object files, and to enable runtime linking with applications that support it."

"If you do not specify a -bE option, all the global symbols are exported except for those symbols that have the hidden or internal visibility attribute."

Yes, export file generation still is required.

Python does not need to utilize runtime linking.  Using -G is a very bad choice and severely discouraged with severe performance and other penalties.

Okay, thanks.  The -G option is of course attractive for Linux projects
because it requires minimal changes in the build machinery.

I've added support for libmpdec/libmpdec++ (next release) for AIX-style
shared libraries (shr.o inside libmpdec.a).  The AIX linker has a certain
elegance, but it requires something like 150 lines of code changes and
conditionals inside the Makefiles.

I can confirm that -G is substantially slower, so I'm going to commit this
patch soon.

I can't find the reason for:

        if test -z "$EXPORTSYMS"; then
            EXPORTSYMS="Modules/python.exp"
        fi


Modules/python.exp is hardcoded elsewhere, and I'd rather set
EXPORTSYMS unconditionally on AIX and unset it unconditionally
for all other systems (which don't build with the patch if
EXPORTSYMS happens to be defined).

New changeset e6dcd371b2c54a94584dd124e8c592a496d46a47 by Stefan Krah in branch 'master':
bpo-19521: Fix parallel build race condition on AIX (GH-21997)
https://github.com/python/cpython/commit/e6dcd371b2c54a94584dd124e8c592a496d46a47

New changeset 88b86a9752afc2c50ca196f6ba1a8d62d71cf398 by Miss Islington (bot) in branch '3.9':
bpo-19521: Fix parallel build race condition on AIX (GH-22001)
https://github.com/python/cpython/commit/88b86a9752afc2c50ca196f6ba1a8d62d71cf398

This is in master and 3.9.1.  I'll not backport to 3.8 because a release candidate is imminent.

I have been asked to backport this to 3.8. There's a very small window
of opportunity:

3.8.6: Monday, 2020-09-21
3.8.7rc1: Monday, 2020-11-02
3.8.7: Monday, 2020-11-16     (final version!)


Backporting procedures since 3.8 are unclear and a source of
constant friction, so most core developers seem to have stopped
doing any backports at all.  It is not a battle I'll choose, but
if you get a second core dev to review this trivial patch I'll
commit it.

There's a simple solution for 3.8: Do not use the parallel build,
the regular build takes around 4 min.


For the buildbots you can ask the operator for a custom command
line.