The commit 1ec63b62035e73111e204a0e03b83503e1c58f2e of bpo-39763 broke the Python compilation on AIX:
https://bugs.python.org/issue39763#msg363749

--

The last successful build was before the commit 1ec63b62035e73111e204a0e03b83503e1c58f2e:
https://buildbot.python.org/all/#/builders/119/builds/383

_socket compilation:
(...) -o build/lib.aix-7100-9898-32-3.9-pydebug/_socket.so

pythoninfo:

sys.path: [
  '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build',
  '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/target/lib/python39.zip',
  '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib',
  '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/build/lib.aix-7100-9898-32-3.9-pydebug',
  '/home/buildbot/.local/lib/python3.9/site-packages']

Both steps use "build/lib.aix-7100-9898-32-3.9-pydebug/" directory.

--

Recent failure:
https://buildbot.python.org/all/#/builders/119/builds/451

_socket compilation:

(...) -o build/lib.aix-7104-1806-32-3.9-pydebug/_socket.so

pythoninfo:

sys.path: [
  '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build',
  '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/target/lib/python39.zip',
  '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib',
  '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/build/lib.aix-7100-9898-32-3.9-pydebug',
  '/home/buildbot/.local/lib/python3.9/site-packages']

So the compilation uses "build/lib.aix-7104-1806-32-3.9-pydebug/" directory, whereas pythoninfo uses "build/lib.aix-7100-9898-32-3.9-pydebug/" directory.

It can explain why setup.py fails to import _socket later: it was built in a different directory.

--

I see that _aix_support._aix_bosmp64() has two code paths depending on the subprocess module can be imported:

# subprocess is not necessarily available early in the build process
# if not available, the config_vars are also definitely not available
# supply substitutes to bootstrap the build
try:
    import subprocess
    _have_subprocess = True
    _tmp_bd = get_config_var("AIX_BUILDDATE")
    _bgt = get_config_var("BUILD_GNU_TYPE")
except ImportError:  # pragma: no cover
    _have_subprocess = False
    _tmp_bd = None
    _bgt = "powerpc-ibm-aix6.1.7.0"

def _aix_bosmp64():
    # type: () -> Tuple[str, int]
    """
    Return a Tuple[str, int] e.g., ['7.1.4.34', 1806]
    The fileset bos.mp64 is the AIX kernel. It's VRMF and builddate
    reflect the current ABI levels of the runtime environment.
    """
    if _have_subprocess:
        # We expect all AIX systems to have lslpp installed in this location
        out = subprocess.check_output(["/usr/bin/lslpp", "-Lqc", "bos.mp64"])
        out = out.decode("utf-8").strip().split(":")  # type: ignore
        # Use str() and int() to help mypy see types
        return str(out[2]), int(out[-1])
    else:
        from os import uname

        osname, host, release, version, machine = uname()
        return "{}.{}.0.0".format(version, release), _MISSING_BD

--

_aix_support._aix_bosmp64() is called by _aix_support.aix_platform() which is called by get_host_platform() of distutils.util.

Currently, setup.py does:

* Inject _bootsubprocess into sys.modules['subprocess'] so "import subprocess" works
* Build all C extensions
* Remove sys.modules['subprocess'], so the next "import subprocess" may or may not load Lib/subprocess.py which uses the newly built C extensions like _posixsubprocess and select
* Attempt to load C extensions: if an import fails, rename the file: it happens for _asyncio on AIX, that's the bug

AIX_BUILDDATE variable is written in pyconfig.h by configure:

> https://buildbot.python.org/all/#/builders/119/builds/383

configure: "checking for the system builddate... 1806"

> https://buildbot.python.org/all/#/builders/119/builds/451

configure: "checking for the system builddate... 1806"

So AIX_BUILDDATE seems to be set in both case.

But on build 451, the "build/lib.aix-7100-9898-32-3.9-pydebug" directory name contains 9898: this number comes from _aix_support._MISSING_BD. It's used when _have_subprocess is false or when get_config_var("AIX_BUILDDATE") cannot be converted into an integer.

While looking through the history of bot builds - I consistently see 420 tests being done on one bot - but test_socket does not always show up in the list.

I'll look at this as I can, but "free-time" is limited. Delay is not a lack of interest.

Michael.Felt: Can you please try to build the master branch of Python with the attached setup-aix.patch applied?

Something like:

git checkout master
git apply setup-aix.patch
./configure CFLAGS="-O0"
make 2>&1|tee make.log

Then attach make.log.

Another approach to patch _aix_support.py, replace:

try:
    import subprocess
    _have_subprocess = True
    _tmp_bd = 
    _bgt = get_config_var("BUILD_GNU_TYPE")
except ImportError:  # pragma: no cover
    _have_subprocess = False
    _tmp_bd = None
    _bgt = "powerpc-ibm-aix6.1.7.0"

with:

try:
    import subprocess
    _have_subprocess = True
    _tmp_bd = get_config_var("AIX_BUILDDATE")
    _bgt = get_config_var("BUILD_GNU_TYPE")
except ImportError:  # pragma: no cover
    import _bootsubprocess as subprocess
    _have_subprocess = True
    _tmp_bd = None
    _bgt = "powerpc-ibm-aix6.1.7.0"

Use _bootsubprocess when subprocess is not available. If this approach works, _have_subprocess=False code path can be removed.

I tried your patch on AIX 7.2, looks like you need to shift sys import to the top


Traceback (most recent call last):
  File "/home/isidentical/cpython/./setup.py", line 16, in <module>
    sys.modules['subprocess'] = _bootsubprocess
NameError: name 'sys' is not defined
make: The error code from the last command is 1.


Stop.

Yeah, sys must be imported first. Did you try to fix the patch? Does it work?

I'll take a look at what you are suggesting.

The starting point (before the rm command) is the make command that I run again.

What I notice - read am thinking - is that _socket.so is being created by the "setup.py build" - so, if you can help me make that bit more verbose, I'll work on that too.

+++
Script command is started on Wed Mar 11 15:29:44 CDT 2020.
$
$
$
$ rm ./build/lib.aix-7200-1543-32-3.9/_socket.so ./build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/socketmodule.o
$
$
$ make V=1
 CC='gcc -pthread' LDSHARED='Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp    ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall'  _TCLTK_INCLUDES='' _TCLTK_LIBS=''  ./python -E ./setup.py  build
running build
running build_ext
ldd: /lib/libreadline.a: File is an archive.
building '_asyncio' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include
 -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_asynciomodule.c -o  build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/_asynciomodule.o
Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/_asynciomodule.o -o build/lib.aix-7200-1543-32-3.9/_asyncio.so
building '_socket' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include
 -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/socketmodule.c -o build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/socketmodule.o
Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/socketmodule.o -o build/lib.aix-7200-1543-32-3.9/_socket.so
*** WARNING: renaming "_asyncio" since importing it failed: No module named '_socket'

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Following modules built successfully but were removed because they could not be imported:
_asyncio

So, with the patch - the process stops at:

aixtools@x064:[/home/aixtools/python-3.9]make
 CC='xlc_r' LDSHARED='Modules/ld_so_aix xlc_r -bI:Modules/python.exp    ' OPT='-DNDEBUG -O'  _TCLTK_INCLUDES='' _TCLTK_LIBS=''  ./python -E ./setup.py  build
Traceback (most recent call last):
  File "/home/aixtools/python-3.9/Lib/subprocess.py", line 72, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aixtools/python-3.9/./setup.py", line 4, in <module>
    import subprocess
  File "/home/aixtools/python-3.9/Lib/subprocess.py", line 77, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aixtools/python-3.9/./setup.py", line 16, in <module>
    sys.modules['subprocess'] = _bootsubprocess
NameError: name 'sys' is not defined
make: 1254-004 The error code from the last command is 1.


Stop.

So, this is what I have on screen. Will add the log in a moment.

[1] +  Done                    make 2>&1|tee make.log &
aixtools@x064:[/home/aixtools/python-3.9]find . | grep socket
./Doc/howto/sockets.rst
./Doc/library/socket.rst
./Doc/library/socketserver.rst
./Lib/__pycache__/socket.cpython-39.opt-1.pyc
./Lib/__pycache__/socket.cpython-39.opt-2.pyc
./Lib/__pycache__/socket.cpython-39.opt-4.pyc
./Lib/__pycache__/socket.cpython-39.pyc
./Lib/__pycache__/socketserver.cpython-39.pyc
./Lib/socket.py
./Lib/socketserver.py
./Lib/test/__pycache__/mock_socket.cpython-39.pyc
./Lib/test/__pycache__/test_socket.cpython-39.pyc
./Lib/test/__pycache__/test_socketserver.cpython-39.pyc
./Lib/test/mock_socket.py
./Lib/test/test_socket.py
./Lib/test/test_socketserver.py
./Modules/socketmodule.c
./Modules/socketmodule.h
./PCbuild/_socket.vcxproj
./PCbuild/_socket.vcxproj.filters
./build/lib.aix-7104-1806-32-3.9/_socket.so
./build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o
aixtools@x064:[/home/aixtools/python-3.9]rm ./build/lib.aix-7104-1806-32-3.9/_socket.so ./build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o

aixtools@x064:[/home/aixtools/python-3.9]make V=1
 CC='xlc_r' LDSHARED='Modules/ld_so_aix xlc_r -bI:Modules/python.exp    ' OPT='-DNDEBUG -O'  _TCLTK_INCLUDES='' _TCLTK_LIBS=''  ./python -E ./setup.py  build
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
building '_asyncio' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/_asynciomodule.c -o build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o -o build/lib.aix-7104-1806-32-3.9/_asyncio.so
building '_socket' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/socketmodule.c -o build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o
    1500-030: (I) INFORMATION: PyInit__socket: Additional optimization may be attained by recompiling and specifying MAXMEM option with a value greater than 8192.
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o -o build/lib.aix-7104-1806-32-3.9/_socket.so
*** WARNING: renaming "_asyncio" since importing it failed: No module named '_socket'

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm                 _lzma                 _sqlite3
_tkinter              readline
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Following modules built successfully but were removed because they could not be imported:
_asyncio

OK. I removed the _aix_support.py from the formula.

After make see the new (rather) old build paths for "socket"

aixtools@x064:[/home/aixtools/python-3.9]find . | grep socket
./Doc/howto/sockets.rst
./Doc/library/socket.rst
./Doc/library/socketserver.rst
./Lib/__pycache__/socket.cpython-39.opt-1.pyc
./Lib/__pycache__/socket.cpython-39.opt-2.pyc
./Lib/__pycache__/socket.cpython-39.opt-4.pyc
./Lib/__pycache__/socket.cpython-39.pyc
./Lib/__pycache__/socketserver.cpython-39.pyc
./Lib/socket.py
./Lib/socketserver.py
./Lib/test/__pycache__/mock_socket.cpython-39.pyc
./Lib/test/__pycache__/test_socket.cpython-39.pyc
./Lib/test/__pycache__/test_socketserver.cpython-39.pyc
./Lib/test/mock_socket.py
./Lib/test/test_socket.py
./Lib/test/test_socketserver.py
./Modules/socketmodule.c
./Modules/socketmodule.h
./PCbuild/_socket.vcxproj
./PCbuild/_socket.vcxproj.filters
./build/lib.aix-7.1-3.9/_socket.so
./build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/socketmodule.o

I remove these two files again and run make:

aixtools@x064:[/home/aixtools/python-3.9]make
 CC='xlc_r' LDSHARED='Modules/ld_so_aix xlc_r -bI:Modules/python.exp    ' OPT='-DNDEBUG -O'  _TCLTK_INCLUDES='' _TCLTK_LIBS=''  ./python -E ./setup.py  build
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
building '_asyncio' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/_asynciomodule.c -o build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o -o build/lib.aix-7.1-3.9/_asyncio.so
building '_socket' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/socketmodule.c -o build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/socketmodule.o
    1500-030: (I) INFORMATION: PyInit__socket: Additional optimization may be attained by recompiling and specifying MAXMEM option with a value greater than 8192.
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/socketmodule.o -o build/lib.aix-7.1-3.9/_socket.so
*** WARNING: renaming "_asyncio" since importing it failed: No module named '_socket'

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm                 _lzma                 _sqlite3
_tkinter              readline
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Following modules built successfully but were removed because they could not be imported:
_asyncio

The diff!

diff --git a/Lib/_aix_support.py b/Lib/_aix_support.py
index 2c5cd3297d..c7f4491633 100644
--- a/Lib/_aix_support.py
+++ b/Lib/_aix_support.py
@@ -12,7 +12,8 @@ try:
     _tmp_bd = get_config_var("AIX_BUILDDATE")
     _bgt = get_config_var("BUILD_GNU_TYPE")
 except ImportError:  # pragma: no cover
-    _have_subprocess = False
+    import _bootsubprocess as subprocess
+    _have_subprocess = True
     _tmp_bd = None
     _bgt = "powerpc-ibm-aix6.1.7.0"

diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py
index 4b002ecef1..513af52ecd 100644
--- a/Lib/distutils/util.py
+++ b/Lib/distutils/util.py
@@ -79,8 +79,9 @@ def get_host_platform():
             machine += ".%s" % bitness[sys.maxsize]
         # fall through to standard osname-release-machine representation
     elif osname[:3] == "aix":
-        from _aix_support import aix_platform
-        return aix_platform()
+        return "%s-%s.%s" % (osname, version, release)
+        # from _aix_support import aix_platform
+        # return aix_platform()
     elif osname[:6] == "cygwin":
         osname = "cygwin"
         rel_re = re.compile (r'[\d.]+', re.ASCII)
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
index 4003726dc9..b116e96007 100644
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -666,8 +666,9 @@ def get_platform():
             machine += ".%s" % bitness[sys.maxsize]
         # fall through to standard osname-release-machine representation
     elif osname[:3] == "aix":
-        from _aix_support import aix_platform
-        return aix_platform()
+        return "%s-%s.%s" % (osname, version, release)
+        # from _aix_support import aix_platform
+        # return aix_platform()
     elif osname[:6] == "cygwin":
         osname = "cygwin"
         import re

IMHO - this problem has nothing to do with _aix_support.py - That issue was fixed earlier by the new _bootsubprocess.

Michael: this issue is about bootstraping Python. If you want to test a patch or test something else, you must restart from a clean copy of the source code. Either use "make distclean", "git clean -fdx", or recreate the source directory (git clone, decompress an archive, etc.). Then restart from scratch: ./configure && make.

> NameError: name 'sys' is not defined

Right, that's a stupid bug in my patch. You must import sys before.

Try to replace Lib/_aix_support.py with attached _aix_support.py: in short, it uses _bootsubprocess if subprocess is not available.

Actually, I had already done that:

diff --git a/Lib/_aix_support.py b/Lib/_aix_support.py
index 2c5cd3297d..c7f4491633 100644
--- a/Lib/_aix_support.py
+++ b/Lib/_aix_support.py
@@ -12,7 +12,8 @@ try:
     _tmp_bd = get_config_var("AIX_BUILDDATE")
     _bgt = get_config_var("BUILD_GNU_TYPE")
 except ImportError:  # pragma: no cover
-    _have_subprocess = False
+    import _bootsubprocess as subprocess
+    _have_subprocess = True
     _tmp_bd = None
     _bgt = "powerpc-ibm-aix6.1.7.0"

And after that test, I tried with no calls to _aix_support.py

diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py
index 4b002ecef1..513af52ecd 100644
--- a/Lib/distutils/util.py
+++ b/Lib/distutils/util.py
@@ -79,8 +79,9 @@ def get_host_platform():
             machine += ".%s" % bitness[sys.maxsize]
         # fall through to standard osname-release-machine representation
     elif osname[:3] == "aix":
-        from _aix_support import aix_platform
-        return aix_platform()
+        return "%s-%s.%s" % (osname, version, release)
+        # from _aix_support import aix_platform
+        # return aix_platform()
     elif osname[:6] == "cygwin":
         osname = "cygwin"
         rel_re = re.compile (r'[\d.]+', re.ASCII)
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
index 4003726dc9..b116e96007 100644
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -666,8 +666,9 @@ def get_platform():
             machine += ".%s" % bitness[sys.maxsize]
         # fall through to standard osname-release-machine representation
     elif osname[:3] == "aix":
-        from _aix_support import aix_platform
-        return aix_platform()
+        return "%s-%s.%s" % (osname, version, release)
+        # from _aix_support import aix_platform
+        # return aix_platform()
     elif osname[:6] == "cygwin":
         osname = "cygwin"
         import re

±±±±±±±±

So, even when _aix_support.py is not involved at all, _socket.so is being ignored - even though it was built.

re: Michael: this issue is about bootstraping Python. If you want to test a patch or test something else, you must restart from a clean copy of the source code. Either use "make distclean", "git clean -fdx", or recreate the source directory (git clone, decompress an archive, etc.). Then restart from scratch: ./configure && make.

Understood: between these two timestamps: Date: 2020-03-11 18:55 and Date: 2020-03-11 21:36 - I was doing just that on 4 different VM's: AIX 5.3, 6.1, 7.1 and 7.2 - just in case there may be anything specific re: AIX versions (the bots run on AIX 7.1 and 7.2). I also switched away from any NFS based builds because test behaves differently when the build area is an NFS area compared to jfs2.

FYI: I have decided to focus on AIX 7.1 and AIX 7.2 - and am only using gcc (I was also testing runs with both xlc and gcc before I started posting).

Further, I like your suggestion to try: subprocess except _bootsubprocess and shall make a PR to that end. Much more clear about what is going on.

What I am going to do bewtween now and my next post is again - clean slate - update master to latest version - on two servers.

On one server I'll apply your suggestion to _aix_platform.py; on the other I'll patch Lib/distutils/util.py and Lib/sysconfig.py to not even call _aix_platform.py. Ideally, there will be a difference that leads to an understanding of the root cause.

The good news! Your patch, better rewrite, of _aix_platform.py is working!

Many thanks!

I converted attached _aix_support.py into PR 18970 (with minor changes).

New changeset c846ef004d79ee8e9645d3e5e8b3b0cb97b5013f by Victor Stinner in branch 'master':
bpo-39936: _aix_support uses _bootsubprocess (GH-18970)
https://github.com/python/cpython/commit/c846ef004d79ee8e9645d3e5e8b3b0cb97b5013f

Fantastic! Many thanks!

POWER6 AIX 3.x and PPC64 AIX 3.x buildbots are back to green.