My AIX bot has been very consistent - only the multiprocessing tests failing when run by bot, but 4 or 5 days ago 3 to 5 additional tests - that, afaik, had never failed before, are now failing.

These may also be compiler related specifics, or the presence (better lack of) 3rd party packages.

Anyway - summary of output:

INFO: Can't locate Tcl/Tk libs and/or headers
building '_pickle' extension
xlc_r -O2 -I/opt/include -I/opt/buildaix/include -g -I/opt/include -I/opt/buildaix/include -g -I./Include/internal -I./Include -I. -I/opt/include -I/opt/buildaix/include -I/home/buildbot/cpython-master/Include -I/home/buildbot/cpython-master -c /home/buildbot/cpython-master/Modules/_pickle.c -o build/temp.aix-7.1-3.8-pydebug/home/buildbot/cpython-master/Modules/_pickle.o -D Py_BUILD_CORE_MODULE
1506-261 (W) Suboption  Py_BUILD_CORE_MODULE is not valid for option D.
"/home/buildbot/cpython-master/Modules/_pickle.c", line 8.4: 1506-205 (S) #error "Py_BUILD_CORE_BUILTIN or Py_BUILD_CORE_MODULE must be defined"
building '_json' extension
xlc_r -O2 -I/opt/include -I/opt/buildaix/include -g -I/opt/include -I/opt/buildaix/include -g -I./Include/internal -I./Include -I. -I/opt/include -I/opt/buildaix/include -I/home/buildbot/cpython-master/Include -I/home/buildbot/cpython-master -c /home/buildbot/cpython-master/Modules/_json.c -o build/temp.aix-7.1-3.8-pydebug/home/buildbot/cpython-master/Modules/_json.o -D Py_BUILD_CORE_MODULE
1506-261 (W) Suboption  Py_BUILD_CORE_MODULE is not valid for option D.
"/home/buildbot/cpython-master/Modules/_json.c", line 8.4: 1506-205 (S) #error "Py_BUILD_CORE_BUILTIN or Py_BUILD_CORE_MODULE must be defined"
"./Include/internal/pycore_accu.h", line 13.4: 1506-205 (S) #error "this header requires Py_BUILD_CORE define"
building '_testinternalcapi' extension
xlc_r -O2 -I/opt/include -I/opt/buildaix/include -g -I/opt/include -I/opt/buildaix/include -g -I./Include/internal -I./Include -I. -I/opt/include -I/opt/buildaix/include -I/home/buildbot/cpython-master/Include -I/home/buildbot/cpython-master -c /home/buildbot/cpython-master/Modules/_testinternalcapi.c -o build/temp.aix-7.1-3.8-pydebug/home/buildbot/cpython-master/Modules/_testinternalcapi.o -D Py_BUILD_CORE_MODULE
1506-261 (W) Suboption  Py_BUILD_CORE_MODULE is not valid for option D.
"/home/buildbot/cpython-master/Modules/_testinternalcapi.c", line 6.4: 1506-205 (S) #error "Py_BUILD_CORE_BUILTIN or Py_BUILD_CORE_MODULE must be defined"
"./Include/internal/pycore_coreconfig.h", line 8.4: 1506-205 (S) #error "this header requires Py_BUILD_CORE define"

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_curses_panel         _gdbm                 _tkinter
ossaudiodev           readline              spwd
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Failed to build these modules:
_json                 _pickle               _testinternalcapi

running build_scripts
copying and adjusting /home/buildbot/cpython-master/Tools/scripts/pydoc3 -> build/scripts-3.8
copying and adjusting /home/buildbot/cpython-master/Tools/scripts/idle3 -> build/scripts-3.8
copying and adjusting /home/buildbot/cpython-master/Tools/scripts/2to3 -> build/scripts-3.8
changing mode of build/scripts-3.8/pydoc3 from 644 to 755
changing mode of build/scripts-3.8/idle3 from 644 to 755
changing mode of build/scripts-3.8/2to3 from 644 to 755
renaming build/scripts-3.8/pydoc3 to build/scripts-3.8/pydoc3.8
renaming build/scripts-3.8/idle3 to build/scripts-3.8/idle3.8
renaming build/scripts-3.8/2to3 to build/scripts-3.8/2to3-3.8
         ./python -E -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(), *sys.version_info[:2]))' >platform
         ./python  ./Tools/scripts/run_tests.py
/home/buildbot/cpython-master/python -u -W default -bb -E -m test -r -w -j 0 -u all,-largefile,-audio,-gui
== CPython 3.8.0a3+ (heads/master:3e986de0d6, Apr 21 2019, 17:04:13) [C]
...
running: test_venv (3 min 38 sec), test_decimal (2 min 27 sec), test_ftplib (30 sec 50 ms), test_zipfile (1 min 43 sec)
0:04:00 [ 40/420/1] test_ftplib failed -- running: test_venv (3 min 53 sec), test_decimal (2 min 42 sec), test_zipfile (1 min 58 sec)
test test_ftplib failed -- Traceback (most recent call last):
  File "/home/buildbot/cpython-master/Lib/test/test_ftplib.py", line 605, in test_storlines
    self.client.storlines('stor', f)
  File "/home/buildbot/cpython-master/Lib/ftplib.py", line 526, in storlines
    conn.unwrap()
  File "/home/buildbot/cpython-master/Lib/ssl.py", line 1094, in unwrap
    s = self._sslobj.shutdown()
socket.timeout: The read operation timed out
0:04:05 [ 41/420/1] test_locale passed -- running: test_venv (3 min 58 sec), test_decimal (2 min 47 sec), test_zipfile (2 min 3 sec)
...
0:12:14 [110/420/2] test_embed failed -- running: test_multiprocessing_forkserver (1 min 25 sec), test_compileall (2 min 9 sec)
test test_embed failed -- multiple errors occurred; run in verbose mode for details
running: test_multiprocessing_forkserver (1 min 55 sec), test_faulthandler (30 sec 42 ms), test_compile (56 sec 571 ms), test_compileall (2 min 39 sec)
0:12:55 [111/420/2] test_faulthandler passed (36 sec 350 ms) -- running: test_multiprocessing_forkserver (2 min 6 sec), test_compile (1 min 7 sec), test_compileall (2 min             50 sec)
0:13:46 [125/420/3] test_json failed -- running: test_multiprocessing_forkserver (2 min 57 sec)
test test_json crashed -- Traceback (most recent call last):
  File "/home/buildbot/cpython-master/Lib/test/libregrtest/runtest.py", line 166, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/home/buildbot/cpython-master/Lib/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 777, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/buildbot/cpython-master/Lib/test/test_json/__init__.py", line 12, in <module>
    cjson.JSONDecodeError = cjson.decoder.JSONDecodeError = json.JSONDecodeError
AttributeError: 'NoneType' object has no attribute 'JSONDecodeError'
...
0:24:49 [212/420/5] test_pydoc failed -- running: test_pyclbr (1 min 2 sec), test_subprocess (11 min 1 sec)
test test_pydoc crashed -- Traceback (most recent call last):
  File "/home/buildbot/cpython-master/Lib/test/libregrtest/runtest.py", line 166, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/home/buildbot/cpython-master/Lib/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 777, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/buildbot/cpython-master/Lib/test/test_pydoc.py", line 9, in <module>
    import _pickle
ModuleNotFoundError: No module named '_pickle'
...
0:24:59 [214/420/6] test_pyclbr failed -- running: test_subprocess (11 min 12 sec)
*** Pickler
test test_pyclbr failed -- Traceback (most recent call last):
  File "/home/buildbot/cpython-master/Lib/test/test_pyclbr.py", line 227, in test_others
    cm('pickle', ignore=('partial',))
  File "/home/buildbot/cpython-master/Lib/test/test_pyclbr.py", line 144, in checkModule
    self.assertHaskey(dict, name, ignore)
  File "/home/buildbot/cpython-master/Lib/test/test_pyclbr.py", line 48, in assertHaskey
    self.assertIn(key, obj)
AssertionError: 'Pickler' not found in {'partial': <pyclbr.Class object at 0x306a1990>, 'PickleError': <pyclbr.Class object at 0x30834f30>, 'PicklingError': <pyclbr.Class             object at 0x30834e40>, 'UnpicklingError': <pyclbr.Class object at 0x30834f00>, '_Stop': <pyclbr.Class object at 0x30834f60>, '_Framer': <pyclbr.Class object at 0x30834f90>            , '_Unframer': <pyclbr.Class object at 0x30839150>, '_getattribute': <pyclbr.Function object at 0x308392a0>, 'whichmodule': <pyclbr.Function object at 0x30839300>, 'encode            _long': <pyclbr.Function object at 0x30839330>, 'decode_long': <pyclbr.Function object at 0x30839360>, '_Pickler': <pyclbr.Class object at 0x308392d0>, '_Unpickler': <pycl            br.Class object at 0x308398d0>, '_dump': <pyclbr.Function object at 0x30840750>, '_dumps': <pyclbr.Function object at 0x30840780>, '_load': <pyclbr.Function object at 0x30            8407b0>, '_loads': <pyclbr.Function object at 0x308407e0>, '_test': <pyclbr.Function object at 0x30840810>}
...
1:08:14 [398/420/8] test_inspect failed -- running: test_tools (10 min 44 sec), test_multiprocessing_spawn (11 min 35 sec), test_io (5 min 23 sec)
test test_inspect crashed -- Traceback (most recent call last):
  File "/home/buildbot/cpython-master/Lib/test/libregrtest/runtest.py", line 166, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/home/buildbot/cpython-master/Lib/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 777, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/buildbot/cpython-master/Lib/test/test_inspect.py", line 11, in <module>
    import _pickle
ModuleNotFoundError: No module named '_pickle'
...
388 tests OK.

9 tests failed:
    test_embed test_ftplib test_inspect test_json
    test_multiprocessing_fork test_multiprocessing_forkserver
    test_multiprocessing_spawn test_pyclbr test_pydoc

Maybe, XLC doesn't support -D name.  -Dname should be used instead.

On 22/04/2019 14:15, Inada Naoki wrote:
> Inada Naoki <songofacandy@gmail.com> added the comment:
>
> Maybe, XLC doesn't support -D name.  -Dname should be used instead.

Excellent hint: the diff between bot run 1013 and run 1014 reveals:

diff --git a/setup.py b/setup.py
index 9c83914fd9..c470719a6a 100644
--- a/setup.py
+++ b/setup.py
@@ -725,13 +725,13 @@ class PyBuildExt(build_ext):
         # heapq
         self.add(Extension("_heapq", ["_heapqmodule.c"]))
         # C-optimized pickle replacement
-        self.add(Extension("_pickle", ["_pickle.c"]))
+        self.add(Extension("_pickle", ["_pickle.c"],
+                           extra_compile_args=['-D Py_BUILD_CORE_MODULE']))
         # atexit
         self.add(Extension("atexit", ["atexitmodule.c"]))
         # _json speedups
         self.add(Extension("_json", ["_json.c"],
-                           # pycore_accu.h requires Py_BUILD_CORE_BUILTIN
-                           extra_compile_args=['-DPy_BUILD_CORE_BUILTIN']))
+                           extra_compile_args=['-D Py_BUILD_CORE_MODULE']))

         # profiler (_lsprof is for cProfile.py)
         self.add(Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']))

So, some simple editing: and the diff with master (of yesterday):

diff --git a/setup.py b/setup.py
index 3d6404f89e..58c16e8ba4 100644
--- a/setup.py
+++ b/setup.py
@@ -726,12 +726,12 @@ class PyBuildExt(build_ext):
         self.add(Extension("_heapq", ["_heapqmodule.c"]))
         # C-optimized pickle replacement
         self.add(Extension("_pickle", ["_pickle.c"],
-                           extra_compile_args=['-D Py_BUILD_CORE_MODULE']))
+                           extra_compile_args=['-DPy_BUILD_CORE_MODULE']))
         # atexit
         self.add(Extension("atexit", ["atexitmodule.c"]))
         # _json speedups
         self.add(Extension("_json", ["_json.c"],
-                           extra_compile_args=['-D Py_BUILD_CORE_MODULE']))
+                           extra_compile_args=['-DPy_BUILD_CORE_MODULE']))

         # profiler (_lsprof is for cProfile.py)
         self.add(Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']))
@@ -816,7 +816,7 @@ class PyBuildExt(build_ext):

         # Python Internal C API test module
         self.add(Extension('_testinternalcapi', ['_testinternalcapi.c'],
-                           extra_compile_args=['-D Py_BUILD_CORE_MODULE']))
+                           extra_compile_args=['-DPy_BUILD_CORE_MODULE']))

         # Python PEP-3118 (buffer protocol) test module
         self.add(Extension('_testbuffer', ['_testbuffer.c']))

And now the modules build as expected.

Should I open PR - or is this better fixed in the original PR? What
keeps things cleaner?

>
> ----------
> nosy: +inada.naoki
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue36696>
> _______________________________________
>

> Should I open PR - or is this better fixed in the original PR? What keeps things cleaner?

I wrote PR 12922.

I really hate -Ddefine syntax, but it seems like we have to use it just for AIX :-( I tried to use "-D Py_BUILD_CORE_..." in Modules/Setup, but Module/makesetup is also stupid and doesn't support this syntax :-( I was too lazy to fix Module/makesetup.

New changeset 574913479f26b5ff48827861bce68281be01d16e by Victor Stinner in branch 'master':
bpo-36635, bpo-36696: Fix setup.py on AIX (GH-12922)
https://github.com/python/cpython/commit/574913479f26b5ff48827861bce68281be01d16e