possible multiple regressions on AIX #80877

aixtools · 2019-04-22T06:58:38Z

BPO	36696
Nosy	@vstinner, @methane, @aixtools
PRs	bpo-36635, bpo-36696: Fix setup.py on AIX #12922

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-04-23.10:29:59.791>
created_at = <Date 2019-04-22.06:58:37.681>
labels = ['build', '3.8']
title = 'possible multiple regressions on AIX'
updated_at = <Date 2019-04-23.10:29:59.778>
user = 'https://github.com/aixtools'

bugs.python.org fields:

activity = <Date 2019-04-23.10:29:59.778>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-04-23.10:29:59.791>
closer = 'vstinner'
components = ['Build']
creation = <Date 2019-04-22.06:58:37.681>
creator = 'Michael.Felt'
dependencies = []
files = []
hgrepos = []
issue_num = 36696
keywords = ['patch']
message_count = 5.0
messages = ['340643', '340650', '340704', '340705', '340711']
nosy_count = 3.0
nosy_names = ['vstinner', 'methane', 'Michael.Felt']
pr_nums = ['12922']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36696'
versions = ['Python 3.8']

aixtools · 2019-04-22T06:58:37Z

My AIX bot has been very consistent - only the multiprocessing tests failing when run by bot, but 4 or 5 days ago 3 to 5 additional tests - that, afaik, had never failed before, are now failing.

These may also be compiler related specifics, or the presence (better lack of) 3rd party packages.

Anyway - summary of output:

INFO: Can't locate Tcl/Tk libs and/or headers
building '_pickle' extension
xlc_r -O2 -I/opt/include -I/opt/buildaix/include -g -I/opt/include -I/opt/buildaix/include -g -I./Include/internal -I./Include -I. -I/opt/include -I/opt/buildaix/include -I/home/buildbot/cpython-master/Include -I/home/buildbot/cpython-master -c /home/buildbot/cpython-master/Modules/_pickle.c -o build/temp.aix-7.1-3.8-pydebug/home/buildbot/cpython-master/Modules/_pickle.o -D Py_BUILD_CORE_MODULE
1506-261 (W) Suboption Py_BUILD_CORE_MODULE is not valid for option D.
"/home/buildbot/cpython-master/Modules/_pickle.c", line 8.4: 1506-205 (S) #error "Py_BUILD_CORE_BUILTIN or Py_BUILD_CORE_MODULE must be defined"
building '_json' extension
xlc_r -O2 -I/opt/include -I/opt/buildaix/include -g -I/opt/include -I/opt/buildaix/include -g -I./Include/internal -I./Include -I. -I/opt/include -I/opt/buildaix/include -I/home/buildbot/cpython-master/Include -I/home/buildbot/cpython-master -c /home/buildbot/cpython-master/Modules/_json.c -o build/temp.aix-7.1-3.8-pydebug/home/buildbot/cpython-master/Modules/_json.o -D Py_BUILD_CORE_MODULE
1506-261 (W) Suboption Py_BUILD_CORE_MODULE is not valid for option D.
"/home/buildbot/cpython-master/Modules/_json.c", line 8.4: 1506-205 (S) #error "Py_BUILD_CORE_BUILTIN or Py_BUILD_CORE_MODULE must be defined"
"./Include/internal/pycore_accu.h", line 13.4: 1506-205 (S) #error "this header requires Py_BUILD_CORE define"
building '_testinternalcapi' extension
xlc_r -O2 -I/opt/include -I/opt/buildaix/include -g -I/opt/include -I/opt/buildaix/include -g -I./Include/internal -I./Include -I. -I/opt/include -I/opt/buildaix/include -I/home/buildbot/cpython-master/Include -I/home/buildbot/cpython-master -c /home/buildbot/cpython-master/Modules/_testinternalcapi.c -o build/temp.aix-7.1-3.8-pydebug/home/buildbot/cpython-master/Modules/_testinternalcapi.o -D Py_BUILD_CORE_MODULE
1506-261 (W) Suboption Py_BUILD_CORE_MODULE is not valid for option D.
"/home/buildbot/cpython-master/Modules/_testinternalcapi.c", line 6.4: 1506-205 (S) #error "Py_BUILD_CORE_BUILTIN or Py_BUILD_CORE_MODULE must be defined"
"./Include/internal/pycore_coreconfig.h", line 8.4: 1506-205 (S) #error "this header requires Py_BUILD_CORE define"

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_curses_panel _gdbm _tkinter
ossaudiodev readline spwd
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time

Failed to build these modules:
_json _pickle _testinternalcapi

running build_scripts
copying and adjusting /home/buildbot/cpython-master/Tools/scripts/pydoc3 -> build/scripts-3.8
copying and adjusting /home/buildbot/cpython-master/Tools/scripts/idle3 -> build/scripts-3.8
copying and adjusting /home/buildbot/cpython-master/Tools/scripts/2to3 -> build/scripts-3.8
changing mode of build/scripts-3.8/pydoc3 from 644 to 755
changing mode of build/scripts-3.8/idle3 from 644 to 755
changing mode of build/scripts-3.8/2to3 from 644 to 755
renaming build/scripts-3.8/pydoc3 to build/scripts-3.8/pydoc3.8
renaming build/scripts-3.8/idle3 to build/scripts-3.8/idle3.8
renaming build/scripts-3.8/2to3 to build/scripts-3.8/2to3-3.8
         ./python -E -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(), *sys.version_info[:2]))' >platform
         ./python  ./Tools/scripts/run_tests.py
/home/buildbot/cpython-master/python -u -W default -bb -E -m test -r -w -j 0 -u all,-largefile,-audio,-gui
== CPython 3.8.0a3+ (heads/master:3e986de0d6, Apr 21 2019, 17:04:13) [C]
...
running: test_venv (3 min 38 sec), test_decimal (2 min 27 sec), test_ftplib (30 sec 50 ms), test_zipfile (1 min 43 sec)
0:04:00 [ 40/420/1] test_ftplib failed -- running: test_venv (3 min 53 sec), test_decimal (2 min 42 sec), test_zipfile (1 min 58 sec)
test test_ftplib failed -- Traceback (most recent call last):
  File "/home/buildbot/cpython-master/Lib/test/test_ftplib.py", line 605, in test_storlines
    self.client.storlines('stor', f)
  File "/home/buildbot/cpython-master/Lib/ftplib.py", line 526, in storlines
    conn.unwrap()
  File "/home/buildbot/cpython-master/Lib/ssl.py", line 1094, in unwrap
    s = self._sslobj.shutdown()
socket.timeout: The read operation timed out
0:04:05 [ 41/420/1] test_locale passed -- running: test_venv (3 min 58 sec), test_decimal (2 min 47 sec), test_zipfile (2 min 3 sec)
...
0:12:14 [110/420/2] test_embed failed -- running: test_multiprocessing_forkserver (1 min 25 sec), test_compileall (2 min 9 sec)
test test_embed failed -- multiple errors occurred; run in verbose mode for details
running: test_multiprocessing_forkserver (1 min 55 sec), test_faulthandler (30 sec 42 ms), test_compile (56 sec 571 ms), test_compileall (2 min 39 sec)
0:12:55 [111/420/2] test_faulthandler passed (36 sec 350 ms) -- running: test_multiprocessing_forkserver (2 min 6 sec), test_compile (1 min 7 sec), test_compileall (2 min             50 sec)
0:13:46 [125/420/3] test_json failed -- running: test_multiprocessing_forkserver (2 min 57 sec)
test test_json crashed -- Traceback (most recent call last):
  File "/home/buildbot/cpython-master/Lib/test/libregrtest/runtest.py", line 166, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/home/buildbot/cpython-master/Lib/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 777, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/buildbot/cpython-master/Lib/test/test_json/__init__.py", line 12, in <module>
    cjson.JSONDecodeError = cjson.decoder.JSONDecodeError = json.JSONDecodeError
AttributeError: 'NoneType' object has no attribute 'JSONDecodeError'
...
0:24:49 [212/420/5] test_pydoc failed -- running: test_pyclbr (1 min 2 sec), test_subprocess (11 min 1 sec)
test test_pydoc crashed -- Traceback (most recent call last):
  File "/home/buildbot/cpython-master/Lib/test/libregrtest/runtest.py", line 166, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/home/buildbot/cpython-master/Lib/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 777, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/buildbot/cpython-master/Lib/test/test_pydoc.py", line 9, in <module>
    import _pickle
ModuleNotFoundError: No module named '_pickle'
...
0:24:59 [214/420/6] test_pyclbr failed -- running: test_subprocess (11 min 12 sec)
*** Pickler
test test_pyclbr failed -- Traceback (most recent call last):
  File "/home/buildbot/cpython-master/Lib/test/test_pyclbr.py", line 227, in test_others
    cm('pickle', ignore=('partial',))
  File "/home/buildbot/cpython-master/Lib/test/test_pyclbr.py", line 144, in checkModule
    self.assertHaskey(dict, name, ignore)
  File "/home/buildbot/cpython-master/Lib/test/test_pyclbr.py", line 48, in assertHaskey
    self.assertIn(key, obj)
AssertionError: 'Pickler' not found in {'partial': <pyclbr.Class object at 0x306a1990>, 'PickleError': <pyclbr.Class object at 0x30834f30>, 'PicklingError': <pyclbr.Class             object at 0x30834e40>, 'UnpicklingError': <pyclbr.Class object at 0x30834f00>, '_Stop': <pyclbr.Class object at 0x30834f60>, '_Framer': <pyclbr.Class object at 0x30834f90>            , '_Unframer': <pyclbr.Class object at 0x30839150>, '_getattribute': <pyclbr.Function object at 0x308392a0>, 'whichmodule': <pyclbr.Function object at 0x30839300>, 'encode            _long': <pyclbr.Function object at 0x30839330>, 'decode_long': <pyclbr.Function object at 0x30839360>, '_Pickler': <pyclbr.Class object at 0x308392d0>, '_Unpickler': <pycl            br.Class object at 0x308398d0>, '_dump': <pyclbr.Function object at 0x30840750>, '_dumps': <pyclbr.Function object at 0x30840780>, '_load': <pyclbr.Function object at 0x30            8407b0>, '_loads': <pyclbr.Function object at 0x308407e0>, '_test': <pyclbr.Function object at 0x30840810>}
...
1:08:14 [398/420/8] test_inspect failed -- running: test_tools (10 min 44 sec), test_multiprocessing_spawn (11 min 35 sec), test_io (5 min 23 sec)
test test_inspect crashed -- Traceback (most recent call last):
  File "/home/buildbot/cpython-master/Lib/test/libregrtest/runtest.py", line 166, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/home/buildbot/cpython-master/Lib/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 777, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/buildbot/cpython-master/Lib/test/test_inspect.py", line 11, in <module>
    import _pickle
ModuleNotFoundError: No module named '_pickle'
...
388 tests OK.

9 tests failed:
test_embed test_ftplib test_inspect test_json
test_multiprocessing_fork test_multiprocessing_forkserver
test_multiprocessing_spawn test_pyclbr test_pydoc

methane · 2019-04-22T12:15:22Z

Maybe, XLC doesn't support -D name. -Dname should be used instead.

aixtools · 2019-04-23T08:47:01Z

On 22/04/2019 14:15, Inada Naoki wrote:

Inada Naoki <songofacandy@gmail.com> added the comment:

Maybe, XLC doesn't support -D name. -Dname should be used instead.

Excellent hint: the diff between bot run 1013 and run 1014 reveals:

diff --git a/setup.py b/setup.py
index 9c83914fd9..c470719a6a 100644
--- a/setup.py
+++ b/setup.py
@@ -725,13 +725,13 @@ class PyBuildExt(build_ext):
         # heapq
         self.add(Extension("_heapq", ["_heapqmodule.c"]))
         # C-optimized pickle replacement
-        self.add(Extension("_pickle", ["_pickle.c"]))
+        self.add(Extension("_pickle", ["_pickle.c"],
+                           extra_compile_args=['-D Py_BUILD_CORE_MODULE']))
         # atexit
         self.add(Extension("atexit", ["atexitmodule.c"]))
         # _json speedups
         self.add(Extension("_json", ["_json.c"],
-                           # pycore_accu.h requires Py_BUILD_CORE_BUILTIN
-                           extra_compile_args=['-DPy_BUILD_CORE_BUILTIN']))
+                           extra_compile_args=['-D Py_BUILD_CORE_MODULE']))

# profiler (_lsprof is for cProfile.py)
self.add(Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']))

So, some simple editing: and the diff with master (of yesterday):

diff --git a/setup.py b/setup.py
index 3d6404f89e..58c16e8ba4 100644
--- a/setup.py
+++ b/setup.py
@@ -726,12 +726,12 @@ class PyBuildExt(build_ext):
         self.add(Extension("_heapq", ["_heapqmodule.c"]))
         # C-optimized pickle replacement
         self.add(Extension("_pickle", ["_pickle.c"],
-                           extra_compile_args=['-D Py_BUILD_CORE_MODULE']))
+                           extra_compile_args=['-DPy_BUILD_CORE_MODULE']))
         # atexit
         self.add(Extension("atexit", ["atexitmodule.c"]))
         # _json speedups
         self.add(Extension("_json", ["_json.c"],
-                           extra_compile_args=['-D Py_BUILD_CORE_MODULE']))
+                           extra_compile_args=['-DPy_BUILD_CORE_MODULE']))

# profiler (_lsprof is for cProfile.py)
self.add(Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']))
@@ -816,7 +816,7 @@ class PyBuildExt(build_ext):

         # Python Internal C API test module
         self.add(Extension('_testinternalcapi', ['_testinternalcapi.c'],
-                           extra_compile_args=['-D Py_BUILD_CORE_MODULE']))
+                           extra_compile_args=['-DPy_BUILD_CORE_MODULE']))

# Python PEP-3118 (buffer protocol) test module
self.add(Extension('_testbuffer', ['_testbuffer.c']))

And now the modules build as expected.

Should I open PR - or is this better fixed in the original PR? What
keeps things cleaner?

----------
nosy: +inada.naoki

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue36696\>

vstinner · 2019-04-23T08:52:33Z

Should I open PR - or is this better fixed in the original PR? What keeps things cleaner?

I wrote PR 12922.

I really hate -Ddefine syntax, but it seems like we have to use it just for AIX :-( I tried to use "-D Py_BUILD_CORE_..." in Modules/Setup, but Module/makesetup is also stupid and doesn't support this syntax :-( I was too lazy to fix Module/makesetup.

vstinner · 2019-04-23T10:26:39Z

New changeset 5749134 by Victor Stinner in branch 'master':
bpo-36635, bpo-36696: Fix setup.py on AIX (GH-12922)
5749134

vstinner added 3.8 only security fixes build The build process and cross-build labels Apr 23, 2019

vstinner closed this as completed Apr 23, 2019

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

possible multiple regressions on AIX #80877

possible multiple regressions on AIX #80877

aixtools commented Apr 22, 2019

aixtools commented Apr 22, 2019

methane commented Apr 22, 2019

aixtools commented Apr 23, 2019

vstinner commented Apr 23, 2019

vstinner commented Apr 23, 2019

possible multiple regressions on AIX #80877

possible multiple regressions on AIX #80877

Comments

aixtools commented Apr 22, 2019

aixtools commented Apr 22, 2019

methane commented Apr 22, 2019

aixtools commented Apr 23, 2019

vstinner commented Apr 23, 2019

vstinner commented Apr 23, 2019