Python fails to build _asyncio on module on AIX #84117
Comments
|
The commit 1ec63b6 of bpo-39763 broke the Python compilation on AIX: -- The last successful build was before the commit 1ec63b6: _socket compilation: pythoninfo: sys.path: [ Both steps use "build/lib.aix-7100-9898-32-3.9-pydebug/" directory. -- Recent failure: _socket compilation: (...) -o build/lib.aix-7104-1806-32-3.9-pydebug/_socket.so pythoninfo: sys.path: [ So the compilation uses "build/lib.aix-7104-1806-32-3.9-pydebug/" directory, whereas pythoninfo uses "build/lib.aix-7100-9898-32-3.9-pydebug/" directory. It can explain why setup.py fails to import _socket later: it was built in a different directory. -- I see that _aix_support._aix_bosmp64() has two code paths depending on the subprocess module can be imported: # subprocess is not necessarily available early in the build process def _aix_bosmp64():
# type: () -> Tuple[str, int]
"""
Return a Tuple[str, int] e.g., ['7.1.4.34', 1806]
The fileset bos.mp64 is the AIX kernel. It's VRMF and builddate
reflect the current ABI levels of the runtime environment.
"""
if _have_subprocess:
# We expect all AIX systems to have lslpp installed in this location
out = subprocess.check_output(["/usr/bin/lslpp", "-Lqc", "bos.mp64"])
out = out.decode("utf-8").strip().split(":") # type: ignore
# Use str() and int() to help mypy see types
return str(out[2]), int(out[-1])
else:
from os import uname-- _aix_support._aix_bosmp64() is called by _aix_support.aix_platform() which is called by get_host_platform() of distutils.util. Currently, setup.py does:
|
|
AIX_BUILDDATE variable is written in pyconfig.h by configure: configure: "checking for the system builddate... 1806" configure: "checking for the system builddate... 1806" So AIX_BUILDDATE seems to be set in both case. But on build 451, the "build/lib.aix-7100-9898-32-3.9-pydebug" directory name contains 9898: this number comes from _aix_support._MISSING_BD. It's used when _have_subprocess is false or when get_config_var("AIX_BUILDDATE") cannot be converted into an integer. |
|
While looking through the history of bot builds - I consistently see 420 tests being done on one bot - but test_socket does not always show up in the list. I'll look at this as I can, but "free-time" is limited. Delay is not a lack of interest. |
|
Michael.Felt: Can you please try to build the master branch of Python with the attached setup-aix.patch applied? Something like: git checkout master Then attach make.log. |
|
Another approach to patch _aix_support.py, replace: try: with: try: Use _bootsubprocess when subprocess is not available. If this approach works, _have_subprocess=False code path can be removed. |
|
I tried your patch on AIX 7.2, looks like you need to shift sys import to the top Traceback (most recent call last):
File "/home/isidentical/cpython/./setup.py", line 16, in <module>
sys.modules['subprocess'] = _bootsubprocess
NameError: name 'sys' is not defined
make: The error code from the last command is 1.Stop. |
|
Yeah, sys must be imported first. Did you try to fix the patch? Does it work? |
|
I'll take a look at what you are suggesting. The starting point (before the rm command) is the make command that I run again. What I notice - read am thinking - is that _socket.so is being created by the "setup.py build" - so, if you can help me make that bit more verbose, I'll work on that too. +++ The following modules found by detect_modules() in setup.py, have been Following modules built successfully but were removed because they could not be imported: |
|
So, with the patch - the process stops at: aixtools@x064:[/home/aixtools/python-3.9]make
CC='xlc_r' LDSHARED='Modules/ld_so_aix xlc_r -bI:Modules/python.exp ' OPT='-DNDEBUG -O' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build
Traceback (most recent call last):
File "/home/aixtools/python-3.9/Lib/subprocess.py", line 72, in <module>
import msvcrt
ModuleNotFoundError: No module named 'msvcrt'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/aixtools/python-3.9/./setup.py", line 4, in <module>
import subprocess
File "/home/aixtools/python-3.9/Lib/subprocess.py", line 77, in <module>
import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/aixtools/python-3.9/./setup.py", line 16, in <module>
sys.modules['subprocess'] = _bootsubprocess
NameError: name 'sys' is not defined
make: 1254-004 The error code from the last command is 1.Stop. |
|
So, this is what I have on screen. Will add the log in a moment. [1] + Done make 2>&1|tee make.log & aixtools@x064:[/home/aixtools/python-3.9]make V=1 Python build finished successfully! The following modules found by detect_modules() in setup.py, have been Following modules built successfully but were removed because they could not be imported: |
|
OK. I removed the _aix_support.py from the formula. After make see the new (rather) old build paths for "socket" aixtools@x064:[/home/aixtools/python-3.9]find . | grep socket I remove these two files again and run make: aixtools@x064:[/home/aixtools/python-3.9]make Python build finished successfully! The following modules found by detect_modules() in setup.py, have been Following modules built successfully but were removed because they could not be imported: The diff! diff --git a/Lib/_aix_support.py b/Lib/_aix_support.py
index 2c5cd3297d..c7f4491633 100644
--- a/Lib/_aix_support.py
+++ b/Lib/_aix_support.py
@@ -12,7 +12,8 @@ try:
_tmp_bd = get_config_var("AIX_BUILDDATE")
_bgt = get_config_var("BUILD_GNU_TYPE")
except ImportError: # pragma: no cover
- _have_subprocess = False
+ import _bootsubprocess as subprocess
+ _have_subprocess = True
_tmp_bd = None
_bgt = "powerpc-ibm-aix6.1.7.0"
diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py
index 4b002ecef1..513af52ecd 100644
--- a/Lib/distutils/util.py
+++ b/Lib/distutils/util.py
@@ -79,8 +79,9 @@ def get_host_platform():
machine += ".%s" % bitness[sys.maxsize]
# fall through to standard osname-release-machine representation
elif osname[:3] == "aix":
- from _aix_support import aix_platform
- return aix_platform()
+ return "%s-%s.%s" % (osname, version, release)
+ # from _aix_support import aix_platform
+ # return aix_platform()
elif osname[:6] == "cygwin":
osname = "cygwin"
rel_re = re.compile (r'[\d.]+', re.ASCII)
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
index 4003726dc9..b116e96007 100644
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -666,8 +666,9 @@ def get_platform():
machine += ".%s" % bitness[sys.maxsize]
# fall through to standard osname-release-machine representation
elif osname[:3] == "aix":
- from _aix_support import aix_platform
- return aix_platform()
+ return "%s-%s.%s" % (osname, version, release)
+ # from _aix_support import aix_platform
+ # return aix_platform()
elif osname[:6] == "cygwin":
osname = "cygwin"
import reIMHO - this problem has nothing to do with _aix_support.py - That issue was fixed earlier by the new _bootsubprocess. |
|
Michael: this issue is about bootstraping Python. If you want to test a patch or test something else, you must restart from a clean copy of the source code. Either use "make distclean", "git clean -fdx", or recreate the source directory (git clone, decompress an archive, etc.). Then restart from scratch: ./configure && make.
Right, that's a stupid bug in my patch. You must import sys before. |
|
Try to replace Lib/_aix_support.py with attached _aix_support.py: in short, it uses _bootsubprocess if subprocess is not available. |
|
Actually, I had already done that: diff --git a/Lib/_aix_support.py b/Lib/_aix_support.py
index 2c5cd3297d..c7f4491633 100644
--- a/Lib/_aix_support.py
+++ b/Lib/_aix_support.py
@@ -12,7 +12,8 @@ try:
_tmp_bd = get_config_var("AIX_BUILDDATE")
_bgt = get_config_var("BUILD_GNU_TYPE")
except ImportError: # pragma: no cover
- _have_subprocess = False
+ import _bootsubprocess as subprocess
+ _have_subprocess = True
_tmp_bd = None
_bgt = "powerpc-ibm-aix6.1.7.0"And after that test, I tried with no calls to _aix_support.py diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py
index 4b002ecef1..513af52ecd 100644
--- a/Lib/distutils/util.py
+++ b/Lib/distutils/util.py
@@ -79,8 +79,9 @@ def get_host_platform():
machine += ".%s" % bitness[sys.maxsize]
# fall through to standard osname-release-machine representation
elif osname[:3] == "aix":
- from _aix_support import aix_platform
- return aix_platform()
+ return "%s-%s.%s" % (osname, version, release)
+ # from _aix_support import aix_platform
+ # return aix_platform()
elif osname[:6] == "cygwin":
osname = "cygwin"
rel_re = re.compile (r'[\d.]+', re.ASCII)
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
index 4003726dc9..b116e96007 100644
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -666,8 +666,9 @@ def get_platform():
machine += ".%s" % bitness[sys.maxsize]
# fall through to standard osname-release-machine representation
elif osname[:3] == "aix":
- from _aix_support import aix_platform
- return aix_platform()
+ return "%s-%s.%s" % (osname, version, release)
+ # from _aix_support import aix_platform
+ # return aix_platform()
elif osname[:6] == "cygwin":
osname = "cygwin"
import re±±±±±±±± So, even when _aix_support.py is not involved at all, _socket.so is being ignored - even though it was built. |
|
re: Michael: this issue is about bootstraping Python. If you want to test a patch or test something else, you must restart from a clean copy of the source code. Either use "make distclean", "git clean -fdx", or recreate the source directory (git clone, decompress an archive, etc.). Then restart from scratch: ./configure && make. Understood: between these two timestamps: Date: 2020-03-11 18:55 and Date: 2020-03-11 21:36 - I was doing just that on 4 different VM's: AIX 5.3, 6.1, 7.1 and 7.2 - just in case there may be anything specific re: AIX versions (the bots run on AIX 7.1 and 7.2). I also switched away from any NFS based builds because test behaves differently when the build area is an NFS area compared to jfs2. FYI: I have decided to focus on AIX 7.1 and AIX 7.2 - and am only using gcc (I was also testing runs with both xlc and gcc before I started posting). Further, I like your suggestion to try: subprocess except _bootsubprocess and shall make a PR to that end. Much more clear about what is going on. What I am going to do bewtween now and my next post is again - clean slate - update master to latest version - on two servers. On one server I'll apply your suggestion to _aix_platform.py; on the other I'll patch Lib/distutils/util.py and Lib/sysconfig.py to not even call _aix_platform.py. Ideally, there will be a difference that leads to an understanding of the root cause. |
|
The good news! Your patch, better rewrite, of _aix_platform.py is working! Many thanks! |
|
I converted attached _aix_support.py into PR 18970 (with minor changes). |
|
Fantastic! Many thanks! |
|
POWER6 AIX 3.x and PPC64 AIX 3.x buildbots are back to green. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: