This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Python > 3.7 build fails with IBM XL compiler
Type: compile error Stage:
Components: Installation Versions: Python 3.11, Python 3.10, Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eric.smith, jack__d, quellyn
Priority: normal Keywords:

Created on 2021-08-09 23:37 by quellyn, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
cpython.tar.gz quellyn, 2021-08-09 23:37 Tarball of output logs from configure and make
Messages (9)
msg399296 - (view) Author: Quellyn Snead (quellyn) * Date: 2021-08-09 23:37
Python 3.8 and above fails to build with the IBM XL compiler on Power9 platforms.

## System and Environment Information

```
$ arch
ppc64le
```

```
$ echo $CC
xlc -F/projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/etc/xlc.cfg.rhel.7.8.gcc.8.3.0.cuda.10.1
```

```
$ xlc --version
IBM XL C/C++ for Linux, V16.1.1 (5725-C73, 5765-J13)
Version: 16.01.0001.0007
```

## Test Procedure
```
$ git clone git@github.com:quellyn/cpython.git
$ cd cpython
$ checkout 3.8

$ ./configure --with-pydebug
$ make -j2 | tee -a make.out
```

I tested for 3.7, 3.8, 3.9, 3.10, and master. In all cases the `make` failed for versions 3.8+. I've attached both the `config.log` and the make output logs for all. 

- Power9 architecture (ppc64le)
- IBM XL C/C++ 16.1.1.7

Python 3.7 builds without issue.
msg399307 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2021-08-10 09:12
Rather than have us wading through all of the output, can you summarize the problem? After a brief look I don't see any actual compiler errors.

In make.out_3.9, at least, it looks like "generate-posix-vars failed", but I can't see why. What does

./python -E -S -m sysconfig --generate-posix-vars

produce?

Also, I'm not sure what the ".9" and ".10" files are supposed to be. Maybe accidentally included?
msg399335 - (view) Author: Quellyn Snead (quellyn) * Date: 2021-08-10 15:57
Hi Eric,

Looks like I didn't get the full output captured before. Sorry about that. The error appears when I run the make command:

```
[quellyn@cn2020 cpython]$ make -j
./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
	echo "generate-posix-vars failed" ; \
	rm -f ./pybuilddir.txt ; \
	exit 1 ; \
fi
Fatal Python error: take_gil: NULL tstate
Python runtime state: preinitialized

Current thread 0x0000200000045fe0 (most recent call first):
/bin/sh: line 5: 79067 Segmentation fault      (core dumped) ./python -E -S -m sysconfig --generate-posix-vars
generate-posix-vars failed
make: *** [pybuilddir.txt] Error 1
```

P.S. The tarball I attached before contains the configure and make output for tests on multiple python version branches. I differentiated them by adding a version identifier suffix (_3.7, _3.8, _3.9, _3.10, _3.11)
msg399349 - (view) Author: Jack DeVries (jack__d) * Date: 2021-08-10 18:48
There is a related failure message in the file name ".9" in the tarball (line 175):

./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
	echo "generate-posix-vars failed" ; \
	rm -f ./pybuilddir.txt ; \
	exit 1 ; \
fi
Fatal Python error: _PyMem_DebugMalloc: Python memory allocator called without holding the GIL
Python runtime state: preinitialized

Current thread 0x0000200000045fe0 (most recent call first):
<no Python frame>
/bin/sh: line 5: 122035 Aborted                 (core dumped) ./python -E -S -m sysconfig --generate-posix-vars
generate-posix-vars failed
msg399350 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2021-08-10 18:50
> P.S. The tarball I attached before contains the configure and make output for tests on multiple python version branches. I differentiated them by adding a version identifier suffix (_3.7, _3.8, _3.9, _3.10, _3.11)

I was referring to the two files named exactly ".9" and ".10", which are hidden by default by the shell, since they begin with dots.
msg399351 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2021-08-10 18:52
As to the actual problem, I think you're going to need to get out a debugger and at least get a stack trace.
msg399352 - (view) Author: Jack DeVries (jack__d) * Date: 2021-08-10 19:16
I'm sure you are aware of this, but also note that the issue could be in
pandas or ibm-db, which include C extensions. I'm pretty sure those are
the only two dependencies you listed there that have C dependencies.
msg399353 - (view) Author: Jack DeVries (jack__d) * Date: 2021-08-10 19:17
Woah, oops, nevermind! I was confusing this with a different bpo in my
head. Sorry for the noise!
msg399355 - (view) Author: Quellyn Snead (quellyn) * Date: 2021-08-10 20:42
> As to the actual problem, I think you're going to need to get out a debugger and at least get a stack trace.

Here's my attempt:

$ gdb ./python
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "ppc64le-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /vast/home/quellyn/GIT/cpython/python...done.
(gdb) set args -E -S -m sysconfig --generate-posix-vars
(gdb) run
Starting program: /vast/home/quellyn/GIT/cpython/./python -E -S -m sysconfig --generate-posix-vars
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: take_gil: NULL tstate
Python runtime state: preinitialized

Current thread 0x0000200000045fe0 (most recent call first):

Program received signal SIGSEGV, Segmentation fault.
0x00000000105a3870 in ?? ()
Missing separate debuginfos, use: debuginfo-install nss-softokn-freebl-3.53.1-6.el7_9.ppc64le
(gdb) bt
#0  0x00000000105a3870 in ?? ()
#1  0x000000001021c3a0 in dump_traceback (fd=2, tstate=0x105a3870, write_header=0)
    at Python/traceback.c:805
#2  0x000000001021c620 in _Py_DumpTracebackThreads (fd=2, interp=0x105a2b90, 
    current_tstate=0x105a3870) at Python/traceback.c:915
#3  0x00000000101ee2ac in _Py_FatalError_DumpTracebacks (fd=2, interp=0x0, tstate=0x105a3870)
    at Python/pylifecycle.c:1981
#4  0x00000000101ee800 in fatal_error (prefix=0x0, msg=0x1041c7b4 "take_gil: NULL tstate", 
    status=-1) at Python/pylifecycle.c:2159
#5  0x00000000101ee8b0 in Py_FatalError (msg=0x1041c7b4 "take_gil: NULL tstate")
    at Python/pylifecycle.c:2193
#6  0x000000001017ff04 in take_gil (ceval=0x1055c178 <_PyRuntime+584>, tstate=0x0)
    at Python/ceval_gil.h:187
#7  0x0000000010191db4 in PyEval_InitThreads () at Python/ceval.c:213
#8  0x00000000101f3490 in pycore_create_interpreter (runtime=0x1055bf30 <_PyRuntime>, 
    config=0x105a2c40, interp_p=0x7fffffffd3c0) at Python/pylifecycle.c:550
#9  0x00000000101f35e0 in pyinit_config (runtime=0x1055bf30 <_PyRuntime>, interp_p=0x7fffffffd918, 
    config=0x7fffffffd5f0) at Python/pylifecycle.c:674
#10 0x00000000101f3b38 in pyinit_core (runtime=0x1055bf30 <_PyRuntime>, src_config=0x7fffffffdac0, 
    interp_p=0x7fffffffd918) at Python/pylifecycle.c:858
#11 0x00000000101f3dc4 in Py_InitializeFromConfig (config=0x7fffffffdac0)
    at Python/pylifecycle.c:1031
#12 0x000000001002409c in pymain_init (args=0x7fffffffde50) at Modules/main.c:78
---Type <return> to continue, or q <return> to quit---
#13 0x000000001002428c in pymain_main (args=0x7fffffffde50) at Modules/main.c:710
#14 0x0000000010024394 in Py_BytesMain (argc=6, argv=0x7fffffffe2d8) at Modules/main.c:743
#15 0x0000000010021e18 in main (argc=6, argv=0x7fffffffe2d8) at ./Programs/python.c:16
(gdb)
History
Date User Action Args
2022-04-11 14:59:48adminsetgithub: 89040
2021-08-10 20:42:01quellynsetmessages: + msg399355
2021-08-10 19:17:34jack__dsetmessages: + msg399353
2021-08-10 19:16:09jack__dsetmessages: + msg399352
2021-08-10 18:52:58eric.smithsetmessages: + msg399351
2021-08-10 18:50:28eric.smithsetmessages: + msg399350
2021-08-10 18:48:16jack__dsetnosy: + jack__d
messages: + msg399349
2021-08-10 15:57:13quellynsetmessages: + msg399335
2021-08-10 09:12:41eric.smithsetnosy: + eric.smith
messages: + msg399307
2021-08-09 23:37:13quellyncreate