Build issue while compiling cpp files in AIX #79379

Ayappan · 2018-11-09T15:20:45Z

BPO	35198
Nosy	@vstinner, @aixtools, @miss-islington, @kadler
PRs	bpo-35198 Fix C++ extension compilation on AIX #10437 [3.7] bpo-35198 Fix C++ extension compilation on AIX (GH-10437) #12162
Files	pip3_build_pandas_0001.text: complete pip3 build pandas output

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-03-04.15:25:03.009>
created_at = <Date 2018-11-09.15:20:45.102>
labels = ['3.8', 'build', '3.7']
title = 'Build issue while compiling cpp files in AIX'
updated_at = <Date 2019-03-04.15:25:03.009>
user = 'https://bugs.python.org/Ayappan'

bugs.python.org fields:

activity = <Date 2019-03-04.15:25:03.009>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-03-04.15:25:03.009>
closer = 'vstinner'
components = ['Build']
creation = <Date 2018-11-09.15:20:45.102>
creator = 'Ayappan'
dependencies = []
files = ['47919']
hgrepos = []
issue_num = 35198
keywords = ['patch']
message_count = 15.0
messages = ['329522', '329622', '329625', '329684', '329722', '332961', '332971', '332984', '332989', '332991', '333231', '333342', '337123', '337128', '337130']
nosy_count = 6.0
nosy_names = ['vstinner', 'David.Edelsohn', 'Michael.Felt', 'Ayappan', 'miss-islington', 'kadler']
pr_nums = ['10437', '12162']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue35198'
versions = ['Python 3.7', 'Python 3.8']

Ayappan · 2018-11-09T15:20:45Z

I am trying to build pandas-0.23.4 using python3 in AIX.
Everything goes fine till it encounters a cpp file. Below is the compile error output.
building 'pandas._libs.window' extension
gcc -fPIC -maix64 -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DAIX_GENUINE_CPLUSCPLUS -D_LINUX_SOURCE_COMPAT -Wl,-brtl -O2 -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DAIX_GENUINE_CPLUSCPLUS -D_LINUX_SOURCE_COMPAT -Wl,-brtl -O2 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/freeware/lib64/python3.6/site-packages/numpy/core/include -I/opt/freeware/include/python3.6m -c pandas/_libs/window.cpp -o build/temp.aix-7.2-3.6/pandas/_libs/window.o -Wno-unused-function
In file included from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1821,
from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from pandas/_libs/window.cpp:587:
/opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^~~~~~~
g++ gcc -maix64 -pthread -bI:/opt/freeware/lib/python3.6/config-3.6m/python.exp -L. -L/usr/include -L/opt/freeware/include -L/usr/lib/threads -L/opt/freeware/lib64 -L/opt/freeware/lib -L/usr/lib64 -L/usr/lib -L. -L/usr/include -L/opt/freeware/include -L/usr/lib/threads -L/opt/freeware/lib64 -L/opt/freeware/lib -L/usr/lib64 -L/usr/lib build/temp.aix-7.2-3.6/pandas/_libs/window.o -L/opt/freeware/lib64 -o build/lib.aix-7.2-3.6/pandas/_libs/window.so
g++: error: gcc: No such file or directory
g++: error: unrecognized command line option '-bI:/opt/freeware/lib/python3.6/config-3.6m/python.exp'
error: command 'g++' failed with exit status 1

Seems like somewhere setting up the proper compile options for compiling cpp files is not proper. Any hint will be really useful.

aixtools · 2018-11-10T15:41:08Z

There are, perhaps, other issues as well.

After a build of "master"

a) pip3 install pandas

failed to find/download and build numpy
after "manual" pip3 install numpy and got to " six, python-dateutil, pytz, pandas"

Then:
Running setup.py install for pandas ... error
Complete output from command /opt/bin/python3.8 -u -c "import setuptools, tokenize;file='/tmp/pip-install-f5rvzoc0/pandas/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-kmi2p9sd/install-record.txt --single-version-externally-managed --compile:
...
UPDATING build/lib.aix-6.1-3.8-pydebug/pandas/_version.py
set build/lib.aix-6.1-3.8-pydebug/pandas/_version.py to '0.23.4'
running build_ext
building 'pandas._libs.algos' extension
creating build/temp.aix-6.1-3.8-pydebug
creating build/temp.aix-6.1-3.8-pydebug/pandas
creating build/temp.aix-6.1-3.8-pydebug/pandas/_libs
xlc_r -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/algos.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/algos.o -Wno-unused-function
xlc_r: 1501-210 (S) command option Wno-unused-function contains an incorrect subargument
error: command 'xlc_r' failed with exit status 40

\----------------------------------------

Command "/opt/bin/python3.8 -u -c "import setuptools, tokenize;file='/tmp/pip-install-f5rvzoc0/pandas/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-kmi2p9sd/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-f5rvzoc0/pandas/

So, it looks like, using xlc - I may not have even gotten as far window.gcc due to issues in pandas (the incorrect flag -Wno-used-function - have to find where that came from!)

aixtools · 2018-11-10T16:17:23Z

Getting farther - after "hacking" pandas setup.py to not force a gcc flag.

Still not getting as far, I think, as the initial poster.

...
/opt/include/python3.8dm -c pandas/_libs/parsers.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/parsers.o
xlc_r -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/src/parser/tokenizer.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/src/parser/tokenizer.o
"/opt/include/python3.8dm/pyconfig.h", line 1538.9: 1506-236 (W) Macro name _POSIX_C_SOURCE has been redefined.
"/opt/include/python3.8dm/pyconfig.h", line 1538.9: 1506-358 (I) "_POSIX_C_SOURCE" is defined on line 160 of /usr/include/standards.h.
"/opt/include/python3.8dm/pyconfig.h", line 1553.9: 1506-236 (W) Macro name _XOPEN_SOURCE has been redefined.
"/opt/include/python3.8dm/pyconfig.h", line 1553.9: 1506-358 (I) "_XOPEN_SOURCE" is defined on line 156 of /usr/include/standards.h.
"/usr/include/unistd.h", line 171.17: 1506-343 (S) Redeclaration of lseek64 differs from previous declaration on line 169 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 171.17: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the previous return type "long".
"/usr/include/unistd.h", line 171.17: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
"/usr/include/sys/lockf.h", line 64.20: 1506-343 (S) Redeclaration of lockf64 differs from previous declaration on line 62 of "/usr/include/sys/lockf.h".
"/usr/include/sys/lockf.h", line 64.20: 1506-377 (I) The type "long long" of parameter 3 differs from the previous type "long".
"/usr/include/unistd.h", line 809.33: 1506-343 (S) Redeclaration of ftruncate64 differs from previous declaration on line 807 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 809.33: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
"/usr/include/unistd.h", line 845.33: 1506-343 (S) Redeclaration of truncate64 differs from previous declaration on line 843 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 845.33: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
"/usr/include/unistd.h", line 862.33: 1506-343 (S) Redeclaration of pread64 differs from previous declaration on line 859 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 862.33: 1506-377 (I) The type "long long" of parameter 4 differs from the previous type "long".
"/usr/include/unistd.h", line 863.33: 1506-343 (S) Redeclaration of pwrite64 differs from previous declaration on line 860 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 863.33: 1506-377 (I) The type "long long" of parameter 4 differs from the previous type "long".
"/usr/include/unistd.h", line 942.25: 1506-343 (S) Redeclaration of fclear64 differs from previous declaration on line 939 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 942.25: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the previous return type "long".
"/usr/include/unistd.h", line 942.25: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
"/usr/include/unistd.h", line 943.25: 1506-343 (S) Redeclaration of fsync_range64 differs from previous declaration on line 940 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 943.25: 1506-377 (I) The type "long long" of parameter 3 differs from the previous type "long".
"pandas/_libs/src/parser/tokenizer.c", line 265.40: 1506-280 (W) Function argument assignment between types "long long*" and "unsigned long*" is not allowed.
"pandas/_libs/src/parser/tokenizer.c", line 292.30: 1506-280 (W) Function argument assignment between types "long long*" and "unsigned long*" is not allowed.
"pandas/_libs/src/parser/tokenizer.c", line 323.28: 1506-280 (W) Function argument assignment between types "long long*" and "unsigned long*" is not allowed.
error: command 'xlc_r' failed with exit status 1
root@x066:[/data/prj/python/pandas-0.23.4]set | grep OBJ
root@x066:[/data/prj/python/pandas-0.23.4]python3
Python 3.8.0a0 (heads/master-dirty:b9498e2367, Nov 10 2018, 13:51:41) [C] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.maxsize()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable
>>> sys.maxsize
2147483647

+++++
Will uninstall and build a 64-bit python3 and see if that helps.
+++++

aixtools · 2018-11-11T20:10:55Z

Still getting the same errors, even with 64-bit build, so still not close to testing the actual problem.
FYI: last time I build pandas the version was 0.19.0 - then it was 'simple'.

Anyway, short of a simple (test) python module that includes c++ code, I'll look at changing the pandas build files to force any c++ file to the front.

More - (asap, but still when I have time)

Ayappan · 2018-11-12T07:17:05Z

The PR 10437 worked !!!.
Now i am able to compile pandas-0.23.4

Michael Felt,

About the re-declaration issues you are facing, that is because of the "#define _LARGE_FILES 1" in pyconfig-ppc64.h file.
For 64bit build we actually don't need it.
For 32bit build, we need it and also we have to pass the flag "-D_LARGE_FILES" through the _sysconfigdata_m_aix6_.py , otherwise one will get this re-declaration error.

kadler · 2019-01-04T05:31:45Z

Just a friendly ping that there's a PR for this bug waiting to be reviewed.

aixtools · 2019-01-04T11:54:42Z

While the PR probably solves this - there is a 'bug' in pandas (I expect) that prevents me from completing the test - as, I expect LONG before the .cpp source is to be compiled - there is a error because a wrong flag is passed to the compiler (-Wno-unused-function) and the build stops.

Have not had the time to disect pandas so that I can get a .cpp source compiled.

+++++
Successfully installed pip-18.1
root@x066:[/data/prj/python/git/kadler]pip list
/opt/lib/python3.8/site-packages/pip/_vendor/html5lib/_trie/_base.py:3: DeprecationWdeprecated, and in 3.8 it will stop working
from collections import Mapping
Package Version
---------- -------
pip 18.1
setuptools 39.0.1
root@x066:[/data/prj/python/git/kadler]pip install numpy
/opt/lib/python3.8/site-packages/pip/_vendor/html5lib/_trie/_base.py:3: DeprecationWdeprecated, and in 3.8 it will stop working
from collections import Mapping
Collecting numpy
/opt/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py:220: PendingDepreca
warnings.warn(
Using cached https://files.pythonhosted.org/packages/2d/80/1809de155bad674b494248b
Installing collected packages: numpy
Running setup.py install for numpy ... done
Successfully installed numpy-1.15.4

...
Installing collected packages: six, python-dateutil, pytz, pandas
...
copying pandas/io/formats/templates/html.tpl -> build/lib.aix-6.1-3.8-pydebug/pandas/io/formats/templates
UPDATING build/lib.aix-6.1-3.8-pydebug/pandas/_version.py
set build/lib.aix-6.1-3.8-pydebug/pandas/_version.py to '0.23.4'
running build_ext
building 'pandas._libs.algos' extension
creating build/temp.aix-6.1-3.8-pydebug
creating build/temp.aix-6.1-3.8-pydebug/pandas
creating build/temp.aix-6.1-3.8-pydebug/pandas/_libs
xlc_r -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/algos.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/algos.o -Wno-unused-function
xlc_r: 1501-210 (S) command option Wno-unused-function contains an incorrect subargument
error: command 'xlc_r' failed with exit status 40

\----------------------------------------

Command "/opt/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-install-xnscgehv/pandas/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-0w46p413/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-xnscgehv/pandas/

kadler · 2019-01-04T16:08:23Z

Ah. We always compile with GCC, so would not have hit that particular problem.

aixtools · 2019-01-04T18:37:32Z

On 04/01/2019 17:08, Kevin wrote:

Kevin <kadler@us.ibm.com> added the comment:

Ah. We always compile with GCC, so would not have hit that particular problem.

----------

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue35198\>

FYI. I found a 'hack' in ./setup.py to skip adding the argument.
Progressing.

aixtools · 2019-01-04T19:04:33Z

Further along - however, I never get to the "link" routine.

Again, this is likely a pandas coding issue - currently python is calling xlc_r ..., but when I manually modify it to xlC_r I get the same error.

xlc_r -D_LARGE_FILES -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs -I./pandas/_libs -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/window.cpp -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/window.o
"pandas/_libs/window.cpp", line 9943.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 10030.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 11165.21: 1540-0063 (S) The text "(" is unexpected.
error: command 'xlc_r' failed with exit status 1
root@x066:[/data/prj/aixtools/git/pandas-master]ebug/pandas/_libs/window.o <
"pandas/_libs/window.cpp", line 9943.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 10030.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 11165.21: 1540-0063 (S) The text "(" is unexpected.

root@x066:[/data/prj/aixtools/git/pandas-master]xlC_r -D_LARGE_FILES -O -I/opt/include -O2 -qm>
"pandas/_libs/window.cpp", line 9943.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 10030.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 11165.21: 1540-0063 (S) The text "(" is unexpected.

FYI. Will look more tomorrow, but if you have an idea, like the -D_LARGE_FILES fix, I am much obliged.

Ayappan · 2019-01-08T14:40:04Z

Not sure what went wrong here.
I used gcc & g++ and didn't hit this issue.

aixtools · 2019-01-09T19:18:38Z

On 08/01/2019 15:40, Ayappan wrote:

Ayappan <ayappanec@gmail.com> added the comment:

Not sure what went wrong here.
I used gcc & g++ and didn't hit this issue.

----------

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue35198\>

Well, it is probably an issue with xlc as I have vac.C and
vacpp.cmp.core installed (aka xlc and xlC). These days it seems only gcc
environment gets tested in depth and portability issues with "posix"
compilers get missed.

GCC is a fine compiler - just not the one I am using for a number of
"logistical" reasons.

Thanks for the update!

Michael

vstinner · 2019-03-04T14:48:46Z

New changeset 800d5cd by Victor Stinner (Kevin Adler) in branch 'master':
bpo-35198 Fix C++ extension compilation on AIX (GH-10437)
800d5cd

miss-islington · 2019-03-04T15:06:40Z

New changeset 06e9953 by Miss Islington (bot) in branch '3.7':
bpo-35198 Fix C++ extension compilation on AIX (GH-10437)
06e9953

vstinner · 2019-03-04T15:25:03Z

Thanks Kevin Adler.

Ayappan mannequin added 3.7 (EOL) end of life build The build process and cross-build labels Nov 9, 2018

aixtools added the 3.8 only security fixes label Nov 10, 2018

vstinner closed this as completed Mar 4, 2019

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build issue while compiling cpp files in AIX #79379

Build issue while compiling cpp files in AIX #79379

Ayappan mannequin commented Nov 9, 2018

Ayappan mannequin commented Nov 9, 2018

aixtools commented Nov 10, 2018

aixtools commented Nov 10, 2018

aixtools commented Nov 11, 2018

Ayappan mannequin commented Nov 12, 2018

kadler mannequin commented Jan 4, 2019

aixtools commented Jan 4, 2019

kadler mannequin commented Jan 4, 2019

aixtools commented Jan 4, 2019

aixtools commented Jan 4, 2019

Ayappan mannequin commented Jan 8, 2019

aixtools commented Jan 9, 2019

vstinner commented Mar 4, 2019

miss-islington commented Mar 4, 2019

vstinner commented Mar 4, 2019

Build issue while compiling cpp files in AIX #79379

Build issue while compiling cpp files in AIX #79379

Comments

Ayappan mannequin commented Nov 9, 2018

Ayappan mannequin commented Nov 9, 2018

aixtools commented Nov 10, 2018

aixtools commented Nov 10, 2018

aixtools commented Nov 11, 2018

Ayappan mannequin commented Nov 12, 2018

kadler mannequin commented Jan 4, 2019

aixtools commented Jan 4, 2019

kadler mannequin commented Jan 4, 2019

aixtools commented Jan 4, 2019

aixtools commented Jan 4, 2019

Ayappan mannequin commented Jan 8, 2019

aixtools commented Jan 9, 2019

vstinner commented Mar 4, 2019

miss-islington commented Mar 4, 2019

vstinner commented Mar 4, 2019