classification
Title: Build issue while compiling cpp files in AIX
Type: compile error Stage: patch review
Components: Build Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ayappan, David.Edelsohn, Michael.Felt, kadler
Priority: normal Keywords: patch

Created on 2018-11-09 15:20 by Ayappan, last changed 2019-01-09 19:18 by Michael.Felt.

Files
File name Uploaded Description Edit
pip3_build_pandas_0001.text Michael.Felt, 2018-11-10 15:41 complete pip3 build pandas output
Pull Requests
URL Status Linked Edit
PR 10437 open kadler, 2018-11-09 17:34
Messages (12)
msg329522 - (view) Author: (Ayappan) Date: 2018-11-09 15:20
I am trying to build pandas-0.23.4 using python3 in AIX. 
Everything goes fine till it encounters a cpp file. Below is the compile error output.
building 'pandas._libs.window' extension
gcc -fPIC -maix64 -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DAIX_GENUINE_CPLUSCPLUS -D_LINUX_SOURCE_COMPAT -Wl,-brtl -O2 -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DAIX_GENUINE_CPLUSCPLUS -D_LINUX_SOURCE_COMPAT -Wl,-brtl -O2 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/freeware/lib64/python3.6/site-packages/numpy/core/include -I/opt/freeware/include/python3.6m -c pandas/_libs/window.cpp -o build/temp.aix-7.2-3.6/pandas/_libs/window.o -Wno-unused-function
In file included from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1821,
                 from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from pandas/_libs/window.cpp:587:
/opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^~~~~~~
g++ gcc -maix64 -pthread -bI:/opt/freeware/lib/python3.6/config-3.6m/python.exp -L. -L/usr/include -L/opt/freeware/include -L/usr/lib/threads -L/opt/freeware/lib64 -L/opt/freeware/lib -L/usr/lib64 -L/usr/lib -L. -L/usr/include -L/opt/freeware/include -L/usr/lib/threads -L/opt/freeware/lib64 -L/opt/freeware/lib -L/usr/lib64 -L/usr/lib build/temp.aix-7.2-3.6/pandas/_libs/window.o -L/opt/freeware/lib64 -o build/lib.aix-7.2-3.6/pandas/_libs/window.so
g++: error: gcc: No such file or directory
g++: error: unrecognized command line option '-bI:/opt/freeware/lib/python3.6/config-3.6m/python.exp'
error: command 'g++' failed with exit status 1

Seems like somewhere setting up the proper compile options for compiling cpp files is not proper. Any hint will be really useful.
msg329622 - (view) Author: Michael Felt (Michael.Felt) * Date: 2018-11-10 15:41
There are, perhaps, other issues as well.

After a build of "master"

a) pip3 install pandas
 - failed to find/download and build numpy
 - after "manual" pip3 install numpy and got to " six, python-dateutil, pytz, pandas"

Then:
 Running setup.py install for pandas ... error
    Complete output from command /opt/bin/python3.8 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-f5rvzoc0/pandas/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-kmi2p9sd/install-record.txt --single-version-externally-managed --compile:
...
    UPDATING build/lib.aix-6.1-3.8-pydebug/pandas/_version.py
    set build/lib.aix-6.1-3.8-pydebug/pandas/_version.py to '0.23.4'
    running build_ext
    building 'pandas._libs.algos' extension
    creating build/temp.aix-6.1-3.8-pydebug
    creating build/temp.aix-6.1-3.8-pydebug/pandas
    creating build/temp.aix-6.1-3.8-pydebug/pandas/_libs
    xlc_r -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/algos.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/algos.o -Wno-unused-function
    xlc_r: 1501-210 (S) command option Wno-unused-function contains an incorrect subargument
    error: command 'xlc_r' failed with exit status 40

    ----------------------------------------
Command "/opt/bin/python3.8 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-f5rvzoc0/pandas/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-kmi2p9sd/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-f5rvzoc0/pandas/

So, it looks like, using xlc - I may not have even gotten as far window.gcc due to issues in pandas (the incorrect flag -Wno-used-function - have to find where that came from!)
msg329625 - (view) Author: Michael Felt (Michael.Felt) * Date: 2018-11-10 16:17
Getting farther - after "hacking" pandas setup.py to not force a gcc flag.

Still not getting as far, I think, as the initial poster.

...
/opt/include/python3.8dm -c pandas/_libs/parsers.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/parsers.o
xlc_r -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/src/parser/tokenizer.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/src/parser/tokenizer.o
"/opt/include/python3.8dm/pyconfig.h", line 1538.9: 1506-236 (W) Macro name _POSIX_C_SOURCE has been redefined.
"/opt/include/python3.8dm/pyconfig.h", line 1538.9: 1506-358 (I) "_POSIX_C_SOURCE" is defined on line 160 of /usr/include/standards.h.
"/opt/include/python3.8dm/pyconfig.h", line 1553.9: 1506-236 (W) Macro name _XOPEN_SOURCE has been redefined.
"/opt/include/python3.8dm/pyconfig.h", line 1553.9: 1506-358 (I) "_XOPEN_SOURCE" is defined on line 156 of /usr/include/standards.h.
"/usr/include/unistd.h", line 171.17: 1506-343 (S) Redeclaration of lseek64 differs from previous declaration on line 169 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 171.17: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the previous return type "long".
"/usr/include/unistd.h", line 171.17: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
"/usr/include/sys/lockf.h", line 64.20: 1506-343 (S) Redeclaration of lockf64 differs from previous declaration on line 62 of "/usr/include/sys/lockf.h".
"/usr/include/sys/lockf.h", line 64.20: 1506-377 (I) The type "long long" of parameter 3 differs from the previous type "long".
"/usr/include/unistd.h", line 809.33: 1506-343 (S) Redeclaration of ftruncate64 differs from previous declaration on line 807 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 809.33: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
"/usr/include/unistd.h", line 845.33: 1506-343 (S) Redeclaration of truncate64 differs from previous declaration on line 843 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 845.33: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
"/usr/include/unistd.h", line 862.33: 1506-343 (S) Redeclaration of pread64 differs from previous declaration on line 859 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 862.33: 1506-377 (I) The type "long long" of parameter 4 differs from the previous type "long".
"/usr/include/unistd.h", line 863.33: 1506-343 (S) Redeclaration of pwrite64 differs from previous declaration on line 860 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 863.33: 1506-377 (I) The type "long long" of parameter 4 differs from the previous type "long".
"/usr/include/unistd.h", line 942.25: 1506-343 (S) Redeclaration of fclear64 differs from previous declaration on line 939 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 942.25: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the previous return type "long".
"/usr/include/unistd.h", line 942.25: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
"/usr/include/unistd.h", line 943.25: 1506-343 (S) Redeclaration of fsync_range64 differs from previous declaration on line 940 of "/usr/include/unistd.h".
"/usr/include/unistd.h", line 943.25: 1506-377 (I) The type "long long" of parameter 3 differs from the previous type "long".
"pandas/_libs/src/parser/tokenizer.c", line 265.40: 1506-280 (W) Function argument assignment between types "long long*" and "unsigned long*" is not allowed.
"pandas/_libs/src/parser/tokenizer.c", line 292.30: 1506-280 (W) Function argument assignment between types "long long*" and "unsigned long*" is not allowed.
"pandas/_libs/src/parser/tokenizer.c", line 323.28: 1506-280 (W) Function argument assignment between types "long long*" and "unsigned long*" is not allowed.
error: command 'xlc_r' failed with exit status 1
root@x066:[/data/prj/python/pandas-0.23.4]set | grep OBJ
root@x066:[/data/prj/python/pandas-0.23.4]python3
Python 3.8.0a0 (heads/master-dirty:b9498e2367, Nov 10 2018, 13:51:41) [C] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.maxsize()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable
>>> sys.maxsize
2147483647

+++++
Will uninstall and build a 64-bit python3 and see if that helps.
+++++
msg329684 - (view) Author: Michael Felt (Michael.Felt) * Date: 2018-11-11 20:10
Still getting the same errors, even with 64-bit build, so still not close to testing the actual problem.
FYI: last time I build pandas the version was 0.19.0 - then it was 'simple'.

Anyway, short of a simple (test) python module that includes c++ code, I'll look at changing the pandas build files to force any c++ file to the front.

More - (asap, but still when I have time)
msg329722 - (view) Author: (Ayappan) Date: 2018-11-12 07:17
The PR 10437 worked !!!. 
Now i am able to compile pandas-0.23.4


Michael Felt,

About the re-declaration issues you are facing, that is because of the "#define _LARGE_FILES 1" in pyconfig-ppc64.h file.
For 64bit build we actually don't need it.
For 32bit build, we need it and also we have to pass the flag "-D_LARGE_FILES" through the _sysconfigdata_m_aix6_.py , otherwise one will get this re-declaration error.
msg332961 - (view) Author: Kevin (kadler) * Date: 2019-01-04 05:31
Just a friendly ping that there's a PR for this bug waiting to be reviewed.
msg332971 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-04 11:54
While the PR probably solves this - there is a 'bug' in pandas (I expect) that prevents me from completing the test - as, I expect LONG before the .cpp source is to be compiled - there is a error because a wrong flag is passed to the compiler (-Wno-unused-function) and the build stops.

Have not had the time to disect pandas so that I can get a .cpp source compiled.

+++++
Successfully installed pip-18.1
root@x066:[/data/prj/python/git/kadler]pip list
/opt/lib/python3.8/site-packages/pip/_vendor/html5lib/_trie/_base.py:3: DeprecationWdeprecated, and in 3.8 it will stop working
  from collections import Mapping
Package    Version
---------- -------
pip        18.1
setuptools 39.0.1
root@x066:[/data/prj/python/git/kadler]pip install numpy
/opt/lib/python3.8/site-packages/pip/_vendor/html5lib/_trie/_base.py:3: DeprecationWdeprecated, and in 3.8 it will stop working
  from collections import Mapping
Collecting numpy
/opt/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py:220: PendingDepreca
  warnings.warn(
  Using cached https://files.pythonhosted.org/packages/2d/80/1809de155bad674b494248b
Installing collected packages: numpy
  Running setup.py install for numpy ... done
Successfully installed numpy-1.15.4

...
Installing collected packages: six, python-dateutil, pytz, pandas
...
    copying pandas/io/formats/templates/html.tpl -> build/lib.aix-6.1-3.8-pydebug/pandas/io/formats/templates
    UPDATING build/lib.aix-6.1-3.8-pydebug/pandas/_version.py
    set build/lib.aix-6.1-3.8-pydebug/pandas/_version.py to '0.23.4'
    running build_ext
    building 'pandas._libs.algos' extension
    creating build/temp.aix-6.1-3.8-pydebug
    creating build/temp.aix-6.1-3.8-pydebug/pandas
    creating build/temp.aix-6.1-3.8-pydebug/pandas/_libs
    xlc_r -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/algos.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/algos.o -Wno-unused-function
    xlc_r: 1501-210 (S) command option Wno-unused-function contains an incorrect subargument
    error: command 'xlc_r' failed with exit status 40

    ----------------------------------------
Command "/opt/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-xnscgehv/pandas/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-0w46p413/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-xnscgehv/pandas/
msg332984 - (view) Author: Kevin (kadler) * Date: 2019-01-04 16:08
Ah. We always compile with GCC, so would not have hit that particular problem.
msg332989 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-04 18:37
On 04/01/2019 17:08, Kevin wrote:
> Kevin <kadler@us.ibm.com> added the comment:
>
> Ah. We always compile with GCC, so would not have hit that particular problem.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35198>
> _______________________________________
>
FYI. I found a 'hack' in ./setup.py to skip adding the argument.
Progressing.
msg332991 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-04 19:04
Further along - however, I never get to the "link" routine.

Again, this is likely a pandas coding issue - currently python is calling xlc_r ..., but when I manually modify it to xlC_r I get the same error.

xlc_r -D_LARGE_FILES -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs -I./pandas/_libs -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/window.cpp -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/window.o
"pandas/_libs/window.cpp", line 9943.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 10030.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 11165.21: 1540-0063 (S) The text "(" is unexpected.
error: command 'xlc_r' failed with exit status 1
root@x066:[/data/prj/aixtools/git/pandas-master]ebug/pandas/_libs/window.o                    <
"pandas/_libs/window.cpp", line 9943.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 10030.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 11165.21: 1540-0063 (S) The text "(" is unexpected.

root@x066:[/data/prj/aixtools/git/pandas-master]xlC_r -D_LARGE_FILES -O -I/opt/include -O2 -qm>
"pandas/_libs/window.cpp", line 9943.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 10030.23: 1540-0063 (S) The text "(" is unexpected.
"pandas/_libs/window.cpp", line 11165.21: 1540-0063 (S) The text "(" is unexpected.

FYI. Will look more tomorrow, but if you have an idea, like the -D_LARGE_FILES fix, I am much obliged.
msg333231 - (view) Author: (Ayappan) Date: 2019-01-08 14:40
Not sure what went wrong here.
I used gcc & g++ and didn't hit this issue.
msg333342 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-09 19:18
On 08/01/2019 15:40, Ayappan wrote:
> Ayappan <ayappanec@gmail.com> added the comment:
>
> Not sure what went wrong here.
> I used gcc & g++ and didn't hit this issue.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35198>
> _______________________________________
>
Well, it is probably an issue with xlc as I have vac.C and
vacpp.cmp.core installed (aka xlc and xlC). These days it seems only gcc
environment gets tested in depth and portability issues with "posix"
compilers get missed.

GCC is a fine compiler - just not the one I am using for a number of
"logistical" reasons.

Thanks for the update!

Michael
History
Date User Action Args
2019-01-09 19:18:38Michael.Feltsetmessages: + msg333342
2019-01-08 14:40:04Ayappansetmessages: + msg333231
2019-01-04 19:04:33Michael.Feltsetmessages: + msg332991
2019-01-04 18:37:32Michael.Feltsetmessages: + msg332989
2019-01-04 16:08:22kadlersetmessages: + msg332984
2019-01-04 11:54:42Michael.Feltsetmessages: + msg332971
2019-01-04 05:31:44kadlersetnosy: + kadler
messages: + msg332961
2018-11-12 07:17:05Ayappansetmessages: + msg329722
2018-11-11 20:10:55Michael.Feltsetmessages: + msg329684
2018-11-10 16:17:42Michael.Feltsetversions: + Python 3.8
2018-11-10 16:17:23Michael.Feltsetmessages: + msg329625
2018-11-10 15:41:08Michael.Feltsetfiles: + pip3_build_pandas_0001.text

messages: + msg329622
2018-11-09 19:00:38ned.deilysetnosy: + David.Edelsohn, Michael.Felt
2018-11-09 17:34:33kadlersetkeywords: + patch
stage: patch review
pull_requests: + pull_request9711
2018-11-09 15:20:45Ayappancreate