classification
Title: Python 64-bit build on HP Itanium - Executable built successfully but modules failed with HP Compiler
Type: compile error Stage: resolved
Components: Documentation Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: adiroiban, cheryl.sabella, loewis, michael-o, orsenthil, wah meng
Priority: normal Keywords: patch

Created on 2011-09-16 11:42 by wah meng, last changed 2019-01-04 19:52 by cheryl.sabella. This issue is now closed.

Files
File name Uploaded Description Edit
README.patch orsenthil, 2011-09-21 17:44
Messages (14)
msg144123 - (view) Author: Wong Wah Meng (wah meng) Date: 2011-09-16 11:42
Server Info: HP-UX B.11.31 Itanium ia64.
Evaluation copy is used however the license is being purchased, hopefully this are not these errors are not caused by evaluation copy.
 
# swlist -l product | grep Compiler
  ACXX                  C.06.26.EVAL   HP C/aC++ Compiler 
  C-ANSI-C              C.06.26.EVAL   HP C/aC++ Compiler 
  COMPLIBS              B.11.31        Compiler Support Libraries 


I encountered more failures in module build with my HP-UX ANSI C Compiler versus gcc. I wonder why. I need a Oracle plug-in later, i.e. cx_Oracle and the reason I do not want to ignore these errors now because I thought cx_Oracle will have to (or, better) be built using HP compiler, not gcc for support from Oracle. 

I read through the README file, I followed the instruction to set the required environment variables before the configure is run, and unset them before make is run, also edited Makefile to remove Optimization and the -Olimit 1500. However, it gives me a lot more modules that are failed to be built versus 2 when gcc is used. At the bottom of the mail we can see what failed to be built. 

I have summarized my inquiry as follow:-
1.) Despite following the instruction to edit the Makefile before make is run, I am surprise to see that the Makefile content is changed again when the executable is generated. The changed I edited is restored. Is this expected and did the compiler make use of the content that I edited when it did the build and link? 

$ diff Makefile Makefile_Original
61,62c61,62
< OPT=          -DNDEBUG -O
< BASECFLAGS=   +DD64 -Olimit 1500
---
> OPT=          -DNDEBUG
> BASECFLAGS=   +DD64
 
2.) A lot of failed module and I suspected it is due +DD64 nopt recognized by the compiler. Why the HP-UX compiler did not recognize the option? 

ld -b +DD64 -lxnet build/temp.hp-ux-B.11.31-ia64-2.7/home/r32813/Build/2.7.1/Python-2.7.1/Modules/mathmodule.o build/temp.hp-ux-B.11.31-ia64-2.7/home/r32813/Build/2.7.1/Python-2.7.1/Modules/_math.o -L/usr/local/lib -lm -o build/lib.hp-ux-B.11.31-ia64-2.7/math.so
ld: Unrecognized argument: +DD64
Fatal error.

3.) This is a general question, in the README file there isn't specific instruction on how to enable gcc for 64-bit build. Does gcc support 64-bit build and how to?

4.) These are warning message. I have seen a lot of them. Am I right I can ignore them? I don't see them when I use gcc though. 

        cc -Ae -c +DD64 -g -DNDEBUG  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c
"Include/pyfpe.h", line 8: warning #3750-D: "\" followed by white space is not a line splice

Thanks in advance for your input. Here is the output of my make using HP C compiler. 
 
Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel   
_sqlite3           _ssl               _tkinter        
bsddb185           bz2                dl              
gdbm               imageop            linuxaudiodev   
ossaudiodev        readline           spwd            
sunaudiodev        zlib                               
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_bisect            _codecs_cn         _codecs_hk      
_codecs_iso2022    _codecs_jp         _codecs_kr      
_codecs_tw         _collections       _csv            
_ctypes            _ctypes_test       _elementtree    
_functools         _heapq             _hotshot        
_io                _json              _locale         
_lsprof            _md5               _multibytecodec 
_multiprocessing   _random            _sha            
_socket            _struct            _testcapi       
array              audioop            binascii        
cmath              cPickle            crypt           
cStringIO          datetime           dbm             
fcntl              future_builtins    grp             
itertools          math               mmap            
nis                operator           parser          
pyexpat            resource           select          
strop              syslog             termios         
time               unicodedata                        

running build_scripts
msg144132 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-09-16 13:21
I think you may want to ask these questions on the Python mailing list:

http://mail.python.org/mailman/listinfo/python-list


This is the Python bug-tracker, and it's not obvious to me that
any of your points is a bug in Python.
msg144134 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-09-16 13:50
1) I cannot reproduce this.

2) ld is the linker and not the compiler.

3) and 4) Should definitely be asked on python-list.
msg144136 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-09-16 16:04
> I encountered more failures in module build with my 
> HP-UX ANSI C Compiler versus gcc. I wonder why.

That's really simple to answer. HP-UX is not well supported. You are pretty much on your own.
msg144320 - (view) Author: Wong Wah Meng (wah meng) Date: 2011-09-20 07:32
I think there is something that is not working here at least for the instructions in build README file about setting up 4 environment variables required for HP Itanium 64-bit build using HP compilers. 
                CC=cc
                CXX=aCC
                BASECFLAGS="+DD64"
                LDFLAGS="+DD64 -lxnet"
When above 4 environment variables are set, the configure script will pick them up and build the Makefile that passes in LDFLAGS into the linker. The linker doesn't seem to recognize +DD64 on HP-UX. 

ld -b +DD64 -lxnet build/temp.hp-ux-B.11.31-ia64-2.7/home/r32813/Build/2.7.1/Python-2.7.1/Modules/mathmodule.o build/temp.hp-ux-B.11.31-ia64-2.7/home/r32813/Build/2.7.1/Python-2.7.1/Modules/_math.o -L/usr/local/lib -lm -o build/lib.hp-ux-B.11.31-ia64-2.7/math.so
ld: Unrecognized argument: +DD64
Fatal error.

I did a workaround by only setting "CC=cc +DD64" and "CXX=aCC" and did not set the other 2 environment variables, and able to build python 64-bit without the failed modules that were due to invalid flag passed into the linker. So I am re-opening the issue here to see if this is something we can amend in the instruction of README file for HP 64-bit build using HP Compiler.
msg144323 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-09-20 10:21
The README looks outdated. This isn't surprising, since probably no one
here has access to the HP compiler.

If you want to improve it, please try this:

make distclean
./configure CC=cc CFLAGS="+DD64"
make test



I don't think the linker should be invoked as 'ld'. Several sites
suggest that the HP compiler can be used a the linker front end
(in the same manner as gcc).

http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp?topic=/com.ibm.db2.udb.doc/ad/r0007656.htm
msg144324 - (view) Author: Wong Wah Meng (wah meng) Date: 2011-09-20 10:44
Thanks! I am gonna try this out again. 

Actually I noticed this when you mentioned about using cc as the linker front end. 

I followed instruction in the README file, after I edited Makefile to remove the -O Limit 1500 line, and after python executable is built, the Makefile is overwritten (maybe re-generated by the compiler) again and my changes of -O and -Limit 1500 is added back. However that doesn't create big damage like the +DD64 to the linker (ld). 

I see. Since I have got HP Compiler, I will provide some updates here once I have a newer finding. I have submitted my question of "Why ld doesn't recognize +DD64" flag to HP and they are checking for me. The progress over there is slow. 

Thanks for your reply!

Regards,
Wah Meng
msg144374 - (view) Author: Wong Wah Meng (wah meng) Date: 2011-09-21 11:45
Good news is now I am still using ld, like what is generated in the Makefile file by configure script, and able to do the linking process for the built-in modules successfully. 

I only need to set CC="cc +DD64" and "CXX=aCC" and the process make process is able to build python executable 64-bit and all the built-in modules where the necessary bit can be found.

For other built-in modules, for example, _tkinter, I am also able to build it this time by setting these 3 environment variables where ~/local is where I installed my tcl/tk binary.

export CC="cc +DD64 -I/home/r32813/local/include"
export CXX=aCC
export LDFLAGS="-L/home/r32813/local/lib -lxnet"

I am finally able to build python 64-bit and built-in library on HP ia64 using HP compiler aCC. Thanks for the input everyone. :)
msg144375 - (view) Author: Wong Wah Meng (wah meng) Date: 2011-09-21 11:48
Closing the issue. Please contact me if the README file for HP-UX Itanium 64-bit build should be amended.
msg144385 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2011-09-21 17:32
Wah Meng and I were discussing about this in a separate thread as well.
I think, it is a good idea to improve the Readme for this issue.

Wah Meng,  since you already went at length to contact HP and clarify linker issue with 64 bit and found a solution to it, it would be a good idea to document it.

Is this the only change for the successful compilation - CC="cc +DD64" and CXX="aCC" .

If there are more details, you can provide changes to the Readme in the form a diff and I shall commit to the trunk.

Thanks!
msg144386 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-09-21 17:44
> I think, it is a good idea to improve the Readme for this issue.

+1.

Wah Meng: Building everything is not enough, does 'make test'
complete successfully?

For gcc, Python2.7 relies on two critical options, -fno-strict-aliasing
and -fwrapv. The HP compiler might need similar options.
msg144387 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2011-09-21 17:44
Wah meng - Instructions to get the _tkinter compile would be a good idea too.  Also, can you verify the same instructions would work for 3.2 version of Python?
msg323776 - (view) Author: Michael Osipov (michael-o) * Date: 2018-08-20 08:00
Cannot reproduced on master:
export LDFLAGS="-L/usr/local/lib/hpux64 +DD64"
export CFLAGS=+DD64

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses_panel         _gdbm
_lzma                 _sqlite3              _tkinter
_uuid                 ossaudiodev           readline
spwd                  zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Failed to build these modules:
_ctypes               _dbm                  _elementtree
cmath                 math                  pyexpat


Following modules built successfully but were removed because they could not be imported:
_asyncio

running build_scripts
copying and adjusting /var/osipovmi/cpython/Tools/scripts/pydoc3 -> build/scripts-3.8
copying and adjusting /var/osipovmi/cpython/Tools/scripts/idle3 -> build/scripts-3.8
copying and adjusting /var/osipovmi/cpython/Tools/scripts/2to3 -> build/scripts-3.8
changing mode of build/scripts-3.8/pydoc3 from 640 to 755
changing mode of build/scripts-3.8/idle3 from 640 to 755
changing mode of build/scripts-3.8/2to3 from 640 to 755
renaming build/scripts-3.8/pydoc3 to build/scripts-3.8/pydoc3.8
renaming build/scripts-3.8/idle3 to build/scripts-3.8/idle3.8
renaming build/scripts-3.8/2to3 to build/scripts-3.8/2to3-3.8

$ file ./python
./python:       ELF-64 executable object file - IA64
$ ./python
Python 3.8.0a0 (heads/bpo-34412:f1331c0e83, Aug 20 2018, 09:53:38) [C] on hp-ux11
Type "help", "copyright", "credits" or "license" for more information.
>>>

The necessary bits are libraries in 64 bit which I have only in 32 bit.
This issue can be closed.
msg332996 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2019-01-04 19:52
Closed based on msg323776.
History
Date User Action Args
2019-01-04 19:52:42cheryl.sabellasetstatus: open -> closed

nosy: + cheryl.sabella
messages: + msg332996

resolution: out of date
2018-08-20 08:00:09michael-osetnosy: + michael-o
messages: + msg323776
2014-05-13 22:21:37skrahsetnosy: - skrah
2012-04-12 21:56:15adiroibansetnosy: + adiroiban
2011-09-21 17:44:58orsenthilsetfiles: + README.patch
keywords: + patch
messages: + msg144387
2011-09-21 17:44:35skrahsetmessages: + msg144386
2011-09-21 17:32:39orsenthilsetstatus: closed -> open

assignee: orsenthil
components: + Documentation, - Build
versions: + Python 3.2, Python 3.3
nosy: + orsenthil

messages: + msg144385
resolution: works for me -> (no value)
2011-09-21 11:48:42wah mengsetstatus: open -> closed

messages: + msg144375
2011-09-21 11:45:21wah mengsetmessages: + msg144374
2011-09-20 10:44:33wah mengsetmessages: + msg144324
2011-09-20 10:21:04skrahsetmessages: + msg144323
2011-09-20 07:32:39wah mengsetstatus: closed -> open

messages: + msg144320
2011-09-16 16:04:17loewissetnosy: + loewis
messages: + msg144136
2011-09-16 13:50:37skrahsetstatus: open -> closed
resolution: works for me
messages: + msg144134

stage: resolved
2011-09-16 13:21:28skrahsetnosy: + skrah
messages: + msg144132
2011-09-16 11:42:59wah mengcreate