classification
Title: Improve ./configure's support for 32/64-bit debug|release|profiled builds w/ vendor (non-gcc) compilers on proprietary UNIX systems (Solaris/HP-UX/AIX et al).
Type: enhancement Stage:
Components: Build Versions: Python 3.4, Python 2.7, Python 2.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: trent Nosy List: Arfrever, eric.araujo, gix, jcea, jkloth, loewis, ned.deily, trent
Priority: low Keywords:

Created on 2012-09-18 13:40 by trent, last changed 2014-05-13 21:59 by skrah.

Messages (10)
msg170643 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-09-18 13:40
Gripe: if you want a 64-bit, non-gcc (i.e. vendor's cc) build on a proprietary UNIX system (i.e. Solaris, HP-UX, AIX etc), "you're going to have a bad time".

Coercing a 64-bit build from a vendor's cc currently requires explicit CFLAGS/LDFLAGS/configure gymnastics for each platform.  It's a pain.

Initial goal: use this issue to document the gymnastics.

Assumption: once all the different techniques are documented, it'll be easier to assess what changes would be appropriate to configure.in.

Side bar design question: on BSD/Linux/OSX w/ gcc/clang, if you're on an amd64 platform, you'll get a 64-bit build.  This isn't the case for Solaris, HP-UX and AIX -- the default is always 32-bit.  You need aforementioned gymnastics to coerce a 64-bit build.

Is this by design?  If so, then I guess I'm proposing that ./configure should have a `--with-64`-type argument that'll generate a 64-bit build.

If not, then we need to decide whether to change the default behavior such that ./configure always generates a 64-bit build if you're on a 64-bit platform -- if you want a 32-bit build, you need to explicitly tell ./configure (i.e. --with-32).

Changing the default is probably only viable for 3.4 onwards.  It'd be nice if 2.7->3.3 had generic configure support for 64-bit builds though (via --with-64).

XXX TODO for trent: review autoconf's offerings... getting 64-bit builds from vendor cc's can't be that unusual.
msg170644 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-09-18 13:43
On the s10 slave (Solaris 10/nitrogen) for 3.x:

(cpython@nitrogen:ttypts/4) (Tue/12:32) ..                                                                                                                                     
% ../../src/configure --with-pydebug --without-gcc CFLAGS="-m64 -mt -xcheck=%all -g3 -xarch=native -xchip=native" CPPFLAGS="-IInclude" OPT="" LDFLAGS="-m64 -mt -xcheck=%all -g3 -xarch=native -xchip=native" CC=/opt/solarisstudio12.3/bin/cc
msg170661 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-09-18 18:37
Solaris 10 release (i.e. optimized) build requires the following:

../../src/configure --without-gcc CFLAGS="-v -fsimple=0 -m64 -mt=yes -xbuiltin -xhwcprof -xF -xarch=native -xchip=native -fma=fused -g -xO5 -xlibmil -xlibmopt -xmemalign=8s -xregs=frameptr -xtarget=native -xbuiltin=%all -library=sunperf" CPPFLAGS="-IInclude" OPT="" LDFLAGS="-v  -fsimple=0 -m64 -mt=yes -xbuiltin -xhwcprof -xF -xarch=native -xchip=native -fma=fused -g -xO5 -xlibmil -xlibmopt -xmemalign=8s -xbuiltin=%all -xregs=frameptr -xtarget=native -library=sunperf" CC=/opt/solarisstudio12.3/bin/cc 

(Due to indirect linker invocation via cc, I'm purposely duplicating CFLAGS <-> LDFLAGS for now.  I'll refine later.)
msg171121 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-09-24 12:50
I'm glad I found this. Without the long incantation _decimal is
seriously miscompiled on SPARC/Solaris 10/Sun Studio 12.3. I was
using just ...

  ./configure --without-gcc CFLAGS=-m64 LDFLAGS=-m64

..., and that seems to trigger an optimizer bug that produces
different output for _mpd_div_words_r() when called with exactly
the same input at different locations.


The line that Trent gave appears to work.
msg171855 - (view) Author: GOGER Valentin Eugen (gix) Date: 2012-10-03 06:14
Hello
On AIX 64bit (see http://bugs.python.org/issue6600) you should add in CFLAGS  -U__STR__ to bypass xlc(_r) define/inlining issue.
msg173334 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-10-19 12:52
On Tue, Sep 18, 2012 at 06:40:35AM -0700, Trent Nelson wrote:
> Side bar design question: on BSD/Linux/OSX w/ gcc/clang, if you're on
> an amd64 platform, you'll get a 64-bit build.  This isn't the case for
> Solaris, HP-UX and AIX -- the default is always 32-bit.  You need
> aforementioned gymnastics to coerce a 64-bit build.
> 
> Is this by design?  If so, then I guess I'm proposing that ./configure
> should have a `--with-64`-type argument that'll generate a 64-bit
> build.
> 
> If not, then we need to decide whether to change the default behavior
> such that ./configure always generates a 64-bit build if you're on a
> 64-bit platform -- if you want a 32-bit build, you need to explicitly
> tell ./configure (i.e. --with-32).
> 
> Changing the default is probably only viable for 3.4 onwards.  It'd be
> nice if 2.7->3.3 had generic configure support for 64-bit builds
> though (via --with-64).
> 
> XXX TODO for trent: review autoconf's offerings... getting 64-bit
> builds from vendor cc's can't be that unusual.

    Just wanted to make note of a precedent I noticed yesterday whilst
    coercing local builds of Tcl, expect and a few other things on the
    Solaris 10 SPARC box: they all use `--enable-64bit` as the arg to
    configure to enable 64-bit builds.

    Additionally, none of them default to 64-bit out-of-the-box; they
    all inherit the underlying compilers "default" behavior.  Which
    means gcc/clang gets you 64-bit on amd64, but a cc build on a
    64-bit host may not.  (I need to check a few more permutations on
    this, like what happens with gcc on SPARC/IA64 etc if the host OS
    is 64-bit -- does it default to 64-bit?)

    Side note: some of those projects had *much* nicer autoconf-fu
    than we do.  Tcl implements the --enable-64bit stuff in aclocal.m4.

    So, I'm changing my stance from what I quoted above.  We should try
    mimic the precedent set by other projects:

        - Use --enable-64bit as the flag.
        - "Default" 64-bit builds depend on the underlying compiler.
msg177443 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-12-14 05:26
I spent a little time on this yesterday.  Here's what I came up with.  The idea is to have a standalone block in configure.ac that kicks in when --without-gcc is specified (or if $CC != "gcc") *IFF* CFLAGS/CPPFLAGS haven't been provided by the user.

If they have, we warn, and skip the block.  Rationale is wanting to cover two cases: a) experienced user consciously overriding our attempts at sensible defaults in that block, b) inexperienced user who has no clue what the sensible defaults are but has otherwise inadvertently set CFLAGS/CPPFLAGS (or perhaps just inherited such settings from their existing environment).

So, here's what the "block" looks like currently, albeit just for IRIX at the moment (and this was against 2.7).  I'll eventually expand it to cover Solaris/SPARC|AMD64, AIX, HP-UX (PA-RISC/IA64) and Tru64.

(Also note the new `--enable-64bit` arg, which is intended to be used on these platforms as a way to coerce easy 64-bit builds.  The functionality doesn't apply on x86/x64 "free" *nix platforms that use clang/gcc -- those will default to using whatever memory architecture is being used by the underlying system -- if you want something else, that's the job of cross-compilation, which is a different problem this particular issue is attempting to address.)

+AC_MSG_CHECKING(for --enable-64bit)
+AC_ARG_ENABLE(
+    64bit,
+    AS_HELP_STRING(
+        [--enable-64bit],
+        [attempt 64-bit build (use with --without-gcc)]
+    )
+)
+if test -z "$enable_64bit"; then
+    AC_MSG_RESULT(no)
+else
+    AC_MSG_RESULT(yes)
+fi
+if test -n "$enable_64bit" && test "$without_gcc" != "yes"; then
+    AC_MSG_WARN([
+        --enable-64bit has no effect unless --without-gcc is also specified
+    ])
+fi
+
+# If we're not using GCC, we're probably on a proprietary UNIX, relying
+# on a proprietary compiler, running on non-x86/x64 hardware.  Autoconf
+# is pretty bad at picking sensible defaults for compiler flags in this
+# situation, so, let's try and do as much as we can ourselves.  The aim
+# is to pick the optimal settings for each of the following permutations:
+#
+# Debug builds, 32-bit and 64-bit:
+#   ./configure --without-gcc --with-pydebug
+#   ./configure --without-gcc --with-pydebug --enable-64bit
+#
+# Optimized release builds, 32-bit and 64-bit:
+#   ./configure --without-gcc
+#   ./configure --without-gcc --enable-64bit
+#
+if test "$without_gcc" = "yes" || test "$CC" != "gcc"; then
+    # This whole block assumes we know more than a) the user, and b) autoconf.
+    # If we detect CFLAGS/CPPFLAGS, then we can't safely assume a) anymore,
+    # so print a message instead of doing any customisation.
+    if test -n "$CFLAGS" || test -n "$CPPFLAGS"; then
+        AC_MSG_WARN([
+            You have defined CFLAGS and/or CPPFLAGS in conjunction with
+            --with-gcc; skipping attempt at setting flags to sensible
+            defaults -- make sure you know what you are doing.
+        ])
+    else
+        case $MACHDEP in
+            irix6)
+                # Trying to separate IRIX from C99 and still end up with a
+                # working executable is an exercise in futility; it has a
+                # much greater link between `cc -c99` and available system
+                # facilities/headers than other OSes (which primarily rely
+                # on the usual XPG4/XOPEN_SOURCE etc defines).  Using the
+                # c99 compiler is the lesser of two evils.
+                CC=c99
+                CXX=CC
+                # `-diag_error 1035` makes the compiler treat #error defines
+                # as actual errors (and abort compilation), and not warnings.
+                CFLAGS="-diag_error 1035"
+                # Try to target the underlying host hardware.
+                _platform=`hinv -c processor |
+                           head -n 1         |
+                           cut -f 4 -d ' '`
+                _processor=`hinv -c processor |
+                            head -n 2         |
+                            tail -n 1         |
+                            cut -f 3 -d ' '`
+                CFLAGS="$CFLAGS -TARG:platform=$_platform"
+                CFLAGS="$CFLAGS -TARG:processor=$_processor"
+
+                if test "$with_pydebug" = "yes"; then
+                    OPT="-O -g2"
+                else
+                    OPT="-g3 -Ofast=$_platform -OPT:Olimit=5500"
+                fi
+
+                if test -n "$enable_64bit"; then
+                    CFLAGS="-64 $CFLAGS"
+                    LDFLAGS="-64"
+                else
+                    CFLAGS="-n32 $CFLAGS"
+                    LDFLAGS="-n32"
+                fi
+                ;;
+            *)  ;;
+        esac
     fi
 fi
msg177445 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-12-14 06:28
Without having reviewed the proposed change in detail, a couple of comments.  On current OS X systems and others, the compiler could be clang which perhaps should be treated as gcc for most autoconf purposes.  Also, why are we putting any effort into supporting IRIX anymore?  Both IRIX and the MIPS processor lines it runs on was retired by SGI in 2007.  That's why support for IRIX was pulled from Python 3.  It's a dead end.
msg177459 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-12-14 13:18
On Thu, Dec 13, 2012 at 10:28:00PM -0800, Ned Deily wrote:
> 
> Ned Deily added the comment:
> 
> Without having reviewed the proposed change in detail, a couple of
> comments.  On current OS X systems and others, the compiler could be
> clang which perhaps should be treated as gcc for most autoconf
> purposes.

    I'm not intending to target OS X in the autoconf block I referred
    to; it's a popular platform for developers and users, and doesn't
    have any of the problems that the proprietary *NIXes/compilers
    have.

    This work is solely aimed at Solaris, Tru64, HP-UX, AIX and IRIX.

> Also, why are we putting any effort into supporting IRIX
> anymore?

    We?  It's just me :-)  So a more appropriate question might be why
    am I bothering to put effort into supporting it?  And the answer to
    that is simply because I can; Snakebite has an SGI Origin running
    the latest version of IRIX with the MIPSpro compiler suite, which
    is all the hardware I need to keep things chugging along.  (IRIX
    also has a huge active "fan" base of users that run a lot of OSS
    software -- nekoware et al.)

> Both IRIX and the MIPS processor lines it runs on was retired by SGI
> in 2007.  That's why support for IRIX was pulled from Python 3.

    Support was pulled for, to quote PEP 11, "IRIX 4 and --with-sgi-dl";

    PEP 11 also has a nice clause that basically says platform support
    is primarily based on having an active maintainer willing to keep
    everything in shape -- I'm happy to be marked as the maintainer for
    all the aforementioned *NIX platforms.

    Martin made a good point a few weeks ago when we discussed this on
    infrastructure@; his concern was the effort involved in supporting
    additional platforms could detract developers from more important
    tasks.

    I agree -- they're esoteric platforms at best, EOL at worst, and
    just because I'm maintaining them shouldn't mean other developers
    have to worry about breaking them.  They're not anywhere near as
    important as the big 3 (Linux/Windows/OSX).  If they do break,
    though, I'll keep fixing them as long as I'm actively maintaining
    support for that platform.

    The motivation behind this particular change is simple: it took
    about three days to nail down the exact flags to use on Solaris
    SPARC 32-bit to get a working ./python (with lots of referring
    to various Sun/Oracle compiler docs).  No-one else should have
    to go through this much effort -- ./configure should pick the
    right flags out of the box for the following permutations:

    32/64 bit debug builds:
        ./configure --without-gcc --with-pydebug
        ./configure --without-gcc --with-pydebug --enable-64bit

    32/64 bit release builds:
        ./configure --without-gcc
        ./configure --without-gcc --enable-64bit

    (And again, just to clarify, none of this work will remotely
     affect Linux, OS X or the *BSDs.  It doesn't take three days
     of reading compiler manuals to get working builds on those
     platforms.)
msg205807 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-12-10 12:40
Trent, did the HPUX CFLAGS/LDFLAGS change on the buildbots?  I think
_decimal used to compile on the PA-RISC bot, but now there is an error.

Perhaps -AC99 or something like that is missing.
History
Date User Action Args
2014-05-13 21:59:49skrahsetnosy: - skrah
2013-12-10 12:40:39skrahsetmessages: + msg205807
2012-12-14 13:18:29trentsetmessages: + msg177459
title: Improve ./configure's support for 32/64-bit debug|release|profiled builds w/ vendor (non-gcc) compilers on proprietary UNIX systems (Solaris/HP-UX/AIX et al). -> Improve ./configure's support for 32/64-bit debug|release|profiled builds w/ vendor (non-gcc) compilers on proprietary UNIX systems (Solaris/HP-UX/AIX et al).
2012-12-14 06:28:00ned.deilysetnosy: + ned.deily
messages: + msg177445
2012-12-14 05:26:57trentsetmessages: + msg177443
2012-10-19 12:52:53trentsetmessages: + msg173334
title: Improve ./configure's support for 32/64-bit debug|release|profiled builds w/ vendor (non-gcc) compilers on proprietary UNIX systems (Solaris/HP-UX/AIX et al). -> Improve ./configure's support for 32/64-bit debug|release|profiled builds w/ vendor (non-gcc) compilers on proprietary UNIX systems (Solaris/HP-UX/AIX et al).
2012-10-03 06:14:20gixsetversions: + Python 2.6, Python 2.7
2012-10-03 06:14:04gixsetnosy: + gix
messages: + msg171855
2012-09-28 14:47:22jceasetnosy: + jcea
2012-09-24 16:30:56Arfreversetnosy: + Arfrever
2012-09-24 12:50:32skrahsetnosy: + skrah
messages: + msg171121
2012-09-18 18:37:27trentsetmessages: + msg170661
2012-09-18 16:04:37eric.araujosetnosy: + loewis, eric.araujo

versions: - Python 2.7
2012-09-18 13:46:58jklothsetnosy: + jkloth
2012-09-18 13:43:48trentsetmessages: + msg170644
2012-09-18 13:40:35trentcreate