Gripe: if you want a 64-bit, non-gcc (i.e. vendor's cc) build on a proprietary UNIX system (i.e. Solaris, HP-UX, AIX etc), "you're going to have a bad time".

Coercing a 64-bit build from a vendor's cc currently requires explicit CFLAGS/LDFLAGS/configure gymnastics for each platform.  It's a pain.

Initial goal: use this issue to document the gymnastics.

Assumption: once all the different techniques are documented, it'll be easier to assess what changes would be appropriate to configure.in.

Side bar design question: on BSD/Linux/OSX w/ gcc/clang, if you're on an amd64 platform, you'll get a 64-bit build.  This isn't the case for Solaris, HP-UX and AIX -- the default is always 32-bit.  You need aforementioned gymnastics to coerce a 64-bit build.

Is this by design?  If so, then I guess I'm proposing that ./configure should have a `--with-64`-type argument that'll generate a 64-bit build.

If not, then we need to decide whether to change the default behavior such that ./configure always generates a 64-bit build if you're on a 64-bit platform -- if you want a 32-bit build, you need to explicitly tell ./configure (i.e. --with-32).

Changing the default is probably only viable for 3.4 onwards.  It'd be nice if 2.7->3.3 had generic configure support for 64-bit builds though (via --with-64).

XXX TODO for trent: review autoconf's offerings... getting 64-bit builds from vendor cc's can't be that unusual.

On the s10 slave (Solaris 10/nitrogen) for 3.x:

(cpython@nitrogen:ttypts/4) (Tue/12:32) ..                                                                                                                                     
% ../../src/configure --with-pydebug --without-gcc CFLAGS="-m64 -mt -xcheck=%all -g3 -xarch=native -xchip=native" CPPFLAGS="-IInclude" OPT="" LDFLAGS="-m64 -mt -xcheck=%all -g3 -xarch=native -xchip=native" CC=/opt/solarisstudio12.3/bin/cc

Solaris 10 release (i.e. optimized) build requires the following:

../../src/configure --without-gcc CFLAGS="-v -fsimple=0 -m64 -mt=yes -xbuiltin -xhwcprof -xF -xarch=native -xchip=native -fma=fused -g -xO5 -xlibmil -xlibmopt -xmemalign=8s -xregs=frameptr -xtarget=native -xbuiltin=%all -library=sunperf" CPPFLAGS="-IInclude" OPT="" LDFLAGS="-v  -fsimple=0 -m64 -mt=yes -xbuiltin -xhwcprof -xF -xarch=native -xchip=native -fma=fused -g -xO5 -xlibmil -xlibmopt -xmemalign=8s -xbuiltin=%all -xregs=frameptr -xtarget=native -library=sunperf" CC=/opt/solarisstudio12.3/bin/cc 

(Due to indirect linker invocation via cc, I'm purposely duplicating CFLAGS <-> LDFLAGS for now.  I'll refine later.)

I'm glad I found this. Without the long incantation _decimal is
seriously miscompiled on SPARC/Solaris 10/Sun Studio 12.3. I was
using just ...

  ./configure --without-gcc CFLAGS=-m64 LDFLAGS=-m64

..., and that seems to trigger an optimizer bug that produces
different output for _mpd_div_words_r() when called with exactly
the same input at different locations.


The line that Trent gave appears to work.

Hello
On AIX 64bit (see http://bugs.python.org/issue6600) you should add in CFLAGS  -U__STR__ to bypass xlc(_r) define/inlining issue.

On Tue, Sep 18, 2012 at 06:40:35AM -0700, Trent Nelson wrote:
> Side bar design question: on BSD/Linux/OSX w/ gcc/clang, if you're on
> an amd64 platform, you'll get a 64-bit build.  This isn't the case for
> Solaris, HP-UX and AIX -- the default is always 32-bit.  You need
> aforementioned gymnastics to coerce a 64-bit build.
> 
> Is this by design?  If so, then I guess I'm proposing that ./configure
> should have a `--with-64`-type argument that'll generate a 64-bit
> build.
> 
> If not, then we need to decide whether to change the default behavior
> such that ./configure always generates a 64-bit build if you're on a
> 64-bit platform -- if you want a 32-bit build, you need to explicitly
> tell ./configure (i.e. --with-32).
> 
> Changing the default is probably only viable for 3.4 onwards.  It'd be
> nice if 2.7->3.3 had generic configure support for 64-bit builds
> though (via --with-64).
> 
> XXX TODO for trent: review autoconf's offerings... getting 64-bit
> builds from vendor cc's can't be that unusual.

    Just wanted to make note of a precedent I noticed yesterday whilst
    coercing local builds of Tcl, expect and a few other things on the
    Solaris 10 SPARC box: they all use `--enable-64bit` as the arg to
    configure to enable 64-bit builds.

    Additionally, none of them default to 64-bit out-of-the-box; they
    all inherit the underlying compilers "default" behavior.  Which
    means gcc/clang gets you 64-bit on amd64, but a cc build on a
    64-bit host may not.  (I need to check a few more permutations on
    this, like what happens with gcc on SPARC/IA64 etc if the host OS
    is 64-bit -- does it default to 64-bit?)

    Side note: some of those projects had *much* nicer autoconf-fu
    than we do.  Tcl implements the --enable-64bit stuff in aclocal.m4.

    So, I'm changing my stance from what I quoted above.  We should try
    mimic the precedent set by other projects:

        - Use --enable-64bit as the flag.
        - "Default" 64-bit builds depend on the underlying compiler.

I spent a little time on this yesterday.  Here's what I came up with.  The idea is to have a standalone block in configure.ac that kicks in when --without-gcc is specified (or if $CC != "gcc") *IFF* CFLAGS/CPPFLAGS haven't been provided by the user.

If they have, we warn, and skip the block.  Rationale is wanting to cover two cases: a) experienced user consciously overriding our attempts at sensible defaults in that block, b) inexperienced user who has no clue what the sensible defaults are but has otherwise inadvertently set CFLAGS/CPPFLAGS (or perhaps just inherited such settings from their existing environment).

So, here's what the "block" looks like currently, albeit just for IRIX at the moment (and this was against 2.7).  I'll eventually expand it to cover Solaris/SPARC|AMD64, AIX, HP-UX (PA-RISC/IA64) and Tru64.

(Also note the new `--enable-64bit` arg, which is intended to be used on these platforms as a way to coerce easy 64-bit builds.  The functionality doesn't apply on x86/x64 "free" *nix platforms that use clang/gcc -- those will default to using whatever memory architecture is being used by the underlying system -- if you want something else, that's the job of cross-compilation, which is a different problem this particular issue is attempting to address.)

+AC_MSG_CHECKING(for --enable-64bit)
+AC_ARG_ENABLE(
+    64bit,
+    AS_HELP_STRING(
+        [--enable-64bit],
+        [attempt 64-bit build (use with --without-gcc)]
+    )
+)
+if test -z "$enable_64bit"; then
+    AC_MSG_RESULT(no)
+else
+    AC_MSG_RESULT(yes)
+fi
+if test -n "$enable_64bit" && test "$without_gcc" != "yes"; then
+    AC_MSG_WARN([
+        --enable-64bit has no effect unless --without-gcc is also specified
+    ])
+fi
+
+# If we're not using GCC, we're probably on a proprietary UNIX, relying
+# on a proprietary compiler, running on non-x86/x64 hardware.  Autoconf
+# is pretty bad at picking sensible defaults for compiler flags in this
+# situation, so, let's try and do as much as we can ourselves.  The aim
+# is to pick the optimal settings for each of the following permutations:
+#
+# Debug builds, 32-bit and 64-bit:
+#   ./configure --without-gcc --with-pydebug
+#   ./configure --without-gcc --with-pydebug --enable-64bit
+#
+# Optimized release builds, 32-bit and 64-bit:
+#   ./configure --without-gcc
+#   ./configure --without-gcc --enable-64bit
+#
+if test "$without_gcc" = "yes" || test "$CC" != "gcc"; then
+    # This whole block assumes we know more than a) the user, and b) autoconf.
+    # If we detect CFLAGS/CPPFLAGS, then we can't safely assume a) anymore,
+    # so print a message instead of doing any customisation.
+    if test -n "$CFLAGS" || test -n "$CPPFLAGS"; then
+        AC_MSG_WARN([
+            You have defined CFLAGS and/or CPPFLAGS in conjunction with
+            --with-gcc; skipping attempt at setting flags to sensible
+            defaults -- make sure you know what you are doing.
+        ])
+    else
+        case $MACHDEP in
+            irix6)
+                # Trying to separate IRIX from C99 and still end up with a
+                # working executable is an exercise in futility; it has a
+                # much greater link between `cc -c99` and available system
+                # facilities/headers than other OSes (which primarily rely
+                # on the usual XPG4/XOPEN_SOURCE etc defines).  Using the
+                # c99 compiler is the lesser of two evils.
+                CC=c99
+                CXX=CC
+                # `-diag_error 1035` makes the compiler treat #error defines
+                # as actual errors (and abort compilation), and not warnings.
+                CFLAGS="-diag_error 1035"
+                # Try to target the underlying host hardware.
+                _platform=`hinv -c processor |
+                           head -n 1         |
+                           cut -f 4 -d ' '`
+                _processor=`hinv -c processor |
+                            head -n 2         |
+                            tail -n 1         |
+                            cut -f 3 -d ' '`
+                CFLAGS="$CFLAGS -TARG:platform=$_platform"
+                CFLAGS="$CFLAGS -TARG:processor=$_processor"
+
+                if test "$with_pydebug" = "yes"; then
+                    OPT="-O -g2"
+                else
+                    OPT="-g3 -Ofast=$_platform -OPT:Olimit=5500"
+                fi
+
+                if test -n "$enable_64bit"; then
+                    CFLAGS="-64 $CFLAGS"
+                    LDFLAGS="-64"
+                else
+                    CFLAGS="-n32 $CFLAGS"
+                    LDFLAGS="-n32"
+                fi
+                ;;
+            *)  ;;
+        esac
     fi
 fi

Without having reviewed the proposed change in detail, a couple of comments.  On current OS X systems and others, the compiler could be clang which perhaps should be treated as gcc for most autoconf purposes.  Also, why are we putting any effort into supporting IRIX anymore?  Both IRIX and the MIPS processor lines it runs on was retired by SGI in 2007.  That's why support for IRIX was pulled from Python 3.  It's a dead end.

On Thu, Dec 13, 2012 at 10:28:00PM -0800, Ned Deily wrote:
> 
> Ned Deily added the comment:
> 
> Without having reviewed the proposed change in detail, a couple of
> comments.  On current OS X systems and others, the compiler could be
> clang which perhaps should be treated as gcc for most autoconf
> purposes.

    I'm not intending to target OS X in the autoconf block I referred
    to; it's a popular platform for developers and users, and doesn't
    have any of the problems that the proprietary *NIXes/compilers
    have.

    This work is solely aimed at Solaris, Tru64, HP-UX, AIX and IRIX.

> Also, why are we putting any effort into supporting IRIX
> anymore?

    We?  It's just me :-)  So a more appropriate question might be why
    am I bothering to put effort into supporting it?  And the answer to
    that is simply because I can; Snakebite has an SGI Origin running
    the latest version of IRIX with the MIPSpro compiler suite, which
    is all the hardware I need to keep things chugging along.  (IRIX
    also has a huge active "fan" base of users that run a lot of OSS
    software -- nekoware et al.)

> Both IRIX and the MIPS processor lines it runs on was retired by SGI
> in 2007.  That's why support for IRIX was pulled from Python 3.

    Support was pulled for, to quote PEP 11, "IRIX 4 and --with-sgi-dl";

    PEP 11 also has a nice clause that basically says platform support
    is primarily based on having an active maintainer willing to keep
    everything in shape -- I'm happy to be marked as the maintainer for
    all the aforementioned *NIX platforms.

    Martin made a good point a few weeks ago when we discussed this on
    infrastructure@; his concern was the effort involved in supporting
    additional platforms could detract developers from more important
    tasks.

    I agree -- they're esoteric platforms at best, EOL at worst, and
    just because I'm maintaining them shouldn't mean other developers
    have to worry about breaking them.  They're not anywhere near as
    important as the big 3 (Linux/Windows/OSX).  If they do break,
    though, I'll keep fixing them as long as I'm actively maintaining
    support for that platform.

    The motivation behind this particular change is simple: it took
    about three days to nail down the exact flags to use on Solaris
    SPARC 32-bit to get a working ./python (with lots of referring
    to various Sun/Oracle compiler docs).  No-one else should have
    to go through this much effort -- ./configure should pick the
    right flags out of the box for the following permutations:

    32/64 bit debug builds:
        ./configure --without-gcc --with-pydebug
        ./configure --without-gcc --with-pydebug --enable-64bit

    32/64 bit release builds:
        ./configure --without-gcc
        ./configure --without-gcc --enable-64bit

    (And again, just to clarify, none of this work will remotely
     affect Linux, OS X or the *BSDs.  It doesn't take three days
     of reading compiler manuals to get working builds on those
     platforms.)

Trent, did the HPUX CFLAGS/LDFLAGS change on the buildbots?  I think
_decimal used to compile on the PA-RISC bot, but now there is an error.

Perhaps -AC99 or something like that is missing.