Trying to build latest development revision:

localhost:repo family$ hg summary
parent: 73044:a985d733b3a3 tip
 #12753: Add support for Unicode name aliases and named sequences.
branch: default
commit: (clean)
update: (current)

Constantly getting this error:

./python.exe -SE -m sysconfig --generate-posix-vars
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Assertion failed: (compact->utf8_length == 0), function _PyUnicode_CheckConsistency, file Objects/unicodeobject.c, line 399.
make: *** [sysconfig] Abort trap

However, tag v3.2.2 seems to build successfully.

OS: Mac OS X 10.6.8
GCC: 4.2.1

If you have troubles reproducing this issue, feel free to ask any information I can provide that can help.

Thank you in advance!

Oh yes.
I am building with this command line:
./configure --with-pydebug && make -j2

It is a compiler bug in GCC 4.2.1 with -O3. Try to compile attached unicode.c program with gcc -O3. The correct result is:

$ gcc -O3 unicode.c -o unicode && ./unicode
is ascii? 0
is compact? 1
is compact ascii? 0
_PyUnicode_COMPACT_DATA: 88 vs 88
PyUnicode_DATA: 88 vs 88
explicit cast: 88 vs 88

With gcc 4.2.1 and -O3, you get "is compact ascii? 1".

The bug has been fixed in GCC, try a more recent version using ./configure CC=gccX.Y.

It should also work without --with-pydebug, or with a different compiler.

I believe this was a bug specific to the llvm-gcc compiler shipped with Xcode 4.1 and possibly Xcode 4.0.  I do not have Xcode 4 on OS X 10.6 but haypo's test does fail with the llvm-gcc 4.2.1 that comes with Xcode 4.1 on 10.7.  Can you confirm exactly which compiler you are using, i.e. the complete --version output from the compiler used in your python build?  If you have not installed Xcode 4.2, one solution is to use CC=gcc-4.2 which uses the plain gcc-4.2, not llvm-gcc-4.2.  For 10.6, I recommend sticking with Xcode 3 if at all possible for the time being.

See also http://mail.python.org/pipermail/python-dev/2011-September/113731.html

Thank you all for help. I finally managed to build it. :-)
So here are complete steps I've taken to build python after OS reinstall:
1. Install XCode 4.2
2. Install MacPorts
3. sudo port -v selfupdate
4. sudo port install gcc46
5. ./configure CC=gcc-mp-4.6 LDFLAGS=-L/opt/local/lib --with-pydebug && make -j2

Should I post this somewhere for others who may face similar problems?

In general, I would not recommend using anything other than an Apple-supplied tool chain for building anything on Mac OS X.  To help us better understand the issues, can you say exactly which compilers you tried originally, i.e. the complete --version output from the compiler(s) used in your failing Python builds?

Because of the current somewhat chaotic OS X compiler situation, I am in the process of performing a controlled test of building and testing various current versions of Python with the available Xcode compilers for 10.7 and 10.6.

Ah, of course.
Initially I've tried to compile using this:

i686-apple-darwin10-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)

XCode version 4.2, build 4C199

> i686-apple-darwin10-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)

Oh, that's not GCC but llvm-gcc. It is maybe a bug in LLVM?

Oleg, thanks. That is consistent with previous reports; so far, this issue has only been reported with llvm-gcc, not the plain gcc-4.2's shipped in Xcode prior to 4.2.

I asked for more information about llvm-gcc on IRC (#llvm). Extracts:

 - "llvm-gcc is not maintained." ("Clang and DragonEgg are.) "Apple is the maintainer of Apple's version of llvm-gcc and clang."
 - "Apple ships a [llvm-gcc] branch at a random point in svn + some bug fixes, not a proper release."
 - "llvm-gcc is Apple's GCC (which was never sync'd with mainline past 4.2.1 because the FSF changed the license) with LLVM on the back end."
 - llvm-gcc-4.5 in Ubuntu 10.10 is not llvm-gcc, it's dragonegg, a new project

If I understood correctly, llvm-gcc is only the default C compiler of Xcode 4.1. I don't think that you should workaround llvm-gcc bugs. If you are using Xcode 4.1, you can compile Python with clang. clang is a different C compiler and it doesn't have the bug.

Haypo, the OP has a workaround. IMO, we should not be making any recommendations about which Apple compiler to use on OS X 10.7 until we have completed test cycles of each.  There are too many variables at the moment.  I am doing that and should have results in a few days.

I was unsuccessful building tip with any of the 3 compilers. Full paste of output from ./configure and make, along with haypo's test file are attached.

http://paste.pound-python.org/show/14320/

I'm on 10.7.2, XCode is 4.1 (compiler versions in the paste).

Upgrading to XCode 4.2 is undesirable, since it removes vanilla gcc. Myself and a few others are avoiding upgrading until homebrew adds vanilla gcc and / or Apple gets things fully sorted, so, being able to build with this would be nice.

Julian, I am able to build and run the current default (3.3) tip (52b35dae22d1) on 10.7.2 with the gcc-4.2 from Xcode 4.1, with or without debug enabled.  Are you sure you have pulled all the latest updates and are using a clean build directory?  If so, please open a new issue showing the details and the exact ./configure command used.

Never mind, yes, you're correct, somehow my build dir must have been dirty.

Sorry bout that.

I'm on 10.7.2, with XCode is 4.2 and the problem is still present. The command "ggc -v" produces the following output:

gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)

The command "./configure --with-pydebug && make -j2" fails with the following output:

Assertion failed: (compact->utf8_length == 0), function _PyUnicode_CheckConsistency, file Objects/unicodeobject.c, line 381.
make: *** [sysconfig] Abort trap: 6

The only solution seems to be to install macports and use it to get a more recent gcc like Oleg Plakhotnyuk suggested.

Lucas, exactly what source version of Python are you trying to build (i.e what does "hg summary" say)?

I just cloned cpython today. The output of "hg summary" is:

parent: 73351:2bec7c452b39 tip
 Fix C89 incompatibility.
branch: default
commit: (clean)
update: (current)

On OS X Lion, with XCode 4.2 installed, I find the following works (no need to install macports):

./configure CC=gcc-4.2 --prefix=/dev/null --with-pydebug

Ah, it seems I have XCode 3.2.6 installed alongside XCode 4.2.

Well the configure parameters suggested by Michael Foord worked. I was able to build cpython on a mac. But I could not have guessed how to make it work.

Shouldn't these instructions be added here http://docs.python.org/devguide/ ?

Shouldn't the latest version of OS X be included in the list of *nix systems that cpython is checked to build on ?

Lucas, there is nothing particularly special about the configuration parameters that Michael used other than enabling debug.  Prior to Xcode 4, Xcode supplied a "standard" (non-LLVM) gcc as the default compiler, 4.2 for 10.6 and 4.0 for 10.5 and 10.4.  Python is regularly built and supported on these configurations and is known to work using a default ./configure (with no parameters).  Xcode 4.2 is problematic because it no longer includes the non-LLVM gcc and there have been various rumors of problems and at least one patch applied already.  As noted above, I am testing the various significant configure and compiler options right now for Xcode 4.2 and will update this issue with recommendations and patches (if necessary) when that is complete.

Note that this works for me on a Macbook Air that has never had Snow Leopard, nor XCode 3 installed. 

As far as I can tell non-llvm gcc *is* installed by XCode 4.2: /usr/bin/gcc-4.2

The bug in the 'gcc' command is still present when using Xcode 4.2.1 (that is, the attached unicode.c miscompiles with "gcc -O3").

Clang (again from Xcode 4.2.1) compiles the file correctly, but fails to do a proper build the last few lines of the build log:

clang -bundle -undefined dynamic_lookup build/temp.macosx-10.4-x86_64-3.3/Users/ronald/Projects/python/rw/cpython/Modules/_cursesmodule.o -L/usr/local/lib -lncurses -o build/lib.macosx-10.4-x86_64-3.3/_curses.so
/bin/sh: line 1: 84213 Segmentation fault: 11  CC='clang' LDSHARED='clang -bundle -undefined dynamic_lookup  ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes' ./python.exe -E ./setup.py build
make: *** [sharedmods] Error 139

The last bit of a gdb stack trace:

#0  0x00007fff86cd94f0 in strlen ()
#1  0x000000010005bee1 in PyUnicode_FromString (u=0x4c <Address 0x4c out of bounds>) at unicodeobject.c:1735
#2  0x000000010003bb65 in PyDict_GetItemString (v=0x10075cc58, key=<value temporarily unavailable, due to optimizations>) at dictobject.c:2187
#3  0x000000010004eaf8 in add_getset [inlined] () at /Users/ronald/Projects/python/rw/cpython/Objects/typeobject.c:3771
#4  0x000000010004eaf8 in PyType_Ready (type=0x10159efb8) at typeobject.c:4109
#5  0x000000010159209a in PyInit__curses () at /Users/ronald/Projects/python/rw/cpython/Modules/_cursesmodule.c:3293
#6  0x00000001000c0e20 in _PyImport_LoadDynamicModule (name=0x1010f9f80, path=0x10074c750, fp=<value temporarily unavailable, due to optimizations>) at importdl.c:85
#7  0x00000001000beca0 in imp_load_dynamic (self=<value temporarily unavailable, due to optimizations>, args=<value temporarily unavailable, due to optimizations>) at import.c:3804
#8  0x000000010009f0c8 in call_function [inlined] () at /Users/ronald/Projects/python/rw/cpython/Python/ceval.c:4003

For some reason a clean rebuild got a failure in another location, this time in the Py_XDECREF call on line 2966 in ceval.c:

    while (!EMPTY()) {
        v = POP();
        Py_XDECREF(v);
    }

Due to the optimization level I couldn't get more information.

When I patch the Makefile to use '-O2' instead of '-O3' in CFLAGS the problems stays (obviously with full rebuild).

Patching the optimization level down to '-O1' gives a slightly better result, this time I get:

./python.exe -SE -m sysconfig --generate-posix-vars
Fatal Python error: Py_Initialize: unable to load the file system codec
SystemError: NULL result without error in PyObject_Call

BTW. I tried to build with srcdir==builddir and "./configure CC=clang" on an up-to-date OSX 10.7 system with Xcode 4.2.1.


Summary of compiler results:

- gcc-4.2 (i686-apple-darwin11-gcc-4.2.1):   unicode.c OK, build OK
- gcc (i686-apple-darwin11-llvm-gcc-4.2):    unicode.c fails, build fails
- clang: unicode.c OK, build fails

Here's a minimal test case for #define bug in LLVM GCC.

If the base struct is 8-byte long or smaller, the code runs correctly.

gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)

========
#include <stdio.h>
#include <stdlib.h>

typedef struct {
    int padding;  /* remove this and the sizes will be the same */
    int type;
    int attr;
} struct_a;

typedef struct {
    struct_a base;
    int junk;
} struct_b;

#define BUG(op) \
    ((((struct_a*)(op))->type) ? (void*)((struct_a*)(op) + 1) : \
                                 (void*)((struct_b*)(op) + 1))

static void* func(void* op)
{
    if (((struct_a*)(op))->type)
    {
        return (void*)((struct_a*)(op) + 1);
    }
    return (void*)((struct_b*)(op) + 1);
}

int main(int argc, char** argv)
{
    struct_b* b = malloc(sizeof(struct_b) + 2);
    struct_a* a = (struct_a*)b;
    char* p;
    a->type = 0;
    p = BUG(b);
    printf("expected: %d, actual: %d, b = %p, p = %p\n",
            sizeof(struct_b), p - (char*)b, b, p);
    p = func(b);
    printf("expected: %d, actual: %d, b = %p, p = %p\n",
            sizeof(struct_b), p - (char*)b, b, p);
    return 0;
}
========

$ ./bug
expected: 16, actual: 12, b = 0x10d7008b0, p = 0x10d7008bc
expected: 16, actual: 16, b = 0x10d7008b0, p = 0x10d7008c0

On my system (Snow Leopard OSX 10.6.8, Xcode 4.1) there are several different versions of gcc available in /usr/bin/. Some are symlinks to others; in particular gcc links to llvm-gccc-4.2. By using "./configure CC=gcc-4.2" I get a working config file. I ran "make clean" to avoid reusing possibly corrupt .o files compiled with the default.

That works for Xcode 4.1, which is now out-of-date. The current releases of Xcode, 4.2.x for 10.7 Lion and optionally for 10.6 Snow Leopard, no longer include gcc-4.2, only llvm-gcc and clang.  I am working on completing an extensive set of regression tests for all of the development branches versus the most important set of Xcode/platform options and should have results here later this week.

The version of clang included in Xcode 4.2.1 (the current release) still miscompiles Python 3.3 and both C files included in this issue.

The same is true for the Xcode 4.3 beta.

Python 2.7 seems to work fine with the same compiler, either because we don't trigger the bug or it fails in a more subtle way than crashing.

Has anyone filed a bug with Apple and/or the clang developers?

I've filed an issue in Apple's tracker, #10667893. Filing more issues would be helpful, if more issues in their tracker mention this bug it is more likely to get fixed.

I've found a bug in the curses module that is causing problems:


diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -2056,7 +2056,8 @@
     {"encoding",
      (getter)PyCursesWindow_get_encoding,
      (setter)PyCursesWindow_set_encoding,
-     "the typecode character used to create the array"}
+     "the typecode character used to create the array"},
+    {NULL, NULL, NULL, NULL}
 };
 
 /* -------------------------------------------------------*/

That is, there is no sentinel in the PyCursesWindow_getsets list. Shouldn't this cause failures on more platforms than just OSX?

With this patch I can build python 3.3 with clang (as included in Xcode 4.2.1). 

BTW. clang as included with Xcode 4.2.1 seems to build the attached copy of unicode.c without problems, the gcc command still fails though.

I did run into problems when building outside of the source directory, but that seems to be an unrelated issue.

Ronald's patch does fix the segmentation fault with clang-compiled Python. But this bug report seems to be about LLVM-GCC not compiling correctly.

Both clang and llvm-gcc had problems, the llvm-gcc problem is still there and is caused by a compiler bug. There is nothing we can do about that beyond warning Apple.

It's also pretty clear that llvm-gcc (and gcc in general) is on the way out as the system compiler on OSX. It's therefore unlikely that the compiler bug we ran into will get fixed.

I'm leaning towards adding code to configure that either tests for this issue and refers to documentation in Mac/README (also to be added) when this bug is found, or just switches to clang instead of gcc as the default compiler when a recent version of Xcode is detected (and 'CC' is not set in the environment).

While the full regression testing of all active branches against all of the current compiler versions is in progress, my current (and preliminary) recommendations (despite some test failures):

- for OS X 10.7.x, use the latest released Xcode, currently Xcode 4.2.1, 
and build with clang and debug:

./configure --with-pydebug CC=clang MACOSX_DEPLOYMENT_TARGET=10.7

- for OS X 10.6.x, if possible, continue to use the last released Xcode 
3.2 (3.2.6), which includes Apple gcc-4.2 (/usr/bin/gcc-4.2 not 
llvm-gcc-4.2)

/usr/bin/gcc-4.2 --version
./configure MACOSX_DEPLOYMENT_TARGET=10.6
or
./configure --with-pydebug MACOSX_DEPLOYMENT_TARGET=10.6

- for OS X 10.6.x with Xcode 4 installed (which does not include Apple 
gcc-4.2), use the latest Xcode 4.2 for 10.6 and use clang and debug:

./configure --with-pydebug CC=clang MACOSX_DEPLOYMENT_TARGET=10.6


More results and fixes to follow ...

BTW, the current test failures with clang without pydebug include a 
number of ctypes test failures (in ctypes.test.test_cfuncs.CFunctions).  
If anyone has time to further investigate those, it would be very 
helpful (Issue13370).

Note that apple have made gcc available, as part of command line tools for XCode, freely (free developer registration required though) from:

https://developer.apple.com/downloads/

See: 

http://kennethreitz.com/xcode-gcc-and-homebrew.html

>Note that apple have made gcc available, as part of command line tools >for XCode, freely (free developer registration required though) from:
> https://developer.apple.com/downloads
>See: 
>http://kennethreitz.com/xcode-gcc-and-homebrew.html

No. The new version of Xcode (Xcode 4.3, released last week) referred to in that post does *not* make gcc-4.2 available again.  Like Xcode 4.2, it only includes clang and llvm-gcc; 4.3 includes updated versions of each of the two. gcc-4.2 was removed starting with 4.2 and there is no indication that Apple intends to re-instate it. Besides the compiler updates, Xoode 4.3 has been significantly repackaged, including the ability to download and install its components in smaller sub-packages, instead of the monolithic large downloads of recent past releases.  However, Xcode 4.3 has moved the install location of components like the SDKs (which are used by Python universal builds) from /Developer to within the new Xcode application bundle (/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer); this change will require additional changes to the Python build and Distutils/packaging infrastructure to support properly.  For now, the easiest workaround when installing Xcode 4.3 and removing previous versions is to create a symlink from the app bundle Developer directory above to /Developer.  I will be testing the new compiler versions shortly and working on more general support for 4.3.

Hi Ned,

On a brand new OS X Lion install with the latest XCode (4.3.2) and command line tools*, the following worked:

./configure --with-pydebug CC=clang MACOSX_DEPLOYMENT_TARGET=10.7

That is, everything built cleanly, and all tests ran without failures:
336 tests OK.
2 tests altered the execution environment:
    test_packaging test_site
26 tests skipped:
    test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp
    test_codecmaps_kr test_codecmaps_tw test_curses test_dbm_gnu
    test_devpoll test_epoll test_gdb test_largefile test_lzma
    test_msilib test_ossaudiodev test_smtpnet test_socketserver
    test_startfile test_timeout test_tk test_ttk_guionly
    test_urllib2net test_urllibnet test_winreg test_winsound
    test_xmlrpc_net test_zipfile64
3 skips unexpected on darwin:
    test_lzma test_tk test_ttk_guionly

I presume the aim is to be able to build correctly (on all UNIX platforms) with just ./configure --with-pydebug, right?

How's your configure*-toolchain-fu?

Once we fix this I can add the build slave.

Side bar: note that this was a vanilla 10.6->10.7 install.  The XCode 4.3.2 install on 10.7 was the first XCode ever installed on the box; there's no /Developer or any of the old gcc* cruft.

[*]: Went to https://developer.apple.com/downloads/index.action, logged in with my Apple ID, then downloaded 'Commandline Tools for XCode - Late March 2012' -- dmg was 'cltools_lion_latemarch12.dmg'.

>./configure --with-pydebug CC=clang MACOSX_DEPLOYMENT_TARGET=10.7

Works on my machine too.

Trent, yes, now that the Xcode 4 situation has settled down a bit, clang is the compiler of choice for OS X 10.7 with Xcode 4.3 although there are still some open questions.  I intend to update configure in the near future for all active branches to provide more appropriate defaults for 10.6 and 10.7 and for universal builds with the various SDK locations.

>"Once we fix this I can add the build slave"

I'm missing the context for this.  Can't you override with environment variables or on the configure command itself?

> > "Once we fix this I can add the build slave"

> I'm missing the context for this.

Yeah I uh, ....seemed to have deleted the introductory sentence I wrote that said I was doing some prep work before adding an OS X 10.7 build slave.

> Can't you override with environment variables or on the configure command itself?

Are you referring to when I set up the build slave?  i.e. tweak the local environment the build slave account runs under so that `./configure --with-pydebug` works out of the box?

I don't think that's really a precedent we want to set (having build slave owners hack the local environment to work around build issues).

FWIW, clang from Xcode 4.3.2 build 4E2002 w/ command line tools built everything fine for me too (i.e., ./configure CC=clang).

LM-SJN-00377886:cpython tom$ uname -a
Darwin LM-SJN-00377886 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64

LM-SJN-00377886:cpython tom$ /usr/bin/clang --version
Apple clang version 3.1 (tags/Apple/clang-318.0.58) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin11.3.0
Thread model: posix

LM-SJN-00377886:cpython tom$ hg summary
parent: 76839:bb30116024ac tip
 Minor fix for test_multiprocessing
branch: default
commit: (clean)
update: (current)

Ned, any progress/problems?

I've closed two dupes already in the last few days and we're still in the alphas. We really should get this fixed before beta1 or the tracker will be flooded.

Let me know if you need something. :)

Hynek: Thanks for the offer; I've got it.  I'll have a comprehensive update to support the new multiple dev environments for 10.6 and beyond prior to b1 (and for 2.7.x and 3.2.x).

The attached configure.ac is a fragment of a configure script that tries to detect if AC_PROG_CC would pick llvm-gcc when clang is also available and then explicitly picks clang.

This would avoid "python3.3 won't build on OSX 10.7" bug reports.

This is not yet a patch to the real configure file because I'd like to add support for automatically picking the compiler inside Xcode.app when the user has Xcode 4 installed without the unix tools (e.g. on an new OSX 10.7 where the user has installed Xcode from the appstore without further actions).

The attached patch "issue13241-configure-version1.txt" implements a number of changes to configure (although only configure.ac is present in the patch)

1) Autodetect the compiler on OSX, in particular
   - detect llvm-gcc and use clang instead
   - use clang when gcc cannot be found
   - use xcrun to locate clang when it isn't in /usr/bin
     (no Command Line Tools for Xcode)

2) Detect the most recent working SDK, falls back to the 10.4u SDK
   when using an old version of Xcode or even '/' when that SDK isn't
   present


This patch needs work though:

* I have barely tested the patch, configure seems to generate a valid
  Makefile on OSX 10.7 with the unix tools installed, I haven't tested
  yet on older OSX versions of without the unix tools installed

* Distutils/packaging also need changes: need to detect that the 
  running system has different compiler tools than the build system.
  This is needed to be able to use the binary installers on python.org
  on systems with other compiler versions and installed SDKs

The configure script for python 3.3 detects llvm-gcc and automatically selects clang when it is found. 

That seems to be the best feasible solution for this issue. I therefore propose to close this issue.

I agree to close the issue. I tried following the steps proposed here:

http://docs.python.org/devguide/

And typed these two commands on a OS X 10.8.1 with XCode and command line tools installed.

$ hg clone http://hg.python.org/cpython.
$ run ./configure --with-pydebug && make -j2

Python built perfectly.

I've closed the issue as 'wont fix' because llvm-gcc will miscompile 3.3 due to bugs in that compiler. CPython won't work around that compiler bug (if that's even possible).

The workaround: use clang to compile (and on older OSX releases use a different version of GCC that doesn't use the LLVM backend).