Title: Extension module builds fail on OS X with TypeError if Xcode command line tools not installed
Type: Stage: resolved
Components: Distutils, macOS Versions: Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: Alexey.Borzenkov, hynek, ned.deily, python-dev, ronaldoussoren, samueljohn, schmir, tarek
Priority: normal Keywords:

Created on 2013-05-27 14:07 by samueljohn, last changed 2013-07-31 07:29 by ned.deily. This issue is now closed.

Messages (7)
msg190140 - (view) Author: Samuel John (samueljohn) Date: 2013-05-27 14:07
In the `` module, there seems to be a bug in the method `compiler_fixup` which occurs if 

* the `customize_compiler` method from `distutils/sysconfig` has been called and after that
* `_compile` from `distutils/` is called. The `_compile` method uses `_osx_support.compiler_fixup`.

The critical line in compiler_fixup is:

`compiler_so = list(compiler_so)`.

Usually `compiler_so` is a list like `['cc']` but not when `customize_compiler` (from sysconfig) has been called. The value returned by `sysconfig.get_config_var('LDSHARED')` is a unicode like for example in the case of a Python 2.7.5 built via Homebrew: 
    "cc -bundle -undefined dynamic_lookup -L/homebrew/lib -L/homebrew/opt/sqlite/lib -isysroot /Applications/".
When `list()` is applied to the value, it is split at each character and that cannot be intended.

I am not sure if I fully understand, but to fix this in `compiler_fixup`, instead of `compiler_so = list(compiler_so)`, I'd propose something along the lines of:

    if isinstance(compiler_so, (str,unicode)):
        compiler_so = compiler_so.split()

I found this by trying to `pip install cython` (it uses customize_compiler, I guess) on a homebrewed python.

To reproduce in an interactive shell we can emulate what is going on by calling `compiler_fixup` directly with the values sysconfig has stored. The exact string does not matter, because I think the any value returned by `sysconfig_get_config_var` is a (unicode) string and not a list:

    import sysconfig
    import _osx_support

Not sure if other python versions are affected. I wasn't aware of `_osx_support` at all until now.
msg190142 - (view) Author: Samuel John (samueljohn) Date: 2013-05-27 14:42
The symptom for the end-user looks kind of weird:

    running build_ext
building 'Cython.Plex.Scanners' extension
/ A p p l i c a t i o n s / X c o d e . a p p / C o n t e n t s / D e v e l o p e r / T o o l c h a i n s / X c o d e D e f a u l t . x c t o o l c h a i n / u s r / b i n / c l a n g   - f n o - s t r i c t - a l i a s i n g   - f n o - c o m m o n   - d y n a m i c   - I / h o m e b r e w / i n c l u d e   - I / h o m e b r e w / o p t / s q l i t e / i n c l u d e   - i s y s r o o t   / A p p l i c a t i o n s / X c o d e . a p p / C o n t e n t s / D e v e l o p e r / P l a t f o r m s / M a c O S X . p l a t f o r m / D e v e l o p e r / S D K s / M a c O S X 1 0 . 8 . s d k   - I / A p p l i c a t i o n s / X c o d e . a p p / C o n t e n t s / D e v e l o p e r / P l a t f o r m s / M a c O S X . p l a t f o r m / D e v e l o p e r / S D K s / M a c O S X 1 0 . 8 . s d k / S y s t e m / L i b r a r y / F r a m e w o r k s / T k . f r a m e w o r k / V e r s i o n s / 8 . 5 / H e a d e r s   - D N D E B U G   - g   - f w r a p v   - O 3   - W a l l   - W s t r i c t - p r o t o t y p e s   -I/homebrew/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c Cython/Plex/Scanners.c -o build/temp.macosx-10.8-x86_64-2.7/Cython/Plex/Scanners.o
unable to execute /: Permission denied
error: command '/' failed with exit status 1

Pasted here to help googlers to find it :-)
msg190159 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-05-27 22:51
Sorry, I am unable to reproduce the problem with a vanilla Python 2.7.5 and a downloaded Cython 0.19.1 either installing directly with Cython's or using Pip 1.3.1.  customize_compiler is used as part of every extension module build in Distutils; it is not something special. AFAICT, the compiler_so attribute of a CCompiler instance should always be a list.  It is intended to be set up as such by the set_executables method of CCompiler (Lib/distutils/ which calls set_executable for each item, including compiler_so.

Now there could be a problem in Python 2 if, for some reason, the value for 'compiler_so' has become a Unicode string: set_executable is looking only for an instance of type str.  It looks like the original value for 'compiler_so' is constructed in customize_compiler (in lib/distutils/ by first concatenating CC + ' ' + CFLAGS, where the CC and CFLAGS values are obtained from the Python build (using get_config_vars) and possibly overridden by environment variables of the same names.  Then the value of the config var 'CCSHARED' is appended; it most likely is empty.  So I would check to see the types of distutils.sysconfig.get_config_var("CC") and ("CFLAGS") and ("CCSHARED") as well as os.environ["CC"] and ["CFLAGS"].  If any of them return a type unicode string, I think that could explain the behavior you see. The use of Unicode strings for these kinds of values are likely to lead to trouble, one way or another, in Python 2 for sure.  (Also, be aware that Cython extends Distutils with its own build_ext to allow building of Cython modules. I took a very quick look at it and did not see any obvious problems.)
msg193828 - (view) Author: Alexey Borzenkov (Alexey.Borzenkov) Date: 2013-07-28 14:33
I was unaware of this issue (which homebrew works around incorrectly), and because of this I created issue 18580. The problem is indeed with unicode strings which are coming from _read_output in _osx_support. This is probably because this code was merged from Python 3, which in Python 2 it shouldn't use decode('utf-8') on command output.
msg193829 - (view) Author: Alexey Borzenkov (Alexey.Borzenkov) Date: 2013-07-28 14:37
Also, to reproduce it you have to temporarily move away your /usr/bin/cc and /usr/bin/clang, which then triggers the affected code path.
msg193982 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-07-31 07:15
New changeset addd9210816b by Ned Deily in branch '2.7':
Issue #18071: Extension module builds on OS X could fail with TypeError
msg193985 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-07-31 07:29
Alexey's analysis is correct.  The problem is seen when the Xcode command line tools are not installed.  When building an extension module, Distutils is then unable to find a compiler at the normal path and consults the 'xcrun' utility to find the location of an uninstalled compiler.  The output from the 'xcrun' is mistakenly converted to Unicode and this triggers the odd behavior in customize_compiler.  The fix, applied for release in 2.7.6, is to avoid the Unicode conversion for Python 2.
Date User Action Args
2013-07-31 07:29:33ned.deilysetstatus: open -> closed

components: + Distutils
title: _osx_support compiler_fixup -> Extension module builds fail on OS X with TypeError if Xcode command line tools not installed
nosy: + tarek

messages: + msg193985
resolution: fixed
stage: resolved
2013-07-31 07:15:07python-devsetnosy: + python-dev
messages: + msg193982
2013-07-28 15:04:25ned.deilylinkissue18580 superseder
2013-07-28 14:51:09schmirsetnosy: + schmir
2013-07-28 14:37:19Alexey.Borzenkovsetmessages: + msg193829
2013-07-28 14:33:58Alexey.Borzenkovsetnosy: + Alexey.Borzenkov
messages: + msg193828
2013-05-27 22:51:06ned.deilysetassignee: ronaldoussoren -> ned.deily
messages: + msg190159
2013-05-27 14:42:05samueljohnsetmessages: + msg190142
2013-05-27 14:07:21samueljohncreate