classification
Title: Support builds on macOS without installed system header files
Type: compile error Stage: resolved
Components: Build, macOS Versions: Python 3.8, Python 3.7, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: dimpase, erik.bray, jdemeyer, ned.deily, ronaldoussoren
Priority: normal Keywords: patch

Created on 2019-03-08 10:03 by dimpase, last changed 2019-06-20 08:22 by dimpase. This issue is now closed.

Files
File name Uploaded Description Edit
noincludedirs_OSX.patch dimpase, 2019-03-08 10:03 POC patch for "no-headers" OSX
Pull Requests
URL Status Linked Edit
PR 12825 closed dimpase, 2019-04-14 10:07
PR 13773 merged ned.deily, 2019-06-03 10:16
PR 14208 merged ned.deily, 2019-06-18 19:27
PR 14256 merged ned.deily, 2019-06-20 05:47
Messages (13)
msg337461 - (view) Author: Dmitrii Pasechnik (dimpase) * Date: 2019-03-08 10:03
Neither Xcode nor its command-line tools on macOS 10.14 Mojave come with header files installed in /usr/ and other "normal" directories.

This is not documented in https://devguide.python.org/setup/#macos-and-os-x

While an extra step to handle this, i.e. to install the headers, is available (see a discussion on https://bugs.python.org/issue34956), Apple stated that this workaround will disappear.

It is thus highly desirable to provide a way to deal with headers located not at /usr/include, but at `xcrun --show-sdk-path`/usr/include.
A small change in setup.py along the lines of the following:

--- a/setup.py
+++ b/setup.py
@@ -134,7 +134,8 @@ def macosx_sdk_root():
     cflags = sysconfig.get_config_var('CFLAGS')
     m = re.search(r'-isysroot\s+(\S+)', cflags)
     if m is None:
-        sysroot = '/'
+        import subprocess
+        sysroot = subprocess.check_output(["xcrun", "--show-sdk-path"]).decode("utf-8").strip('\n')
     else:
         sysroot = m.group(1)
     return sysroot
@@ -146,6 +147,7 @@ def is_macosx_sdk_path(path):
     """
     return ( (path.startswith('/usr/') and not path.startswith('/usr/local'))
                 or path.startswith('/System/')
+                or path.startswith('/Applications/')
                 or path.startswith('/Library/') )

with the necessary changes to enable various modules (see attachment for a complete POC diff against the current master), the result builds and passes all the (enabled) tests on an OSX 10.13 system with empty /usr/include and /usr/local/include containing only LZMA headers.

Needless to say, a proper patch would not modify Modules/Setup, it'd adjust configure.ac etc.
msg337540 - (view) Author: Dmitrii Pasechnik (dimpase) * Date: 2019-03-08 22:10
Needless to say, subprocess is most certainly an overkill, something less involved would do the job, without the need for all the module dependencies of subprocess.
msg337680 - (view) Author: Erik Bray (erik.bray) * (Python triager) Date: 2019-03-11 15:49
Perhaps it would be better if the `xcrun --show-sdk-path` thing were run at configure-time and its result shoved into a variable we can read with sysconfig.get_config_var()
msg337906 - (view) Author: Dmitrii Pasechnik (dimpase) * Date: 2019-03-14 09:14
I won't mind to provide a PR for this. but it is not clear what the goal should be. Is it to build a working OSX Python with as few external to Xcode deps (it seems that only lzma is needed) as possible?
msg337912 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-03-14 13:12
Thanks for the report and for the PR offer but let's hold off on that for the moment: I'm planning to merge a somewhat different approach.
msg340202 - (view) Author: Dmitrii Pasechnik (dimpase) * Date: 2019-04-14 10:15
In case,I have opened PR https://github.com/python/cpython/pull/12825
to provide our solution to this issue.
msg344407 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-06-03 10:34
New changeset 0288dd6a5192074fcd5aa0db5d3513c3880209ca by Ned Deily in branch 'master':
bpo-36231:  Support building on macOS without /usr/include (GH-13773)
https://github.com/python/cpython/commit/0288dd6a5192074fcd5aa0db5d3513c3880209ca
msg346008 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-06-18 20:28
New changeset c7302116573d853d3181133477d9d0e4d4d3abfd by Ned Deily in branch '3.7':
bpo-36231:  Support building on macOS without /usr/include (GH-13773) (GH-14208)
https://github.com/python/cpython/commit/c7302116573d853d3181133477d9d0e4d4d3abfd
msg346049 - (view) Author: Dmitrii Pasechnik (dimpase) * Date: 2019-06-19 14:03
I find it puzzling that a relatively clean and short solution using autotools and the Apple-approved way to get the location of the headers is rejected, and a hacky, longer way that scrapes output of the compiler is being pushed instead.
msg346062 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-06-19 16:59
Thank you again for the suggested PR. Using "xcrun --show-sdk-path" at configure time *is* appealing.  Unfortunately, it does not cover all of the necessary use cases.

One, the --show-sdk-path option is not available on old versions of xcrun, versions we still build with to support older releases of macOS.  Second, some users build Python for macOS with compiler tool chains other than the Apple-supplied ones (for example, current gcc) that do not necessarily support the transparent selection of header and library files location via xcrun and friends.  Third, capturing the selected SDK path at configure time is no guarantee that the same SDK path will be used for extension module builds when setup.py runs.  With the Apple-supplied tools, the actual SDK path used is determined each time the compiler front-end is invoked and depends on the then-current selected values (e.g. the most recent value set by 'xcode-select --switch') and the current value of environment variables (e.g. like DEVELOPER_DIR and SDKROOT). Or a different or non-Apple compiler could now be in use by overriding CC.  In other words, lots of edge cases largely due to the inherent flexibility of Apple's compiler frontend.

The other issue here is the behavior of setup.py in trying to make reasonable default choices for finding header and library files for the extension modules it is building; to do so, it tries to guess what the compiler frontend is going to do and that's the real hack.  Life would be much simpler if Python relied on a modern autotools build setup - although some of the same issues of dynamic SDK locations would still apply but at least they would apply consistently - and not the legacy mixture of build tools we have today.  Unfortunately, it would be a big deal to replace the current build system and, while it would be desirable, that's a very big project.

One other point: while scraping the output of the compiler is hacky, it is a well-known and widely-used technique and is already in use elsewhere in setup.py; this code was adapted from that in add_multiarch_paths().
msg346101 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-06-20 05:59
New changeset c421c66a58a6caae30f0679d7e61411418e67cec by Ned Deily in branch '2.7':
bpo-36231: Support building on macOS without /usr/include (GH-13773) (GH-14256)
https://github.com/python/cpython/commit/c421c66a58a6caae30f0679d7e61411418e67cec
msg346104 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-06-20 06:46
Thanks again everyone.  The general solution has now been merged to all active branches for release in 3.8.0, 3.7.4, and 2.7.17.

Note that Issue19960 identifies an additional problem only on 2.7 where a few extension modules, notably zlib, are still not being built when header files are not installed; a PR for that issue is currently awaiting review.
msg346108 - (view) Author: Dmitrii Pasechnik (dimpase) * Date: 2019-06-20 08:22
Thanks for the review of our PR. I think not all points made there are relevant:

* on older systems, where `xcrun --show-sdk-path` is not available, one can assume that /usr/include is there, and simply set the corresponding variable to this value in ./configure

* building Python with a non-system compiler is only possible with system headers, and thus either SDK or at least command line tools, available, and all is well.

* mid-flight switching to a different SDK/compiler is dodgy, and in particular switching to a different set of headers is asking for trouble. And the gcc toolchain is not much different in this way - e.g. on Gentoo one can do gcc-config to switch from one profile to another, with provably "interesting" results if it is done mid-flight.  

* the Python 2.7 zlib issue is not a problem with our PR. 

---------

Most probably, the desired switch to autotools can only happen gradually, by moving more of setup.py to ./configure, and our PR is a step in such direction.
History
Date User Action Args
2019-10-19 09:57:29ned.deilylinkissue37285 superseder
2019-06-20 08:22:24dimpasesetmessages: + msg346108
2019-06-20 06:46:28ned.deilysetstatus: open -> closed
title: no "proper" header files on macOS 10.14 Mojave -> Support builds on macOS without installed system header files
messages: + msg346104

components: + Build
resolution: fixed
stage: patch review -> resolved
2019-06-20 05:59:59ned.deilysetmessages: + msg346101
2019-06-20 05:47:50ned.deilysetpull_requests: + pull_request14086
2019-06-19 17:00:11ned.deilysetversions: + Python 2.7, Python 3.7
2019-06-19 16:59:53ned.deilysetmessages: + msg346062
2019-06-19 14:14:40jdemeyersetnosy: + jdemeyer
2019-06-19 14:03:28dimpasesetmessages: + msg346049
2019-06-18 20:28:35ned.deilysetmessages: + msg346008
2019-06-18 19:27:22ned.deilysetpull_requests: + pull_request14046
2019-06-03 10:34:51ned.deilysetmessages: + msg344407
2019-06-03 10:16:07ned.deilysetpull_requests: + pull_request13658
2019-04-14 10:15:16dimpasesetmessages: + msg340202
2019-04-14 10:07:17dimpasesetstage: patch review
pull_requests: + pull_request12750
2019-03-14 13:12:07ned.deilysetassignee: ned.deily
messages: + msg337912
2019-03-14 09:14:06dimpasesetmessages: + msg337906
2019-03-11 15:49:28erik.braysetnosy: + erik.bray
messages: + msg337680
2019-03-08 22:10:03dimpasesetmessages: + msg337540
2019-03-08 10:03:47dimpasecreate