This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: _tkinter built on macOS 10.14 does not link to Tcl and Tk in /Library/Frameworks
Type: Stage: patch review
Components: macOS Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: anthonypjshaw, dimpase, miss-islington, ned.deily, ronaldoussoren, wordtech
Priority: high Keywords: patch

Created on 2018-10-11 03:29 by wordtech, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 20171 merged ned.deily, 2020-05-18 07:56
PR 20255 merged ned.deily, 2020-05-20 09:19
PR 20257 merged miss-islington, 2020-05-20 09:41
Messages (8)
msg327508 - (view) Author: Kevin Walzer (wordtech) * Date: 2018-10-11 03:29
I'm trying to build Python 3.7.0 on macOS 10.14, and Tkinter is not linking to my installation of Tcl/Tk 8.6.8 in /Library/Frameworks. Instead it is linking to the ancient 8.5 Tk installed in /System/Library/Frameworks. My usual way of forcing Python to link to my installation is to edit setup.py and comment out all search directories except /Library/Frameworks, but that seems to be ignored here. My basic invocation is "./configure --enable-framework" which, along with omitting the system libraries from setup.py, has always been sufficient in the past. Please advise.
msg327522 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-10-11 07:44
macOS 10.14 appears to have introduced some subtle differences in the search order for finding header files and/or frameworks and libraries.  I'm not 100% sure I understand completely what has changed but I'm confident that the workaround outlined below should get you going until we have a better solution (but beware of the potential gotcha).

Python has never fully supported building only from an Xcode-only installation, i.e. we have required at a minimum installing system header files into /usr/include et al by using "xcode-select --install".  As of 10.14, xcode-select no longer installs header files in /usr/include.  So that further cripples Python builds in that header files for several third-party libraries shipped with macOS are no longer found, like zlib and sqlite3.  When using an Xcode-only installation (no Command Line Tools) in previous releases of macOS, I believe it was the case that essentially the system root directory ('/') was searched by the compiler tool chain for header files, for libraries, and for frameworks, and for frameworks the long standard fallback framework search path order was honored by default: first search /Library/Frameworks, then /System/Library/Frameworks.  That worked fine if you were installing frameworks like Tcl and TK into /Library/Frameworks and wanting them to override the system ones.  If you did not install the Command Line Tools, then the tool chain used the MacOSX.sdk embedded in Xcode.app as the system root for searching.  By default, that's found in:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

With an unmodified Xcode install, that directory contains usr and System directories but no Library.  So the contents of /Library were ignored by the tool chain with the net effect that _tkinter builds would also link with the Apple-supplied Tcl and Tk 8.5 in /System/Library/Frameworks.

With 10.14, Xcode-only installs still work the same way but now Command Line Tools seem to effectively use the SDK embedded in /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk as the system root, rather than the system /.  Just like the Xcode copy of the sdk, the CLT copy does not have a Library directory so now the results are the same as Xcode-only builds: _tkinter only links with the /System/Library Tcl and Tk.  From the viewpoint of developers trying to build applications for distribution to others, it makes sense to try to ensure that /Library does not influence the build because /Library is under user control, not the system (Apple).

Until we "recently" started shipping our private version of Tcl and Tk 8.6.x, which we do not install to /Library btw, it made sense to default to linking to /Library and falling back to /System/Library, allowing optional use of a third-party Tcl/Tk (like from ActiveState) and it all sort of worked if you used the tool chain from the CLT.  But with 10.14, that no longer works.  Python's build system complicates this all in large part because the top-level setup.py tries to guess what directories the compiler tool chain are going to search; that was always kind of iffy and now it's much more broken with macOS 10.14.  There are ways to work around most of the issues but we need to have better default behavior.  That's kind of a bigger deal.

For the specific case of building _tkinter to link with Tcl and Tk frameworks in /Library, I think the easiest workaround is to add a Library symlink to the SDK in use, either under /Library/Developer/CommandTools or /Applications/Xcode.app.  "xcode-select --print-path" will show which one is in use, but, alas, not the full path to the SDK.

either:
$ xcode-select --print-path
/Library/Developer/CommandLineTools
$ cd /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/

or:
$ xcode-select --print-path
/Applications/Xcode.app/Contents/Developer
$ xcodebuild -version -sdk macosx Path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
$ cd $(xcodebuild -version -sdk macosx Path)

You can then carefully create a symlink from the SDK to /Library.

$ sudo ln -s /Library .

Now the Python builds should find the Current version of Tcl and Tk frameworks in /Library.

*But* be aware that this might affect other builds if there are other frameworks you have installed in /Library.  So this workaround should be used with caution and you might consider removing the symlink once you are done building Python.

There are other, more complicated workarounds but this seems to me to be the simplest and most foolproof as long as one is aware of the risks and is prepared to potentially recreate the Library symlink after software updates to Xcode or the Command Line Tools.
msg327524 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-10-11 07:51
Correction: in the original second paragraph, the fourth sentence obviously has the case backwards.  It should read:

"When using a *Command Line Tools installation (no Xcode)* in previous releases of macOS, I believe it was the case that essentially the system root directory ('/') was searched by the compiler tool chain for header files, for libraries, and for frameworks, and for frameworks the long standard fallback framework search path order was honored by default: first search /Library/Frameworks, then /System/Library/Frameworks. [...]"
msg327556 - (view) Author: Kevin Walzer (wordtech) * Date: 2018-10-12 01:57
Thank you, this helped.
msg337562 - (view) Author: Dmitrii Pasechnik (dimpase) * Date: 2019-03-09 08:27
On https://bugs.python.org/issue36231 I propose a way to build without creating any symlinks.
msg369195 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-05-18 08:33
New changeset 1731d6da263e9a2d6e783e87a8a5d5ce58fda46d by Ned Deily in branch 'master':
bpo-34956: Fix macOS _tkinter use of Tcl/Tk in /Library/Frameworks (GH-20171)
https://github.com/python/cpython/commit/1731d6da263e9a2d6e783e87a8a5d5ce58fda46d
msg369434 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-05-20 09:41
New changeset bac170cd93bbae939fcb29ccc6b5d423f7f4a089 by Ned Deily in branch 'master':
bpo-34956: edit and format better NEWS item in 3.9.0b1 changelog (GH-20255)
https://github.com/python/cpython/commit/bac170cd93bbae939fcb29ccc6b5d423f7f4a089
msg369435 - (view) Author: miss-islington (miss-islington) Date: 2020-05-20 09:47
New changeset e7bf8684e2f9d5f7e92284a0ea085a791f5173df by Miss Islington (bot) in branch '3.9':
bpo-34956: edit and format better NEWS item in 3.9.0b1 changelog (GH-20255)
https://github.com/python/cpython/commit/e7bf8684e2f9d5f7e92284a0ea085a791f5173df
History
Date User Action Args
2022-04-11 14:59:07adminsetgithub: 79137
2021-11-04 14:23:12erlendaaslandsetnosy: + ronaldoussoren, wordtech, ned.deily, anthonypjshaw, miss-islington, dimpase, - lys.nikolaou, pablogsal, ahmedsayeed1982

components: + macOS, - Parser
versions: + Python 3.6, Python 3.8
2021-11-04 14:21:37erlendaaslandsetmessages: - msg405708
2021-11-04 12:12:21ahmedsayeed1982setversions: - Python 2.7, Python 3.6, Python 3.8
nosy: + pablogsal, ahmedsayeed1982, lys.nikolaou, - ronaldoussoren, wordtech, ned.deily, anthonypjshaw, miss-islington, dimpase

messages: + msg405708

components: + Parser, - macOS
2020-05-20 09:47:11miss-islingtonsetmessages: + msg369435
2020-05-20 09:41:40miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19542
2020-05-20 09:41:40ned.deilysetmessages: + msg369434
2020-05-20 09:19:57ned.deilysetpull_requests: + pull_request19540
2020-05-18 14:04:21ned.deilysetpull_requests: - pull_request19480
2020-05-18 14:02:44anthonypjshawsetnosy: + anthonypjshaw
pull_requests: + pull_request19480
2020-05-18 13:59:47ned.deilylinkissue39162 superseder
2020-05-18 08:33:03ned.deilysetmessages: + msg369195
2020-05-18 07:56:05ned.deilysetkeywords: + patch
stage: patch review
pull_requests: + pull_request19470
2019-03-09 08:27:38dimpasesetnosy: + dimpase
messages: + msg337562
2018-10-12 01:57:38wordtechsetmessages: + msg327556
2018-10-11 07:51:45ned.deilysetmessages: + msg327524
2018-10-11 07:44:33ned.deilysetpriority: normal -> high
title: 3.7.0 _tkinter module links against /System/Library/Frameworks -> _tkinter built on macOS 10.14 does not link to Tcl and Tk in /Library/Frameworks
versions: + Python 2.7, Python 3.6, Python 3.8
messages: + msg327522

assignee: ned.deily
2018-10-11 03:29:08wordtechcreate