classification
Title: Investigate providing Tcl/Tk 8.6 with OS X installers
Type: enhancement Stage:
Components: Build, Installation, Macintosh Versions: Python 3.6
process
Status: open Resolution:
Dependencies: 1584 Superseder:
Assigned To: ned.deily Nosy List: Aivar.Annamaa, benjamin.peterson, cjrh, esc24, georg.brandl, kbk, markroseman, ned.deily, python-dev, rhettinger, ronaldoussoren, samueljohn, terry.reedy
Priority: normal Keywords:

Created on 2012-08-15 09:20 by ned.deily, last changed 2016-09-02 01:03 by cjrh.

Messages (28)
msg168269 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-15 09:20
The current situation with Tcl/Tk on OS X is less than ideal.  The Apple-supplied versions of the Cocoa-based Aqua Tk 8.5 on 10.6 was unusable with IDLE and most Tkinter applications.  The 10.7 and 10.8 versions are better but still have serious crashers that have been more recently fixed.  python.org installers currently also support using ActiveTcl release but that is also not ideal as there are license restrictions for some users and a separate download and install is required.  We should be able to provide a better out-of-the-box experience if python.org installers included its own copies of up-to-date Tcl/Tk libraries (as is the case with the Windows installers).  It would be good to still give users the option to use ActiveTcl for any of its additional features or to use system Python.  Also we could also look at providing Tcl/Tk 8.5 for 10.5 users (possibly only Intel ones) which we currently do not.
msg168270 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-08-15 09:25
A major reason for why the current installers don't install Tk as well is that this would significantly increase the size of the installer.

There's also the question of what bits of Tcl/Tk should be bundled, although I guess we should ship a minimal install of Tk that still supports everything that is wrapped by the python std lib. Alternatively we could ship the same subset as the Windows installer.
msg168271 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-15 09:37
Yes, it would have to be a subset since some of the stuff in the ActiveTcl releases is unavailable anyway. But the current ActiveTcl installer dmgs are around 24Mb while the 3.3.0 installers are a little less than 20Mb.  Even if it were to double the size, 40Mb still seems pretty small by today's standards.  And following the Windows installer lead would be a good starting point.
msg178619 - (view) Author: Samuel John (samueljohn) Date: 2012-12-30 21:59
Sounds reasonable, but please put the libs not in a place that might conflict with a Tk installed by other means (e.g. homebrew's Tk is a `/usr/local`).
msg178646 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-12-31 07:29
The logical location for installing and embedded Tcl/Tk is either in {sys.prefix}/lib (for a unix install of Tk) or in Python.framework/Versions/X.Y/Frameworks (for a Tcl/Tk framework install).

In either case Tcl/Tk would be installed *inside* Python.framework and hence couldn't conflict with other installations of Tcl/Tk (system, activate-state, homebrew, ...).

Note that this would also make it harder for end users to upgrade Tcl/Tk, currently they can use the ActiveState installer to install a better Tk release and that would no longer be necessary.

That would be a small cost relative to the large gain of having a working Tk release.
msg197055 - (view) Author: Roundup Robot (python-dev) Date: 2013-09-06 08:22
New changeset 985384cd6365 by Ned Deily in branch 'default':
Issue #15663: Tcl/Tk 8.5.14 is now included with the OS X 10.6+
http://hg.python.org/cpython/rev/985384cd6365
msg197064 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-09-06 12:51
Great work. Thanks.
msg200365 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-19 04:36
New changeset 7f69f4fadfd8 by Ned Deily in branch 'default':
Issue #15663: Update OS X installer to use Tcl/Tk 8.5.15.
http://hg.python.org/cpython/rev/7f69f4fadfd8
msg200501 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-10-19 21:20
I guess the one remaining question about this issue is whether it should be backported to 2.7.x and/or 3.3.x. The implementation depends on the build changes implemented in Issue1584 to provide options to override default search paths for Tcl and Tk so that would have to be back ported as well.  IMO, the entire set of changes are fairly well isolated and should be low risk: basically, they are all build-related changes and mostly isolated to OS X.  The primary argument for backporting is that Tk support on current releases of OS X (from 10.6 Snow Leopard on) has been a major pain for our users, such that we have to keep recommending that they install a newer third-party Tcl/Tk to use IDLE or anything else Tkinter-based.  So that goes against the batteries-included mission of the python.org installer, especially since the third-party version (ActiveTcl) is not totally free and open source (although it is free as in beer for most uses).  On the other hand, one can argue that this is a new feature and thus prohibited in maintenance releases.  On the other other hand, one can argue that this is a change driven by changes (or lack thereof) in the underlying OS platform and *essentially* only changes the building of the python.org OS X installer, so it is similar in concept to any other third-party distributor of Python changing an underlying system library.  I would be OK with backporting or not but the decision needs to be made by the release managers.  Benjamin?  Georg?
msg200504 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-10-19 21:47
We've upgraded the versions of other embedded libraries in 2.7 bugfix releases, so I'm okay with this for 2.7.
msg200566 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-10-20 11:09
Yep (should there be a policy about this somewhere)?
msg200693 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2013-10-21 03:59
We should advertise in the docs that we have a policy of upgrading the Tcl/Tk versions as necessary to improve stability.
msg201219 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-25 07:53
New changeset 19b512bfc8d2 by Ned Deily in branch '2.7':
Issue #15663: Tcl/Tk 8.5.15 is now included with the OS X 10.6+
http://hg.python.org/cpython/rev/19b512bfc8d2

New changeset 5ff4f0100a40 by Ned Deily in branch '3.3':
Issue #15663: Tcl/Tk 8.5.15 is now included with the OS X 10.6+
http://hg.python.org/cpython/rev/5ff4f0100a40
msg201332 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-26 10:19
New changeset db5a50959dc9 by Ned Deily in branch '2.7':
Issue #15663: OS X installer builtin Tcl/Tk support
http://hg.python.org/cpython/rev/db5a50959dc9

New changeset 150242784de0 by Ned Deily in branch '3.3':
Issue #15663: OS X installer builtin Tcl/Tk support
http://hg.python.org/cpython/rev/150242784de0

New changeset b599584ff1e3 by Ned Deily in branch 'default':
Issue #15663: OS X installer builtin Tcl/Tk support
http://hg.python.org/cpython/rev/b599584ff1e3
msg201413 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-27 05:23
New changeset 0bf4cf62f14c by Ned Deily in branch '3.3':
Issue #15663: Force IDLE.app to run the GUI process in 32-bit mode.
http://hg.python.org/cpython/rev/0bf4cf62f14c

New changeset b6a1a78818fe by Ned Deily in branch 'default':
Issue Issue #15663: merge from 3.3
http://hg.python.org/cpython/rev/b6a1a78818fe
msg202203 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-05 10:51
New changeset fc8f19b4b662 by Ned Deily in branch '2.7':
Issue #15663: Revert OS X installer built-in Tcl/Tk support for 2.7.6.
http://hg.python.org/cpython/rev/fc8f19b4b662

New changeset 268dc81c2527 by Ned Deily in branch '3.3':
Issue #15663: Revert OS X installer built-in Tcl/Tk support for 3.3.3.
http://hg.python.org/cpython/rev/268dc81c2527

New changeset f89beccd470c by Ned Deily in branch 'default':
Issue #15663: merge build-installer.py changes
http://hg.python.org/cpython/rev/f89beccd470c
msg202204 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-11-05 10:56
Due to incompatibilities with some key third-party projects as documented in Issue 19490 and the urgency of getting new maintenance releases out, the best course of action is to revert built-in Tcl/Tk support for 3.3.3 and 2.7.6.  With the knowledge gained, I will try to implement a compatible solution for 3.4.0b1.
msg202207 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-11-05 11:47
I don't think you can provide a solution that's compatible with existing 3th-party extensions that use Tk and includes a private copy of Tcl/Tk. 

IMHO the best solution would be to provide the Tcl/Tk headers in the Python framework as well, to make it possible to link 3th party extension with the Tk that's provided with the Python installer.  

That should be an acceptable solution for 3.4 because that's a new feature release. 

A possible pain-point are 3th-party extensions that use the limited ABI, link with Tk and are used with Tkinter.
msg202216 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-05 15:09
New changeset ba31940588b6 by Ned Deily in branch '2.7':
Issue #15663: Revert OS X installer built-in Tcl/Tk support for 2.7.6.
http://hg.python.org/cpython/rev/ba31940588b6
msg203127 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-17 08:19
New changeset f9927dcc85cf by Ned Deily in branch '3.3':
Issue #15663: Revert OS X installer built-in Tcl/Tk support for 3.3.3.
http://hg.python.org/cpython/rev/f9927dcc85cf
msg203987 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-23 11:33
New changeset d666e8ee687d by Ned Deily in branch 'default':
Issue #15663: Revert OS X installer built-in Tcl/Tk support for 3.4.0b1.
http://hg.python.org/cpython/rev/d666e8ee687d
msg248081 - (view) Author: Mark Roseman (markroseman) * Date: 2015-08-05 21:49
I'd like to revisit this, particularly in the context of some of the discussions regarding future improvements of IDLE.

Assume we have a barebones Tcl/Tk framework build of e.g. 8.6.4 as part of the installer (core Tcl/Tk, not like e.g. ActiveTcl).

Would something like this work?

1. if just system Tcl/Tk already installed (/System/Library/Frameworks), install new /Library/Frameworks/Tcl.framework and /Library/Frameworks/Tk.framework

2. if /Library/Frameworks/Tcl.framework is installed, but doesn't have 8.6.x (e.g. may have Versions/8.4), add the new version into the existing framework, so we'd then have a 8.4.x and a 8.6.4 installed

3. if there's already /Library/Frameworks/Tcl.framework/Versions/8.6 and it's a lower version (e.g. 8.6.1) we could simply nag them in the installer to consider upgrading on their own (the theory being that if they've installed it once somehow they can most likely do it again with a newer version of whatever they have)

This way if there's any existing Tcl/Tk there that other things are linking against, it shouldn't get disturbed. Possible, or just wishful thinking?
msg251902 - (view) Author: Mark Roseman (markroseman) * Date: 2015-09-30 00:32
Ned, is there anything that I might be able to help with here? While I'm not a Mac installer guru, it doesn't look like we'd need anything too fancy here.

Installing an 8.6 variant (via the frameworks approach I mentioned in my previous message) would seem to have a less likely chance of stomping on things than previously, and hopefully shorten the time we have to live with 8.4 and buggy 8.5 releases.
msg251909 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-09-30 01:08
Due to our maintenance release compatibility guidelines, I don't think it is possible to change the Tcl/Tk versions for currently released Python versions (e.g 3.5.x, 3.4.x, and 2.7.x), or, at least, it is not possible to remove support for 8.5 in them.  To do so would break important existing third-party packages which supply binary distributions (as wheels or as binary installers).  So the target for shipping python.org OS X Pythons built with (and including) an 8.6 Tcl/Tk would be in the next feature release, 3.6.  There *might* be ways to include 8.6 as an option with future maintenance releases but it would have to be without breaking current 8.5 (or 8.4) binary support.  In any case, for 3.6 there will likely be major changes in how we ship pre-built Pythons for OS X.
msg251949 - (view) Author: Mark Roseman (markroseman) * Date: 2015-09-30 15:15
Thanks Ned. 

A couple of things. First, you probably know about this, but for future reference in case it might be useful, the install_name_tool lets you point a shared library at a different dependent shared library than the one it was originally compiled to link against (see end of http://www.tkdocs.com/tutorial/fonts.html). This can be sometimes helpful in resolving compatibility issues. 

Secondly, we may want to consider adding something post-install (in IDLE?) where it would detect at least simpler cases of greatly out-of-date Tcl/Tk and offer to download and install a new one. If it were in IDLE, this would have the advantage of not affecting everyone who installs Python, but only (a subset of) those who would benefit. Plus, not being a "you must decide now" like an installer, it could be deferred if people weren't sure. And it would offer to provide a bit more active assistance rather than only "go read a web page".
msg251950 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-09-30 15:46
"the install_name_tool lets you point a shared library at a different dependent shared library than the one it was originally compiled to link against"

Yes, but I'm not sure how we can take advantage of that.  First, keep in mind that there is only one version of _tkinter.so per Python installation and that is used by all tkinter applications, not just IDLE.  Second, the install_name_tool is not part of the base OS X system; it is included with either Xcode.app or the Command Line Tools package.  So we can't assume it will be available on users' systems.  And, third, changing the Tk library path may still break other third-party Python packages that also link directly with Tk and expect it to be in {/System}/Library/Frameworks.

"Secondly, we may want to consider adding something post-install (in IDLE?) where it would detect at least simpler cases of greatly out-of-date Tcl/Tk and offer to download and install a new one." 

That could be a useful addition and one that we could probably do in the context of a maintenance release.  The main problem I see with that is what to offer to install.  Since there is no truly open-source compatible option, I think we would need to provide our own Tcl/Tk 8.5.x package to install.  It's something to consider.
msg252034 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2015-10-01 16:07
W.r.t. install_name_tool: macholib (PyPI project) contains functionality to do this work in pure python.   

I do not propose to include macholib in the stdlib, it is too special purpose and not everyone that has provided patches is a CPython contributor (although the majority of the work was done by Bob Ippolito)
msg260945 - (view) Author: Aivar Annamaa (Aivar.Annamaa) Date: 2016-02-27 20:17
install_name_tool can specify relative paths (see eg. https://wincent.com/wiki/@executable_path,_@load_path_and_@rpath). Therefore you don't need it in users' systems.

I've used it successfully for bundling Python 3.5 and Tk 8.6 with my IDE, see https://bitbucket.org/plas/thonny/src/master/packaging
/mac/build_requirements_old.sh and https://bitbucket.org/plas/thonny/src/master/packaging/mac/update_links.sh
History
Date User Action Args
2016-09-02 01:03:11cjrhsetnosy: + cjrh
2016-02-27 20:17:16Aivar.Annamaasetnosy: + Aivar.Annamaa
messages: + msg260945
2016-02-23 10:10:01ned.deilylinkissue26409 superseder
2015-10-01 16:07:58ronaldoussorensetmessages: + msg252034
2015-09-30 15:46:44ned.deilysetmessages: + msg251950
2015-09-30 15:15:14markrosemansetmessages: + msg251949
2015-09-30 01:08:50ned.deilysettitle: Investigate providing Tcl/Tk 8.5 with OS X installers -> Investigate providing Tcl/Tk 8.6 with OS X installers
messages: + msg251909
versions: + Python 3.6, - Python 3.4
2015-09-30 00:32:07markrosemansetmessages: + msg251902
2015-08-05 21:49:25markrosemansetnosy: + terry.reedy, markroseman
messages: + msg248081
2014-02-02 05:46:03kbksetnosy: + kbk
2013-11-23 11:33:36python-devsetmessages: + msg203987
2013-11-17 08:19:32python-devsetmessages: + msg203127
2013-11-05 15:09:59python-devsetmessages: + msg202216
2013-11-05 11:47:29ronaldoussorensetmessages: + msg202207
2013-11-05 10:56:48ned.deilysetmessages: + msg202204
2013-11-05 10:51:52python-devsetmessages: + msg202203
2013-10-27 05:23:50python-devsetmessages: + msg201413
2013-10-26 10:19:01python-devsetmessages: + msg201332
2013-10-25 07:53:43python-devsetmessages: + msg201219
2013-10-21 03:59:59rhettingersetnosy: + rhettinger
messages: + msg200693
2013-10-20 11:09:59georg.brandlsetmessages: + msg200566
2013-10-19 21:47:49benjamin.petersonsetmessages: + msg200504
2013-10-19 21:20:56ned.deilysetnosy: + georg.brandl, benjamin.peterson
messages: + msg200501
2013-10-19 04:36:24python-devsetmessages: + msg200365
2013-09-06 12:51:00ronaldoussorensetmessages: + msg197064
2013-09-06 08:36:11ned.deilysetdependencies: + Mac OS X: building with X11 Tkinter
2013-09-06 08:22:04python-devsetnosy: + python-dev
messages: + msg197055
2012-12-31 07:29:05ronaldoussorensetmessages: + msg178646
2012-12-30 22:03:12esc24setnosy: + esc24
2012-12-30 21:59:06samueljohnsetnosy: + samueljohn
messages: + msg178619
2012-08-15 09:37:59ned.deilysetmessages: + msg168271
2012-08-15 09:25:01ronaldoussorensetmessages: + msg168270
2012-08-15 09:20:33ned.deilycreate