The current situation with Tcl/Tk on OS X is less than ideal. The Apple-supplied versions of the Cocoa-based Aqua Tk 8.5 on 10.6 was unusable with IDLE and most Tkinter applications. The 10.7 and 10.8 versions are better but still have serious crashers that have been more recently fixed. python.org installers currently also support using ActiveTcl release but that is also not ideal as there are license restrictions for some users and a separate download and install is required. We should be able to provide a better out-of-the-box experience if python.org installers included its own copies of up-to-date Tcl/Tk libraries (as is the case with the Windows installers). It would be good to still give users the option to use ActiveTcl for any of its additional features or to use system Python. Also we could also look at providing Tcl/Tk 8.5 for 10.5 users (possibly only Intel ones) which we currently do not.

A major reason for why the current installers don't install Tk as well is that this would significantly increase the size of the installer.

There's also the question of what bits of Tcl/Tk should be bundled, although I guess we should ship a minimal install of Tk that still supports everything that is wrapped by the python std lib. Alternatively we could ship the same subset as the Windows installer.

Yes, it would have to be a subset since some of the stuff in the ActiveTcl releases is unavailable anyway. But the current ActiveTcl installer dmgs are around 24Mb while the 3.3.0 installers are a little less than 20Mb. Even if it were to double the size, 40Mb still seems pretty small by today's standards. And following the Windows installer lead would be a good starting point.

Sounds reasonable, but please put the libs not in a place that might conflict with a Tk installed by other means (e.g. homebrew's Tk is a /usr/local).

The logical location for installing and embedded Tcl/Tk is either in {sys.prefix}/lib (for a unix install of Tk) or in Python.framework/Versions/X.Y/Frameworks (for a Tcl/Tk framework install).

In either case Tcl/Tk would be installed *inside* Python.framework and hence couldn't conflict with other installations of Tcl/Tk (system, activate-state, homebrew, ...).

Note that this would also make it harder for end users to upgrade Tcl/Tk, currently they can use the ActiveState installer to install a better Tk release and that would no longer be necessary.

That would be a small cost relative to the large gain of having a working Tk release.

New changeset 985384cd6365 by Ned Deily in branch 'default':
Issue bpo-15663: Tcl/Tk 8.5.14 is now included with the OS X 10.6+
http://hg.python.org/cpython/rev/985384cd6365

Great work. Thanks.

New changeset 7f69f4fadfd8 by Ned Deily in branch 'default':
Issue bpo-15663: Update OS X installer to use Tcl/Tk 8.5.15.
http://hg.python.org/cpython/rev/7f69f4fadfd8

I guess the one remaining question about this issue is whether it should be backported to 2.7.x and/or 3.3.x. The implementation depends on the build changes implemented in bpo-1584 to provide options to override default search paths for Tcl and Tk so that would have to be back ported as well. IMO, the entire set of changes are fairly well isolated and should be low risk: basically, they are all build-related changes and mostly isolated to OS X. The primary argument for backporting is that Tk support on current releases of OS X (from 10.6 Snow Leopard on) has been a major pain for our users, such that we have to keep recommending that they install a newer third-party Tcl/Tk to use IDLE or anything else Tkinter-based. So that goes against the batteries-included mission of the python.org installer, especially since the third-party version (ActiveTcl) is not totally free and open source (although it is free as in beer for most uses). On the other hand, one can argue that this is a new feature and thus prohibited in maintenance releases. On the other other hand, one can argue that this is a change driven by changes (or lack thereof) in the underlying OS platform and *essentially* only changes the building of the python.org OS X installer, so it is similar in concept to any other third-party distributor of Python changing an underlying system library. I would be OK with backporting or not but the decision needs to be made by the release managers. Benjamin? Georg?

We've upgraded the versions of other embedded libraries in 2.7 bugfix releases, so I'm okay with this for 2.7.

Yep (should there be a policy about this somewhere)?

We should advertise in the docs that we have a policy of upgrading the Tcl/Tk versions as necessary to improve stability.

New changeset 19b512bfc8d2 by Ned Deily in branch '2.7':
Issue bpo-15663: Tcl/Tk 8.5.15 is now included with the OS X 10.6+
http://hg.python.org/cpython/rev/19b512bfc8d2

New changeset 5ff4f0100a40 by Ned Deily in branch '3.3':
Issue bpo-15663: Tcl/Tk 8.5.15 is now included with the OS X 10.6+
http://hg.python.org/cpython/rev/5ff4f0100a40

New changeset db5a50959dc9 by Ned Deily in branch '2.7':
Issue bpo-15663: OS X installer builtin Tcl/Tk support
http://hg.python.org/cpython/rev/db5a50959dc9

New changeset 150242784de0 by Ned Deily in branch '3.3':
Issue bpo-15663: OS X installer builtin Tcl/Tk support
http://hg.python.org/cpython/rev/150242784de0

New changeset b599584ff1e3 by Ned Deily in branch 'default':
Issue bpo-15663: OS X installer builtin Tcl/Tk support
http://hg.python.org/cpython/rev/b599584ff1e3

New changeset 0bf4cf62f14c by Ned Deily in branch '3.3':
Issue bpo-15663: Force IDLE.app to run the GUI process in 32-bit mode.
http://hg.python.org/cpython/rev/0bf4cf62f14c

New changeset b6a1a78818fe by Ned Deily in branch 'default':
Issue Issue bpo-15663: merge from 3.3
http://hg.python.org/cpython/rev/b6a1a78818fe

New changeset fc8f19b4b662 by Ned Deily in branch '2.7':
Issue bpo-15663: Revert OS X installer built-in Tcl/Tk support for 2.7.6.
http://hg.python.org/cpython/rev/fc8f19b4b662

New changeset 268dc81c2527 by Ned Deily in branch '3.3':
Issue bpo-15663: Revert OS X installer built-in Tcl/Tk support for 3.3.3.
http://hg.python.org/cpython/rev/268dc81c2527

New changeset f89beccd470c by Ned Deily in branch 'default':
Issue bpo-15663: merge build-installer.py changes
http://hg.python.org/cpython/rev/f89beccd470c

Due to incompatibilities with some key third-party projects as documented in bpo-19490 and the urgency of getting new maintenance releases out, the best course of action is to revert built-in Tcl/Tk support for 3.3.3 and 2.7.6. With the knowledge gained, I will try to implement a compatible solution for 3.4.0b1.

I don't think you can provide a solution that's compatible with existing 3th-party extensions that use Tk and includes a private copy of Tcl/Tk.

IMHO the best solution would be to provide the Tcl/Tk headers in the Python framework as well, to make it possible to link 3th party extension with the Tk that's provided with the Python installer.

That should be an acceptable solution for 3.4 because that's a new feature release.

A possible pain-point are 3th-party extensions that use the limited ABI, link with Tk and are used with Tkinter.

New changeset ba31940588b6 by Ned Deily in branch '2.7':
Issue bpo-15663: Revert OS X installer built-in Tcl/Tk support for 2.7.6.
http://hg.python.org/cpython/rev/ba31940588b6

New changeset f9927dcc85cf by Ned Deily in branch '3.3':
Issue bpo-15663: Revert OS X installer built-in Tcl/Tk support for 3.3.3.
http://hg.python.org/cpython/rev/f9927dcc85cf

New changeset d666e8ee687d by Ned Deily in branch 'default':
Issue bpo-15663: Revert OS X installer built-in Tcl/Tk support for 3.4.0b1.
http://hg.python.org/cpython/rev/d666e8ee687d

I'd like to revisit this, particularly in the context of some of the discussions regarding future improvements of IDLE.

Assume we have a barebones Tcl/Tk framework build of e.g. 8.6.4 as part of the installer (core Tcl/Tk, not like e.g. ActiveTcl).

Would something like this work?

1. if just system Tcl/Tk already installed (/System/Library/Frameworks), install new /Library/Frameworks/Tcl.framework and /Library/Frameworks/Tk.framework

2. if /Library/Frameworks/Tcl.framework is installed, but doesn't have 8.6.x (e.g. may have Versions/8.4), add the new version into the existing framework, so we'd then have a 8.4.x and a 8.6.4 installed

3. if there's already /Library/Frameworks/Tcl.framework/Versions/8.6 and it's a lower version (e.g. 8.6.1) we could simply nag them in the installer to consider upgrading on their own (the theory being that if they've installed it once somehow they can most likely do it again with a newer version of whatever they have)

This way if there's any existing Tcl/Tk there that other things are linking against, it shouldn't get disturbed. Possible, or just wishful thinking?

Ned, is there anything that I might be able to help with here? While I'm not a Mac installer guru, it doesn't look like we'd need anything too fancy here.

Installing an 8.6 variant (via the frameworks approach I mentioned in my previous message) would seem to have a less likely chance of stomping on things than previously, and hopefully shorten the time we have to live with 8.4 and buggy 8.5 releases.

Due to our maintenance release compatibility guidelines, I don't think it is possible to change the Tcl/Tk versions for currently released Python versions (e.g 3.5.x, 3.4.x, and 2.7.x), or, at least, it is not possible to remove support for 8.5 in them. To do so would break important existing third-party packages which supply binary distributions (as wheels or as binary installers). So the target for shipping python.org OS X Pythons built with (and including) an 8.6 Tcl/Tk would be in the next feature release, 3.6. There *might* be ways to include 8.6 as an option with future maintenance releases but it would have to be without breaking current 8.5 (or 8.4) binary support. In any case, for 3.6 there will likely be major changes in how we ship pre-built Pythons for OS X.

Thanks Ned.

A couple of things. First, you probably know about this, but for future reference in case it might be useful, the install_name_tool lets you point a shared library at a different dependent shared library than the one it was originally compiled to link against (see end of http://www.tkdocs.com/tutorial/fonts.html). This can be sometimes helpful in resolving compatibility issues.

Secondly, we may want to consider adding something post-install (in IDLE?) where it would detect at least simpler cases of greatly out-of-date Tcl/Tk and offer to download and install a new one. If it were in IDLE, this would have the advantage of not affecting everyone who installs Python, but only (a subset of) those who would benefit. Plus, not being a "you must decide now" like an installer, it could be deferred if people weren't sure. And it would offer to provide a bit more active assistance rather than only "go read a web page".

"the install_name_tool lets you point a shared library at a different dependent shared library than the one it was originally compiled to link against"

Yes, but I'm not sure how we can take advantage of that. First, keep in mind that there is only one version of _tkinter.so per Python installation and that is used by all tkinter applications, not just IDLE. Second, the install_name_tool is not part of the base OS X system; it is included with either Xcode.app or the Command Line Tools package. So we can't assume it will be available on users' systems. And, third, changing the Tk library path may still break other third-party Python packages that also link directly with Tk and expect it to be in {/System}/Library/Frameworks.

"Secondly, we may want to consider adding something post-install (in IDLE?) where it would detect at least simpler cases of greatly out-of-date Tcl/Tk and offer to download and install a new one."

That could be a useful addition and one that we could probably do in the context of a maintenance release. The main problem I see with that is what to offer to install. Since there is no truly open-source compatible option, I think we would need to provide our own Tcl/Tk 8.5.x package to install. It's something to consider.

W.r.t. install_name_tool: macholib (PyPI project) contains functionality to do this work in pure python.

I do not propose to include macholib in the stdlib, it is too special purpose and not everyone that has provided patches is a CPython contributor (although the majority of the work was done by Bob Ippolito)

install_name_tool can specify relative paths (see eg. https://wincent.com/wiki/@executable_path,_@load_path_and_@rpath). Therefore you don't need it in users' systems.

I've used it successfully for bundling Python 3.5 and Tk 8.6 with my IDE, see https://bitbucket.org/plas/thonny/src/master/packaging
/mac/build_requirements_old.sh and https://bitbucket.org/plas/thonny/src/master/packaging/mac/update_links.sh

Is there anything left to do for this? The new 64 bit MacOS 3.7.0b1 links with 8.6.7. Perhaps that should be updated to 8.6.8.

As of 3.7.0b1, the new macOS 10.9+ installer variant includes a private copy of Tcl/Tk 8.6: 8.6.7 for 3.7.0b1, updated to 8.6.8 for 3.7.0b2. The framework layout is pretty much the same as in the original attempt, that is, everything is installed within /Library/Frameworks/Python.framework. One big thing that has changed since the previous attempt: thanks to work by Matthew Brett et al, major Python packages that had C code referencing Tk (like matplotlib and Pillow) now discover Tcl and Tk symbols dynamically at run time rather than having to compile with a particular Tcl/Tk version's header files. That was the hangup previously because third-party apps had hardcoded paths to the Tcl/Tk header files on macOS. Hopefully, there aren't many of those left and, if so, they can borrow code from matplotlib (matplotlib/matplotlib#6442). The legacy 10.6+ installer variants still link with Tcl/Tk 8.5.x from /Library/Frameworks/Tk.framework (a la ActiveTcl); ActiveState now only provides 8.6.x packages for macOS 10.9+ and Tk 8.6.x gets little exposure on older systems and depends more and more on features in newer macOS releases.

Update: as of 3.7.0b3, both 3.7.0 installer variants (10.9+ and 10.6+) provide a built-in Tcl/Tk 8.6.8. And 3.6.5 and 2.7.15 also provide a 10.9+ installer variant with the built-in Tcl/Tk 8.6.8. (For the moment, the 3.6.5 and 2.7.15 10.6+ 64-bit/32-bit installers still link with an external Tcl/Tk 8.5.x.)

New changeset f74cabd by Ned Deily in branch '3.6':
[3.6] bpo-15663: the 10.6+ macOS installers for 3.6/2.7 now provide a private Tcl/Tk 8.6 (GH-11109)
f74cabd

New changeset d0d09b5 by Ned Deily in branch '2.7':
[2.7] bpo-15663: the 10.6+ macOS installers for 3.6/2.7 now provide a private Tcl/Tk 8.6 (GH-11110)
d0d09b5

Update: as of 3.6.8rc1 and the eventual 2.7.16rc1, both installer variants (10.9+ and 10.6+) provide a built-in Tcl/Tk 8.6.8. For python.org installers, Tcl/Tk 8.5.x is finally dead!