Proposal: duplicate nearly all of the existing idlelib *.py files, with new names.  This would implement the intention of PEP 434 with respect to freely changing idlelib APIs.  I view it as a prerequisite for most anticipated patches.  A longer than usual explanation follows.

Last August/September, I realized that the way to modernize and improve IDLE, while maintaining backward compatibility, would be to add copies of existing .py files.  The copies with new names (the idle3 set) could be freely edited  while existing files (the idle2 set) could be pretty much left alone until deleted.  IDLE would temporarily, for a time yet to be decided, have two operating modes.  The idle2 mode would only use exiting files and tk widgets.  The idle3 mode would use new files and ttk widgets.

Renaming and refactoring files, including PEP 8 updates, is needed for bug fixing, testing, and feature upgrades, including addition of a single window multiple pane design.  However, doing so with existing files would break direct imports and subsequent calls (see #24225, for some discussion).  Ttk widgets cannot be used on systems with tcl/tk 8.4. (This and related issues was discussed with Nick, Ned, and release managers, as well as in #24759, especially msg247993 & msg248031, and in #24750).

I originally planned to add new files one or a few at a time.  I did not start doing do immediately, before the November-December releases, because I wanted to fix some bugs in the existing files before they were bifurcated. I did not start doing so after the releases because I could not figure out how to safely manage an 'idle3/ttk' mode that mixed a shrinking subset of idle2 files and the growing set of new idle3 files.  It seemed like a recipe for introducing bugs.  I decided instead that the best solution would be to create a complete, tested idle3 set all at once, using the following steps.  (Idlelib/idle_test/*.py files will be handled similarly, but with differences discussed later.)

0. Decide on the new names.
1. Copy idlelib/*.py files and hg add them.
2. Revise imports in new idle3 files.
3. Test to make sure that the idle3 set works right. 
4. Upload patch for review, and commit.

Possible variation: push after step 1, so that the revisions in step 2 are a separate patch instead of being folded into a combined patch.  I do not propose that because the new files would be useless until revised.


DETAILS OF CURRENT PLAN:

0. @newnames.txt, https://bugs.python.org/file42678/%40newnames.txt is attached to issue #24225.  It lists old name, Sweigart's new name, and my current new name.  Bikeshedding names can be done on that issue.

Leaving aside the unused and deprecated 'idlever' and the startup files, all of which are omitted from the list, there are currently three short lowercase names: 'help', 'rpc', and 'run'.  To reuse these names for the idle3 set, these could be renamed, in a preliminary patch, to have a captital letter.  I think this should be okay because  'help' specifies that is it subject to change, and the other two are extremely unlikely to be used outside of IDLE.  If this were done, implementation files could be identified as 'idle2' versus 'idle3' by the presence of an uppercase letter in the name.

Serhiy requested on #24225 that renames of existing 3.5 files be backported to 2.7.  Since I do not anticipate many patches to idle2 files, let alone backports to 2.7, I am not sure that this is needed.

Alternatively, I could name the copies help3, rpc3, and run3.

1. Use the name list from step 0 to create copyall.bat, to be run in the 3.5 repository idlelib directory.  It will have 'copy src dst' for each file and end with 'hg add ../idlelib' (or 'cd ..; hg add idlelib') to 'add' them all.  I am aware of 'hg copy', but the docs are not clear on whether there is any lingering effect after pushing the adds.  The pre-commit merge effect seems unneeded as no one else is pushing idlelib commits (except Serhiy ocassionally, as part of multifile patches, and he should be reading this ;-).

2. Review Al Sweigart's 'idle_updating_imports.patch' from #24225.  Al ran revised unittests, but the coverage is too spotty to adequately test.  So review is needed.

The tricky part is that many modules and classes have the same name, such as AutoComplete the module and AutoComplete the class.  Module references must be changed to the new name; class references must not be touched.  I would use IDLE's File in Files (grep) with each module name to find references to check.  The import in each client module ('import X' versus 'from X import X' determines how bare 'X' is used in rest of the module.

If the patch seems correct, or mostly so, perhaps after a few corrections,  write a script, using the step 0 name list, to replace Al's new names with my new names, where they are different.

Apply the patch.  Since Al's patch is a year old, some chunks will fail to apply.  Do these by hand.

3. Testing: Make sure that all new files can be imported.  Does test___all__ do this?  Run revised idle_test/htest.py, which creates an example of all visible windows and widgets, and invites some human manipulation. Run revised unittest files, though coverage is spotty. Run idle3 with 'python -m idlelib.shell' and try all menu options.

4. With respect to idlelib, the patch will consist only of additions (of revised versions of copies with new names).  To check the macosx copy, 'python -m idlelib.shell' should be run on a Mac after applying the patch.


OTHER ISSUES

Unittests: The current 'test_xyz' files in 'idle_test' have lowercase names, anticipating the new names proposed here .  For instance, the tests for AutoComplete.py are in 'test_autocomplete.py' rather than 'test_AutoComplete.py'.  If it is desired to keep the incomplete set of tests for idle2 files, the files would have to be copied (with uppercase-containing names) before the imports in the current test files are updated.  A few of the current files will have to be renamed to match new names different from what I anticipated.

The incomplete automated unittests are supplemented by much more complete semi-automated human-mediated tests (htest.py).  But testing is still inadequate.  The November/December releases had the most patches for a long while, but also three regressions.  Two did not manifest on my Windows machine.  The worst (#26673, posted March30) appears to be linux-tk8.6 specific.

IDLE issues need to start following the 'test needed' rule.  To do so, it must be sensibly possible to write complete coverage tests, and that often requires refactoring.  Contemplating 'What tests could have caught the [first two] regressions?" is what prodded me to switch to this all-at-once plan.

Once written, the tests need to be run on all three platforms, perhaps with both tk 8.6 and tk 8.5 on Linux and Mac.  It would help if there were Linux buildbots that ran tkinter and idlelib tests that need a gui resource.  Short of that, refactoring is sometimes needed to separate gui from non-gui code.  I otherwise need to be more aggressive in getting myself and others to do what the buildbots don't or can't.


Revisiion history:  The repository history will stay with idle2 files.  The planned splitting and merging of files in the idle3 group would mix up history anyway.  I plan to add comments to modules, classes, and functions as to where they were copied from, so the history of particular code sections can be  manually traced.


Deletion of idle2 files:  The reason to copy rather than rename files is because renaming has not been considered currently possible.  However, Al Sweigart reported that the IdleX extension test suite ran after his renaming. If our only other back-compatibility concern were running on old Macs, the following might work: copy idlelib to, say, idlelib2; modernize idlelib as described here, except for starting with renames rather than copies; when making an old Mac installer, copy the frozen idlelib2 on top of idlelib.

Barring that alternative, the idle2 files will be deprecated and removed sometime from 3.6 to the planned removal of deprecated modules, after 2.7 support ends.  


PEP 434 and idlelib: PEP 434 declared idlelib 'mostly private', though we have not yet been willing to much act on that.  The purpose was to allow the changes described above, not to deny that some modules are or could be useful as library modules for other code.  With this issue, the idle2 files will be deprecated and mostly untouched, so there will be no point to calling them private.

I want to add an 'idlelib' section to the doc that briefly explains the package contents.  It might say something like "The main purpose of idlelib files is to implement IDLE.  There are two sets of files.  The idle2 files, stable and deprecated, are listed in README2.txt.  The idle3 files, discussed in README3.txt, are permanently private/provisional except and to the degree documented below."

The section would also document public APIs.  For the idle2 files, this would include how to use IDLE's syntax highlighting (as in turtledemo), and probably IDLE's scrolled widgets (unless and until moved to tkinter/).  In February, there were 2 Stackoverflow questions about using idlelib modules. One was about syntax hightlighting (answer: 'See turtledemo').  The other was about autocompletion (answer: "I don't know.  Refactoring is needed.").  So at least a few people are trying to use idlelib modules.

The planned refactoring of idle3 files to make things more usable within IDLE and easier to test will also make the same files more usable outside of IDLE.  For example, class ClassBrowser is hardcoded as a standalone Toplevel window.  The renamed ModuleBrowser will be a widget that can be placed on a Toplevel window, a pane of a PanedWindow, a tab of a Notebook, or anywhere else a widget can be put.  

ClassBrowser also hardcodes the double-click action as calling IDLE's filelist.open.  It has a rather opaque signature.  After refactoring, the new browser module should not need any indlelib imports and the renamed class might be documented as follows:

"browser.ModuleBrowser(master, module, opener, cnf={}, **kw).
 A ttk-based widget that displays the classes and functions of a module.  It comprises a ttk Frame, Treeview, and Scrollbars.  *module* is a path string, Path, or iterable of lines (open file or list of lines).  Double-clicking a tree item calls *opener*(module, line). Details are subject to change."  The allowed configuration options would also be listed.

Idle would pass 'opener=filelist.open'.  Tests would pass a mock that records calls.  Other users would pass their own click handler.  


IDLE and the stdlib:  Guido, last August on idle-sig, you both approved improving IDLE and expressed, as I understood it, an intention to delete idlelib as soon as you felt you could.  What is your intention now, after reading this concrete proposal?  Deprecate and someday remove everything? or keep improved IDLE indefinitely?

I think we need this decided now.  Once idlelib becomes in part a package of reusable and documented editor/IDE components, it would be much harder to remove.  On the otherhand, no one, including me, wants to spend substantial time on code expected to be deleted.

Terry, have you considered also doing a top level idlelib -> _idlelib rename in addition to the file level name changes?

My rationale for suggesting that:

1. The idle2 vs idle3 distinction will be clear at a glance (as they'll be in different directories)
2. The package level rename handles bifurcation of files that aren't themselves being renamed
3. You can subsequently iterate on the internal APIs without worrying about third party breakage
4. You set the stage for eventually moving IDLE to being an independently versioned implicitly pip-installed component (perhaps with an "ensure_default_ide" module akin to "ensurepip")

In relation to that last point, while I think it's important for CPython to come with a default IDE, I *don't* think it's important for the Python standard library to provide a default IDE-building toolkit (and, indeed, Linux distros tend to separate out not only IDLE, but also Tcl/Tk support in general, from the default Python installation).

In response to 1.  I considered copying idlelib as a whole as a serious possibility, meant to include it as an alternative, and am willing to reconsider my choice.  I specifically thought of something like idlelib3 or idlelib/idle3, but not _idlelib,  A complete copy in a new directory would be, as you say, a cleaner separation but it seemed like it would make import changes and startup more complicated.  It is also not obviously covered by PEP 434 -- unless you say so.

For users, making idle2 the copy would immediately break current external imports.  Making idle3 the copy would break new external imports sometime in the future.  But using _idlelib would warn of that possibility.

For importat, IDLE itself uses uses all of 'from idlelib import module', 'from idlelib.module import callable', 'from idlelib.idle_test.htest import run', and unittest.main('idlelib.idle_test.test_module' (but no relative imports).  A new directory would greatly increase the number of changes and mean changing 'idlelib' to 'something' now and 'something' back to 'idlelib' in the future.

I thought of using relative imports, but I have not used them and have read that they are limited, tricky, and discouraged.  I just did some experiments and it seems that they do not work when a file is run as __main__ instead of imported.  (Whatever the limitation, it is not obvious to me from the doc.)

I agree with 2. and 3. and will response to 4. in another message.

Terry, I'm all for having IDLE using the newer Ttk widgets and for refactoring to make it easier to enhance and maintain IDLE.  I'm not sure I understand the proposal to make two sets of IDLE files: does this mean there would be two versions of IDLE in one branch, e.g. Python 3.6?  That doesn't make sense to me.  One issue you mention is continuing to support Tk 8.4 (which lacks Ttk) on legacy systems.  Tk 8.4 support ended some years ago by the Tcl Core team. I think it is time for us to draw a line in the sand and say we no longer need to continue to support it for IDLE.  (It would be nice to not intentionally break basic non-IDLE Tkinter support for 8.4.)  I haven't yet announced any proposals for Mac installer support for 3.6 but I'm willing to commit to not requiring any Tcl/Tk 8.4 dependencies.   So, if we assume 8.4 is no longer an issue, what other issues remain?  I believe you've already stated that you will no longer attempt to keep the 2.7 version of IDLE in sync with the latest IDLE 3; I thinks that's the right approach at this stage.  That means there will be only some backporting of selected IDLE bugfixes to 2.7 and we don't do any direct VCS merging between 2.7 and default anyway.  So it seems to me there is no reason why the default branch (e.g. what is going into 3.6) needs to retain any copies of IDLE files from the 2.7 branch.  In other words, all the file renamings (via hg mv or otherwise) and deletions just take place in default without disturbing 2.7.  If there are any external incompatibilities, it's fine to announce them as part of the 3.6 feature release.  This is assuming that the externally incompatible work is in place in time for the 3.6 feature code cutoff (3.6.0 beta 1, currently scheduled for 2016-09-08).  As far as 3.5.n is concerned, the last bugfix release for 3.5 is likely to be less than a year away (assuming 3.6.0 releases by the end of 2016 as planned).  So I would also recommend not making any refactoring changes for 3.5, just important bugfixes similar to 2.7.  We should try to make it easier for you and the others working on IDLE.  And try to minimize the risks of inadvertent breakages going into maintenance releases.  Does that make sense?

Ned, thank you for the helpful response.  Yes, I *was* proposing two versions in 3.5 and subsequent releases until the current version could be deleted.  Its an ugly, least bad alternative that only made sense under the presumption that 'idle2' had to remain in 3.6.   With that assumption dropped ...

I would be delighted to start renaming (and refactoring) existing files in default *now*.  Whether to renames all at once (the current #24225 proposal) or in batches would be a sub-issue of #24225.

I believe that the PEP 434 exemption from separating 'behavior' from 'enhancement' issue applies during the beta period as well as after.  For IDLE, danger of accidental regression and thoroughness of testing are more relevant for deciding timing of commits.

As for 3.5, if we assume that substantial numbers of people, including instructors, will use the final 3.5 (likely 3.5.3) instead of 3.6.0, then it might be better for users to include 'idle3' in 3.5.3 as an option.  This could be done a couple of weeks before the release by copying 3.6 idlelib into, for instance, 3.5 idlelib._, editing imports, and adding a switch to the startup code.  If Ned's alternative is accepted, the backport decision should be deferred until there is something worth backporting.

Glad to be of help with regards to 8.4 :=)  With my 3.6 release manager hat on, I think the best way to think of this is as a new feature: call it "Ttk IDLE" or some such (and not idle2 vs idle3 which I think is confusing) and, rather than inventing something new, I suggest you should just follow the standard procedure for new feature development as documented in the Developers Guide.  That is, create a temporary repository on hg.python.org and make all the changes in a new branch in that repo.  Or, if you'd rather work in private, just create your own local repo clone and branch within it.  When you are satisfied that it is ready to be integrated into the release, we can either merge the branch in or extract the changes as one big patch to be applied to the main cpython default branch.  And I'll be happy to help with making that happen when the time comes.  That way you and any one else wanting to work on IDLE can work unimpeded, you don't have to worry about temporary changes and hacks to accommodate two versions of IDLE in the same repo, the old version of IDLE continues to work in the main cpython default branch until TtkIDLE is ready, and the cpython repo is not cluttered with all the intermediate commits from the feature dev cycle (although they could be preserved during integration, if desired).  Plus, during feature development you should be able to use TtkIDLE from the feature repo with a standard 3.6 (or 3.5) binary install of Python.

As for timing, I would prefer to get such a major feature in by feature code cutoff but, as you say, IDLE is pretty self-contained so we can be flexible.  A similar argument would apply to 3.5.  But I think it makes sense to focus on 3.6 first and then later separately consider dropping it all into 3.5.x, a decision that would be up to Larry as the 3.5 release manager.  For 3.6, I don't think there's a need for further discussion or to wait to begin, as long as the development work is done in a feature repo.  Have at it!

https://docs.python.org/devguide/committing.html#long-term-development-of-features

By the way, if you are essentially replacing the IDLE files, then (contrary to what I suggested earlier) you probably don't want to use hg mv to rename files.  Otherwise you can end up with needless merge conflicts between 3.x branches.  In the feature branch, you can just do the equivalent of: hg rm Lib/idlelib and hg commit then copy any old idlelib files you want to retain back into Lib/idlelib, and do an hg add and hg commit.  Then the feature branch files are no longer "connected" to the old IDLE files from hg's perspective.  On the other hand, if you go the feature repo route, you won't have two copies of IDLE in the same branch at a time, so you don't *have* to rename files if you don't otherwise want to.  In any case, if the feature is integrated as one big commit, all of this can be handled at integration time.

+1 for Ned's point that PEP 434 grants the IDLE developers *permission* to backport IDLE enhancements to maintenance releases, but doesn't *oblige* them to provide such backports.

For Python 3.6, another point to keep in mind is that if anyone particularly wants to keep the old idlelib API alive, they can either fork the existing code as a PyPI project, or create a compatibility shim that runs atop the new ttk-based API.

I should have explained my usage of 'idle2' and 'idle3'.  'Idle1' was the original IDLE that executed user code in the idle process.  'Idle2' was the re-write, still in use, that executes user code in a separate process.  'Idle3' is intended to be a re-write of similar scope that uses ttk, but also refactors to make automated tests and other changes easier.  (Adding multi-pane windows might be called 'idle4'.)  It is the tests and other changes that require refactoring.  Hence 'ttk-idle' partly misses the point unless understood as including name changes and refactoring.

For the idle2 re-writes, all the new code needed to execute remotely, by default, while keeping local execution as an option, had to be in place for any of it to be useful. The idle3 changes can be broken up into multiple usable and testable pieces.  I wish the renames could be in alpha1 and definitely want them in alpha2, along with a notice in What's New.  Anyone who has code that will break because of the changes should have opportunity to find out as soon as possible.  I would like to do simple ttk replacements by alpha 3.

In response to Nick's point 4: To me, these are important aspects of 'in the stdlib'.

First is having issues on this tracker.  As Guido said in his King Day speech, non-trivial software is a collective project.  IDLE needs continued access to dependency experts, such as Serhiy for tkinter, and platform experts, such as Ned for Mac.  In April, I nosyed Kristján Valur Jónsson to ask an IDLE question that seemed related to a socket patch he pushed, and he helpfully answered.  On the other side, users are much more likely to participate here than on a separate tracker.

Second is being easily installable with the rest of Python.  I think some out-of-the-box augment to the console is crucial on Windows.

I am aware that Linux distributions have a separate tkinter-gui install instead of just a checkbox option.  I think it should consistently, on all plateforms, comprise _tkinter, tkinter, idlelib, turtle, and turtledemo.

All of those except idlelib are documented even though optional.  So I think idlelib could be also.  But documenting direct usage, which Nick recommended in #24225, could be relegated to the modules themselves and the idlelib README or a new text file.  Ditto for indicating replacements for thing known or strongly  suspected of external use.

Both Serhiy and Ned have expressed concern about the difficulty of merging patches from 3.5 to 3.6 once the 3.6 files are renamed.  To me, this is a non-issue since I don't expect to do that more than a few times, and some may be null merges of backports of fixes originally developed for and applied to 3.6.  And I don't expect another core developer to start applying patches to 3.5.

Hence my opinion that I should use hg rename, to keep continuity of revision history as much as possible, and that I should push a complete rename patch as soom as possible.

Two things that I *will* apply to 3.5, if and when possible, are fixing the third regression (Idle preferences not opening on Linux with tk 8.6, #26673) and fixing freezing of Macs when clicking on a completion window (#15786).

To me, this issue is a start on paying down IDLE's technical debt.

I finished reopening #24225 (renaming), for 3.6 only.  I will open a new issue with Larry Hastings nosy, should I proposed (next fall) to backport the revamped idlelib to 3.5.

Files were renamed later in May.  Some things were backported to 3.5 and even 2.7 for 3.5.3 and 2.7.13.  Anything involving ttk, which will soon be nearly all tkinter code, could not and cannot be backported.