Proposal: duplicate nearly all of the existing idlelib *.py files, with new names.  This would implement the intention of PEP 434 with respect to freely changing idlelib APIs.  I view it as a prerequisite for most anticipated patches.  A longer than usual explanation follows.

Last August/September, I realized that the way to modernize and improve IDLE, while maintaining backward compatibility, would be to add copies of existing .py files.  The copies with new names (the idle3 set) could be freely edited  while existing files (the idle2 set) could be pretty much left alone until deleted.  IDLE would temporarily, for a time yet to be decided, have two operating modes.  The idle2 mode would only use exiting files and tk widgets.  The idle3 mode would use new files and ttk widgets.

Renaming and refactoring files, including PEP 8 updates, is needed for bug fixing, testing, and feature upgrades, including addition of a single window multiple pane design.  However, doing so with existing files would break direct imports and subsequent calls (see #24225, for some discussion).  Ttk widgets cannot be used on systems with tcl/tk 8.4. (This and related issues was discussed with Nick, Ned, and release managers, as well as in #24759, especially msg247993 & msg248031, and in #24750).

I originally planned to add new files one or a few at a time.  I did not start doing do immediately, before the November-December releases, because I wanted to fix some bugs in the existing files before they were bifurcated. I did not start doing so after the releases because I could not figure out how to safely manage an 'idle3/ttk' mode that mixed a shrinking subset of idle2 files and the growing set of new idle3 files.  It seemed like a recipe for introducing bugs.  I decided instead that the best solution would be to create a complete, tested idle3 set all at once, using the following steps.  (Idlelib/idle_test/*.py files will be handled similarly, but with differences discussed later.)

0. Decide on the new names.
1. Copy idlelib/*.py files and hg add them.
2. Revise imports in new idle3 files.
3. Test to make sure that the idle3 set works right. 
4. Upload patch for review, and commit.

Possible variation: push after step 1, so that the revisions in step 2 are a separate patch instead of being folded into a combined patch.  I do not propose that because the new files would be useless until revised.


DETAILS OF CURRENT PLAN:

0. @newnames.txt, https://bugs.python.org/file42678/%40newnames.txt is attached to issue #24225.  It lists old name, Sweigart's new name, and my current new name.  Bikeshedding names can be done on that issue.

Leaving aside the unused and deprecated 'idlever' and the startup files, all of which are omitted from the list, there are currently three short lowercase names: 'help', 'rpc', and 'run'.  To reuse these names for the idle3 set, these could be renamed, in a preliminary patch, to have a captital letter.  I think this should be okay because  'help' specifies that is it subject to change, and the other two are extremely unlikely to be used outside of IDLE.  If this were done, implementation files could be identified as 'idle2' versus 'idle3' by the presence of an uppercase letter in the name.

Serhiy requested on #24225 that renames of existing 3.5 files be backported to 2.7.  Since I do not anticipate many patches to idle2 files, let alone backports to 2.7, I am not sure that this is needed.

Alternatively, I could name the copies help3, rpc3, and run3.

1. Use the name list from step 0 to create copyall.bat, to be run in the 3.5 repository idlelib directory.  It will have 'copy src dst' for each file and end with 'hg add ../idlelib' (or 'cd ..; hg add idlelib') to 'add' them all.  I am aware of 'hg copy', but the docs are not clear on whether there is any lingering effect after pushing the adds.  The pre-commit merge effect seems unneeded as no one else is pushing idlelib commits (except Serhiy ocassionally, as part of multifile patches, and he should be reading this ;-).

2. Review Al Sweigart's 'idle_updating_imports.patch' from #24225.  Al ran revised unittests, but the coverage is too spotty to adequately test.  So review is needed.

The tricky part is that many modules and classes have the same name, such as AutoComplete the module and AutoComplete the class.  Module references must be changed to the new name; class references must not be touched.  I would use IDLE's File in Files (grep) with each module name to find references to check.  The import in each client module ('import X' versus 'from X import X' determines how bare 'X' is used in rest of the module.

If the patch seems correct, or mostly so, perhaps after a few corrections,  write a script, using the step 0 name list, to replace Al's new names with my new names, where they are different.

Apply the patch.  Since Al's patch is a year old, some chunks will fail to apply.  Do these by hand.

3. Testing: Make sure that all new files can be imported.  Does test___all__ do this?  Run revised idle_test/htest.py, which creates an example of all visible windows and widgets, and invites some human manipulation. Run revised unittest files, though coverage is spotty. Run idle3 with 'python -m idlelib.shell' and try all menu options.

4. With respect to idlelib, the patch will consist only of additions (of revised versions of copies with new names).  To check the macosx copy, 'python -m idlelib.shell' should be run on a Mac after applying the patch.


OTHER ISSUES

Unittests: The current 'test_xyz' files in 'idle_test' have lowercase names, anticipating the new names proposed here .  For instance, the tests for AutoComplete.py are in 'test_autocomplete.py' rather than 'test_AutoComplete.py'.  If it is desired to keep the incomplete set of tests for idle2 files, the files would have to be copied (with uppercase-containing names) before the imports in the current test files are updated.  A few of the current files will have to be renamed to match new names different from what I anticipated.

The incomplete automated unittests are supplemented by much more complete semi-automated human-mediated tests (htest.py).  But testing is still inadequate.  The November/December releases had the most patches for a long while, but also three regressions.  Two did not manifest on my Windows machine.  The worst (#26673, posted March30) appears to be linux-tk8.6 specific.

IDLE issues need to start following the 'test needed' rule.  To do so, it must be sensibly possible to write complete coverage tests, and that often requires refactoring.  Contemplating 'What tests could have caught the [first two] regressions?" is what prodded me to switch to this all-at-once plan.

Once written, the tests need to be run on all three platforms, perhaps with both tk 8.6 and tk 8.5 on Linux and Mac.  It would help if there were Linux buildbots that ran tkinter and idlelib tests that need a gui resource.  Short of that, refactoring is sometimes needed to separate gui from non-gui code.  I otherwise need to be more aggressive in getting myself and others to do what the buildbots don't or can't.


Revisiion history:  The repository history will stay with idle2 files.  The planned splitting and merging of files in the idle3 group would mix up history anyway.  I plan to add comments to modules, classes, and functions as to where they were copied from, so the history of particular code sections can be  manually traced.


Deletion of idle2 files:  The reason to copy rather than rename files is because renaming has not been considered currently possible.  However, Al Sweigart reported that the IdleX extension test suite ran after his renaming. If our only other back-compatibility concern were running on old Macs, the following might work: copy idlelib to, say, idlelib2; modernize idlelib as described here, except for starting with renames rather than copies; when making an old Mac installer, copy the frozen idlelib2 on top of idlelib.

Barring that alternative, the idle2 files will be deprecated and removed sometime from 3.6 to the planned removal of deprecated modules, after 2.7 support ends.  


PEP 434 and idlelib: PEP 434 declared idlelib 'mostly private', though we have not yet been willing to much act on that.  The purpose was to allow the changes described above, not to deny that some modules are or could be useful as library modules for other code.  With this issue, the idle2 files will be deprecated and mostly untouched, so there will be no point to calling them private.

I want to add an 'idlelib' section to the doc that briefly explains the package contents.  It might say something like "The main purpose of idlelib files is to implement IDLE.  There are two sets of files.  The idle2 files, stable and deprecated, are listed in README2.txt.  The idle3 files, discussed in README3.txt, are permanently private/provisional except and to the degree documented below."

The section would also document public APIs.  For the idle2 files, this would include how to use IDLE's syntax highlighting (as in turtledemo), and probably IDLE's scrolled widgets (unless and until moved to tkinter/).  In February, there were 2 Stackoverflow questions about using idlelib modules. One was about syntax hightlighting (answer: 'See turtledemo').  The other was about autocompletion (answer: "I don't know.  Refactoring is needed.").  So at least a few people are trying to use idlelib modules.

The planned refactoring of idle3 files to make things more usable within IDLE and easier to test will also make the same files more usable outside of IDLE.  For example, class ClassBrowser is hardcoded as a standalone Toplevel window.  The renamed ModuleBrowser will be a widget that can be placed on a Toplevel window, a pane of a PanedWindow, a tab of a Notebook, or anywhere else a widget can be put.  

ClassBrowser also hardcodes the double-click action as calling IDLE's filelist.open.  It has a rather opaque signature.  After refactoring, the new browser module should not need any indlelib imports and the renamed class might be documented as follows:

"browser.ModuleBrowser(master, module, opener, cnf={}, **kw).
 A ttk-based widget that displays the classes and functions of a module.  It comprises a ttk Frame, Treeview, and Scrollbars.  *module* is a path string, Path, or iterable of lines (open file or list of lines).  Double-clicking a tree item calls *opener*(module, line). Details are subject to change."  The allowed configuration options would also be listed.

Idle would pass 'opener=filelist.open'.  Tests would pass a mock that records calls.  Other users would pass their own click handler.  


IDLE and the stdlib:  Guido, last August on idle-sig, you both approved improving IDLE and expressed, as I understood it, an intention to delete idlelib as soon as you felt you could.  What is your intention now, after reading this concrete proposal?  Deprecate and someday remove everything? or keep improved IDLE indefinitely?

I think we need this decided now.  Once idlelib becomes in part a package of reusable and documented editor/IDE components, it would be much harder to remove.  On the otherhand, no one, including me, wants to spend substantial time on code expected to be deleted.