Followup to #24225, which renamed idlelib files.

Currently, an arbitrary set of 9 (of 10, if Check Module is counted as a separate feature) IDLE features are implemented as extensions.  Their menu names are as follows:
  Edit - Show Completions
  Edit - Expand Word
  Edit - Show call tip
  Edit - Show surrounding parens
  Format - Format Paragraph
  Format - Strip trailing whitespace
  Run - Check Module (same file as below)
  Run - Run Module
  Options - Code Context
  Window - Zoom Height

This issue proposes to turn these into normal features implemented in regular code. 

1. The menu entries for extensions are not included in the main menu definition file (mainmenu, previously Bindings).  Rather, they are appended when the file is read.  This is constricting.  The order of the 4 edit entries is the alphabetical order of the corresponding file names. Run Module should be at the top of the Run menu, rather than being shoved down further if and when more non-extension entries are added.

2. Each feature is a separate file, usually small to medium with one class.  This bloats the number of modules in idlelib (now about 60).  I am considering combining autoexpand and parenmatch, paragraph and rstrip, and codecontext and zoomheight (and moving codecontext to the Window menu).  They might be joined with other features from the same menu.

3. The user customization system is not very compatible with change.  The problems with extension customization and changing file names is discussed in the second half of msg266213.  If extensions are not converted, another solution will be needed.

I have not yet tested the effect of adding new key bindings to config-keys.def and config-keys.cfg.  If this causes a problem for current releases, then this is a problem we need to face.  I anticipate wanting to add other new keybindings, such as one to switch tabs when we use a notebook.

I don't have an opinion on the matter, other than a reminder about the current cross-version (and cross-platform) compatibility issues with a user's IDLE configuration files as alluded to in Issue20580.  Perhaps others more familiar with IDLE's internals would have comments?

The version conflict problem is why I reverted changing 'extension names' to match the new file names, before rebase collapsing my patches.  I was really glad most were in a separate patch.

Each time a feature ceases to be an extension, the (old) extension name will have to be added to a set of names ignored when calculating the set of extensions.  I will start with one until I have worked out and tested the details.

The final paragraph of my initial post should have talked about config-main and features, rather than keys.

The following experiment indicates that adding new sections to config-main.def and customizations thereof to config-main.cfg should not be a problem. 

Add the following to config-main.cfg
[NewSection]
new = True

Open IDLE 2.7 from console (python, with import).
Open config dialog.
Change item.
Close with [OK]
confirm that change is written and new section is left alone.
Close IDLE.
Repeat with 3.5.

Old versions will not see new config-main.def, so that is not an issue.

Conclusion: adding a new section that old versions ignore is much safer then adding a new value for a current section&item that old versions do read and act on, and which may refer to something that does not exist.  (This was the problem with IDLE Dark theme and would be with Unix New keyset).

The remaining issues:

4. Finding a place on the dialog itself for new customization widgets. On Font, Theme, and General tabs, there is space available either now or with some changes.  The dialog can be enlarged if needed.

5. Adding the behind-the-scenes plumbing.  This is mostly straightforward.

6. Doing minimal refactoring to make better testing easier.
That would include being able to patch in a .idlerc directory or individual user files.  The files are small and could easily fit in memory as StringIOs.

I would like to be able to set user files, open config and idleConf, open config dialog, simulate user actions on the dialog, close, and check the effect on the files.

The first line of the last message should have said 'in addition to' instead of 'rather than', In the following expanded table, Key is the number of pseudoevents with configurable and fixed key bindings.  Gen is the number of General pseudoevents. (Current, from config-main.def.)

File          Menu                            Key  Gen
------------  ------------------------------  ---  ---
autocomplete  Edit - Show Completions         1,2   1 
autoexpand    Edit - Expand Word              1,-   -
calltips      Edit - Show call tip            1,2   -
parenmatch    Edit - Show surrounding parens  1,1   3
paragraph     Format - Format Paragraph       1,-   1
rstrip        Format - Strip trailing white-  -,-   -
runscript     Run - Check Module              2,0   -
runscript     Run - Run Module                same file
codecontext   Options - Code Context          -,-   2
zoomheight    Window - Zoom Height            1,-   -

My initial experiment with adding a fake event and binding in the user config-keys.cfg does not affect existing versions.  It appears to be completely ignored and does not appear on the list of configurable keys in the dialog.

Ned, I reread #20580.  It reminds me that both Control and Alt are problems.  In built-in config-extensions.def, the builtin fixed bindings and one of the configurable bindings are:

autocomplete: <Key-Tab> <KeyRelease-period> <KeyRelease-slash> <KeyRelease-backslash>
calltips: <KeyRelease-parenleft> <KeyRelease-parenright> <KeyRelease-0>
parenmatch: <KeyRelease-parenright> <KeyRelease-bracketright> <KeyRelease-braceright>
runscript: <Key-F5> (configurable)

I presume that these all work fine on Macs.  Every other configurable binding uses 'Control' or 'Alt'.  When I copy configurable bindings into the file, I have to put them in each section.  Should I change each to 'Command' or 'Option' for the 'Mac' and 'Osx' sections?  Also see comment on #20580.

I am adding #27173 as a dependency, for now, because it might be easier to include the new key set before starting to add new bindings.

Ned, from your response on 20580, there appears to be no conversion rule.  Perhap you could go throuch config-main.def and make a list of what you want for Mac and Unix respectively, if different from the current binding.  Serhiy, should anything be different for Linux?, or for your Modern Linux?

Terry, I don't know that much about the history of the config files.  I do know that during framework installs of Python on OS X, the "install_IDLE" recipe in Mac/Makefile.in (which ./configure uses to produce a configured Mac/Makefile) has some editing steps, using sed, to alter the IDLE .def files as they are being installed.  I'm attaching the two edited .def files as installed by the OS X installer.  And then there's the runtime munging of "Alt-" to "Option-" in GetCurrentKeySet() of idlelib/config.py.

I am thinking that we should maybe add an 'Editor' tab.  Or we could expand the dialog box.  More room on General tab can be obtained by removing the label frames, or at least the labels, which seem a bit redundant.

This has to wait for at least some of the refactoring and increased testing of configdialog, and maybe for the switch to ttk.

Progress Update: I've moved all of the basic functionality of the extensions into the regular parts of IDLE, including menus and keyboard shortcuts. parenmatch and codecontext have all of their settings now in the Highlighting settings tab. I have added theme elements for both, as that was both easier and more in-line with what the rest of the IDLE settings were doing than trying to have a separate color-picker for them. The attached image is what it looks like.

What is left for me to do: add the configuration dialogs and proper setting saving for the remaining extensions/modules. Change help texts. Once the configdialog changes patch (#30779) goes through, I will of course, need to fix anything both touch, but that shouldn't be too difficult.

30779 was merged and backported today.

Pull request won't pass build test, but passes the full idle test on my workstation. 

Primary cause seems to be:
ImportError: cannot import name 'EditorWindow' from 'idlelib.editor'

I can also import EditorWindow by itself on my workstation. None of the changes to EditorWindow seem like they should be causing this error. Any ideas?

When I download the patch and run the tests, I get a lot of warnings (mostly on custom themes and custom keysets).  I also get errors in test_parenmatch like this one:

ERROR: test_paren_styles (idlelib.idle_test.test_parenmatch.ParenMatchTest) (style='expression')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/cheryl/cpython/Lib/idlelib/idle_test/test_parenmatch.py", line 64, in test_paren_styles
    pm.flash_paren_event('event')
  File "/home/cheryl/cpython/Lib/idlelib/parenmatch.py", line 98, in flash_paren_event
    self.create_tag(indices)
  File "/home/cheryl/cpython/Lib/idlelib/parenmatch.py", line 154, in create_tag_expression
    self.text.tag_config("paren", self.HILITE_CONFIG)
  File "/home/cheryl/cpython/Lib/tkinter/__init__.py", line 3364, in tag_configure
    return self._configure(('tag', 'configure', tagName), cnf, kw)
  File "/home/cheryl/cpython/Lib/tkinter/__init__.py", line 1470, in _configure
    self.tk.call(_flatten((self._w, cmd)) + self._options(cnf))
_tkinter.TclError: unknown color name "000000"

Thank you,  Cheryl Sabella. I think I've found which tests I've missed running. Now to fix all the bugs...

Pull Request is passing build tests. Please check for pushing to master at your convenience.

I will look at the patch after doing more with config and configdialog.

Changes to master have introduced tests with hardcoded values for what extensions are expected to be loaded by IDLE. Given that this patch turn all current extensions into mainlined modules, none of them are loaded as extensions and all of the related tests fail. What should I do about this?

Either the hard-coding in the config test will have to be changed or it will have to be replaced by something adaptive.  I have not quite decided yet.  Once the new tests are merged, I expect to do some followup improvements.

Equal important for this issue is minimal testing of each extension converted.  They do not have to be complete, but should demonstrate that the extension is present in the menu and basically runs.  Testing for presence in the menu has to be worked out.  There should be one or more separate issues that will be dependencies of this.  Does your patch put menu entries in the same place?  I will want to move some, but that can be a followup issue, which will require change in the test.  Ditto for combining some of the small files into fewer files.

Menus items were placed where they appeared as extensions. It would be no difficulty for me to move them around.

Same place is good for now as testing will be easier.  Moving menu items will be a separate issue later.

I am still thinking about how to handle user customization of key defs and values.

The current extensions tab does not enable custom key defs, so that would be a new feature, and hence could be handled separately, and normally.  I have not yet looked at the patch to see if it does anything in this regard.

I would like to move custom values to config-main.  But if we do that, existing customizations would be ignored and new customization will not be seen by older versions.  A second possibility is to write changes to both config-main and config-extensions, but reading them from both places would be a nuisance.  A third is to keep those options on config-extensions and ignore the enable fields even though idleConf would continue to read them. Whatever we do, we will need additions to config help and a 'see help' message.

Complicating the decision is that the extension tab was first added, two years ago I think, as a separate dialog with a separate system for handling user changes.  When we moved it to a new tab, we did *not* convert it to using the existing system.  Another existing deficiency is that there is no validity check on entries.

Fortunately, the decision of where to display is separate from where to store.  But if displays are moved, the machinery for extension display will have to be changed to ignore values displayed elsewhere.

The patch moves all config variables to config-main, config-highlight (for highlight colors), and config-keys (for keys). Keys and highlights are configurable in their respective tabs. Parens and Code Context options are in the Highlighting tab, Format Paragraph configurables are in the General tab. The sections in the main.def file where the options are saved are respective to their tab.

If you want, I can change the modules so they also load from their old location if an overriding option isn't found in main.def . However, I would recommend cleanly breaking from the old extensions, with a notice somewhere of where the new options are.

Changing the behavior of the extensions Tab / System is a separate issue that I think would be best handled after this, as this patch already makes a lot of changes.

'Clean break' is easy to say. I won't decide yet.  Other opinions? I think I will post to idle-dev also.  Even if no one responds, there will have been the chance.

I don't think a clean break is possible as there's only one set of config files on the system.  So, if someone runs Idle 3.7 (this version) and Idle 2.7, they would probably want their settings to be the same for both since that's how it would currently work.

Unfortunately, it looks like config deletes settings for extensions when they are not found - which is what will happen with this patch. One solution would be to have separate config files for 2 and 3.

There is a reason that I said, in msg266221, that I would experiment with moving 1 feature ;-). I knew from past experience that changing anything that affected user configuration would have to consider the effect on previous releases.  A total 'clean break' now would mean not a Py2 versus Py3 set of files but a Py3.6.2- versus a Py3.6.3+ set of files and I will not do this.

But it is OK with me that you plunged ahead to learn for yourself.  I expect to use most of what you have done.  I would like to get this into 3.6.3, whose release candidate is scheduled for late September.  But see below. That I one reason why I have focused for the last month on getting config and configdialog ready for changes.

When I said, in msg298982, that the extensions tab has 'a separate system for handling user changes', I meant for putting user changes into idleConf.  IdleConf itself handles the 2 sets of 4 files (or the 4 pairs) uniformly.  I intend to remove that separate system *before* working on this.  The changes instance of the new config.OptionChanges is already initialized with an 'extensions' dictionary that corresponds to the idleConf.userCfg['extensions'] instance of IdleUserConfigParser.

So, *do not change your patch to use the separate system*! I expect that we should just need to replace 'main' with 'extensions' in something like "changes.add_option('main', 'AutoComplete', 'popupwait', value)".

Changes that impact users should have a net user benefit, either directly or indirectly.  Moving menu entries, making the menu different in different releases, will bother some people, but I think a few changes will be justified.

Moving the tab where non-key options are displayed and changed also makes successive releases different.  The (potential) benefit is that it will be easier to select and check entries.  I just discovered that one can currently set editor width, for instance, to the string "nonsense"!. So the other tabs are also deficient. The benefit needs to be that entries, both existing and added, *are* checked.  Moving the file where non-key options are *stored* has no user benefit to counter the severe incompatibility problem.  Satisfying my aesthetic sensibilities is not good enough to justify the breakage.

Extension key-binding options are already displayed on the keys tab.  The issue is saving *all* key changes to the key file.  The user benefit is to make it easier to design a new custom keyset.  Some people do this by copying a section into config-keys.cfg or a separate file and editing it in an editor with enough lines to see everything.  This seems easier than laboriously scanning the box showing 10 bindings at a time and changing one at a time with the popup.  But the editor method misses the extension binding.  In any case, even after adding an error check from a 2009 submission, there are 9 remaining keys issues and as many potential issues on my list.  Improving the keys tab for all bindings is as important as moving the storage of some.

Changing existing non-buggy behavior is much more problemmatical than fixing buggy behavior or adding new behavior.  So we could defer the above changes to 3.7.  But I don't like that either.  It deprives 3.6 users, which there will be for several years, of the benefits.  Diverging the files would make backing other improvements to 3.6 harder and less likely.

This issue is worse for the big change to tabbed windows.  Proposed solution: make tabbed windows a 'beta preview' option for the 3.6.x release after a minimal version is ready.  (There would be the option of leaving the option in 3.7, though perhaps as opt-out.) 

The same could be done with the changes in this issue.  Make them opt-in in 3.6.  Add 'Preview' before 'Help' with two items on the drop down: 'Explanation', and 'Selected'.  Explanation would popup a text view window.  Selected would set a variable.

I am working on removing from ConfigDialog a class for each tab.  When that is done, we put the opt-in changes for this issue in a subclass of each.  Then ConfigDialog

The main refactoring of ConfigDialog into multiple classes is done.  Since you kept this patch up to date with the changes, I will review it next, before making further changing to individual tab pages.

On problem with implementing some features as extensions is that associated keybindings are global to all keysets, instead of being specific to one keyset.  This meant that the defaults were necessarily the same for Windows, *nix, and OSX.  (Not necessarily bad, but inflexible.)  This means that any change changes the supposedly fixed built-in keysets and that users cannot try out different bindings in different custom keysets.  This strikes me a  something of a bug, so I am looking more favorably at moving all keybindings (other than true, 3rd-party extensions) to config-keys.def/cfg.

I am not sure I understand your concern. As part of mainlining the extensions, all of their keybindings were moved to config-keys.def . There is nothing to prevent users from changing the keybindings or preventing us from making the defaults different across the different default key sets.

Before reading the patch, more thoughts on the features, options, and behavior: nothing is Shell only; codecontext, rstrip, and runscript are properly limited to editors; the others are enabled for both.  I think paragraph could have been editor only, but I would not enforce that now.  I want current enable defaults to be build-in, with needed disabling handled otherwise.

autocomplete: popupwait is needed, a large value will disable.  (In root.after(n, func), a negative delay is the same as no delay.)

codecontext: has multiple bugs, including the usage of 'visible', which should no longer be used; see *22703.  I think the actual number of lines should be variable, up to numlines.  Implementing that would be another issue, but numlines = 0 should disable the feature.  (This is interpreted the same as numlines=1 in current IDLE when codecontext is enabled.)  The menu entry should continue to toggle code context for the current window, but should not set 'visible'.

Some sort of distinct highlighting is needed.

Adding new shortcuts, such as for code-context, is not part of this issue.  There are other menu entries, such as 'Recent files' that need a shortcut just as much or more.  This should be a separate issue.

paragraph: max_width is needed.

parenmatch: 'style' is needed. I presume some timer is.  I am not sure why this particular instance of an audible signal for error (as opposed to SyntaxError, for instance) is needed.

Add a new, distinct highlighting for paren matching should be a separate issue

zoomheight: useful when it works right, but more or less useless on Windows as it zooms under the taskbar, obscuring the status bar.  How about on other systems?

On Windows 10, pulling the top of the window up to the top of the screen or the bottom down to (but not under) the taskbar zooms the opposite border. Pulling either border away pulls the opposite away also.

My 'concern' was in relation to the old situation and why I have become more willing to change default behavior.

ZoomHeight has been put back into being an extension. This was because configdialog assumes autoexpand was an extension, and the idleconf tests all need at least one extension to exist to be useful. Zoomheight is pointless in almost any modern desktop environment, but it's very useful as an example extension and as a model extension for tests.

I'm going to fix autocomplete and paragraph options. I had their options in the general tab in an earlier build, but it looks like I accidentally reverted it. The rest of the options are in the highlights tab. The bell was in the extension before I got to it. Fixing that and whatever bugs are in codecontext seems outside the scope of this patch.

zoomheight is OK on Ubuntu.

The transferred options belong on the General tab.  I have already planned to condense what is there as part of #31051.  I held off because I did not know if it would conflict with your patch.  Since, as far as I can tell, your patch currently does not touch GenPage, I will change it tomorrow to make room for new stuff and think about what layout I would prefer at least for the present.

Configdialog does not import EditorWindow and should not need to with the change.  It should also not import the separate modules.  I outlined how I plan to update the current extension values in #22707, msg300973.

PR3239 for #31051 prepares GenPage for the transferred options.  msg300973 specifies which goes where on the page.

I believe PR2494 is about ready to merge.  It is a big, complicated patch, so before merging, I would prefer to have it pass a full -uGUI test on something other than Windows.

I expect to polish the General page in a follow-up.  I expect it will soon need to be split.

The test pass on Ubuntu.  I'm getting a warning, but it's not because of this change.

test_init (idlelib.idle_test.test_tree.TreeTest) ... invalid command name "140245621847896timer_event"
    while executing
"140245621847896timer_event"
    ("after" script)
invalid command name "140245613968200font_timer_event"
    while executing
"140245613968200font_timer_event"
    ("after" script)

The warning is from code context.  The callback after loops run whether one uses context or not and there is currently no mechanism to stop them.  I will add a cancel function soon.  Or the loops might be eliminated by adding context to the config change notification mechanism.

New changeset 58fc71c447049d0efe4e11db1b55edc307f1bede by Terry Jan Reedy (wohlganger) in branch 'master':
bpo-27099: IDLE - Convert built-in extensions to regular features (#2494)
https://github.com/python/cpython/commit/58fc71c447049d0efe4e11db1b55edc307f1bede

New changeset d6c397bf772a8f17e00afc5e0a2cf37fdebcdf29 by Terry Jan Reedy in branch '3.6':
[3.6] bpo-27099: IDLE - Convert built-in extensions to regular features (GH-2494) (#3487)
https://github.com/python/cpython/commit/d6c397bf772a8f17e00afc5e0a2cf37fdebcdf29

Here is a high-level overview of the steps involved in the patch, and a few TODOs for new issues.

Move menu specifications from feature files to mainmenu.py.  Remove enable items from config-extensions.def sections.  Most features were enabled for both shell and editor. FormatParagraph appears on the Format menu which is only present for editor windows.  Ditto for Check Module and Run Module and the Run menu. Disable CodeText in Outwin.  TODO: Also grey-out the menu item.

Handle user-configurable pseudoevent -- key sequence definitions the same for these features as all others.  This fixes oddities that could be considered bugs.  Delete them from config-extensions.def sections.  Add them to the default keysets in config-keys.def, with adjustments for the Mac keysets.  Let user changes be stored in custom keysets in config-keys.cfg instead of diverting them to config-extensions.cfg.  Do not warn when they are missing from older custom keysets (when they get default values).  TODO: Augment config.IdleConf to add new pairs to existing custom keysets, at least when they are used.

I have tested that 2.7 runs with an augmented custom keyset, ignoring the new keys, and does not delete the new definitions when an older one is changed and the keyset changed.  TODO: Fix regressions in keyset handling that predate this patch and were not caught by existing tests.

Handle fixed pseudoevent -- key sequence pairs more like existing non-extension pairs.  Delete them from config-extensions.def.  For the present, put event_add() calls in EditorWindow.__init__.  TODO: If I am correct that these calls are needed just once and not for each instance, move them.  (Not an immediate priority.)

Remove sections that are now empty, leaving 4 with non-key options.  Add widgets to the General Page of ConfigDialog for customizing these options.  Make the option values class attributes in the respective classes, initialize by a class method that is called upon import and when config dialog changes are accepted.  (This means that changes now take effect immediately rather than when IDLE is started.)  For the present, for back-compatibility, leave these sections in config-extensions.def rather than moving them to config-main.def (and moving user values between the corresponding .cfg files.)  Add a note in config-extensions.def explaining that these sections do not represent extensions but why there are there.  Modify IdleConf to not treat these as extension sections.  TODO: re-evaluate whether some of the options should really be kept as options.

Add a dummy extension Zzdummy to serve as an example and for use in tests.  TODO: Modify test so that Zzdummy can be disabled by default.

Anything else that I forgot or is too trivial to mention ;-).

Charles, thank you for the work.  This was pretty tough for both of us.  My takeaway: getting configdialog nearly covered first was essential; even better tests for configdialog and other modules would have helped.

New changeset adb4cd2a2a59019ac6955e0fd531c9fec9258962 by Terry Jan Reedy in branch 'master':
bpo-27099: Finish updating IDLE doc and help text. (#3510)
https://github.com/python/cpython/commit/adb4cd2a2a59019ac6955e0fd531c9fec9258962

New changeset 01dcaa5c996baf019656cf51451bb1b6ecd720fa by Terry Jan Reedy (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-27099: Finish updating IDLE doc and help text. (GH-3510) (#3511)
https://github.com/python/cpython/commit/01dcaa5c996baf019656cf51451bb1b6ecd720fa

Two of the TODOs already listed, and 2 more found in review, have been done.  With these additions, I felt IDLE was ready for the 3.6.3rc1 and 3.7.0a1 releases last Monday.

#31502 Fix problem with deleting user themes and keysets.
#31480 Disable ZzDummy by default, change tests to match.
#31477 Improve rstrip entry in IDLE doc.
#31488 Make non-key options take affect immediately.  (Parenmatch required revision for this to happen for its option in existing windows.  The options were previously 'baked in' in the __init__ method.)

The following remaining to be done:
Grey-out code context toggle on non-editor windows.
#31547 Write augmented user keysets back to file.
Call each event_add(pseudoevent, sequence) just once.
Re-evaluate some of options.

I don't think this should have been backported to 3.6.

This week, I'm teaching a class in Python3.6 and tab completion seems to be broken for everyone from a fresh install of 3.6.4 directly from www.python.org.  Also, the extension tab now shows ZzDummy.

Leaving this as closed.  Just wanted to note that it was a rough upgrade to 3.6.4 because of this backport.

These patches were first released with the 3.6.3 release candidate.  Nothing was changed for 3.6.4.  Rechecking the changes to autocomplete.py in PR 2494, the only expected effect, after the followup in #31488, PR 3612, is that changes to the popup wait time should take effect immediately instead of sometime later.  I considered making this happen for the 'extension' features, as with with other features, a type of bugfix.

If you have an issue with completions, other than those listed in #27609, please open a new issue.

I know zzdummy is listed.  An explanation that it is there for testing and example should have been added to the [Help] buttom text for the extension tab.  I opened #32631 for this and possible changes to zzdummy.py.