Title: IDLE: turn builting extensions into regular modules
Type: behavior Stage: test needed
Components: IDLE Versions: Python 3.7, Python 3.6
Status: open Resolution:
Dependencies: 24225 27173 30779 30780 30781 Superseder:
Assigned To: terry.reedy Nosy List: csabella, louielu, ned.deily, serhiy.storchaka, terry.reedy, wohlganger
Priority: normal Keywords:

Created on 2016-05-24 02:02 by terry.reedy, last changed 2017-07-25 22:01 by terry.reedy.

File name Uploaded Description Edit
config-extensions.def ned.deily, 2016-06-14 04:47
config-main.def ned.deily, 2016-06-14 04:47
2017-07-07 13_58_45-Settings.png wohlganger, 2017-07-07 19:18 setting screenshot - highlight tab.
Pull Requests
URL Status Linked Edit
PR 2494 open wohlganger, 2017-07-10 21:15
Messages (27)
msg266214 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-05-24 02:02
Followup to #24225, which renamed idlelib files.

Currently, an arbitrary set of 9 (of 10, if Check Module is counted as a separate feature) IDLE features are implemented as extensions.  Their menu names are as follows:
  Edit - Show Completions
  Edit - Expand Word
  Edit - Show call tip
  Edit - Show surrounding parens
  Format - Format Paragraph
  Format - Strip trailing whitespace
  Run - Check Module (same file as below)
  Run - Run Module
  Options - Code Context
  Window - Zoom Height

This issue proposes to turn these into normal features implemented in regular code. 

1. The menu entries for extensions are not included in the main menu definition file (mainmenu, previously Bindings).  Rather, they are appended when the file is read.  This is constricting.  The order of the 4 edit entries is the alphabetical order of the corresponding file names. Run Module should be at the top of the Run menu, rather than being shoved down further if and when more non-extension entries are added.

2. Each feature is a separate file, usually small to medium with one class.  This bloats the number of modules in idlelib (now about 60).  I am considering combining autoexpand and parenmatch, paragraph and rstrip, and codecontext and zoomheight (and moving codecontext to the Window menu).  They might be joined with other features from the same menu.

3. The user customization system is not very compatible with change.  The problems with extension customization and changing file names is discussed in the second half of msg266213.  If extensions are not converted, another solution will be needed.

I have not yet tested the effect of adding new key bindings to config-keys.def and config-keys.cfg.  If this causes a problem for current releases, then this is a problem we need to face.  I anticipate wanting to add other new keybindings, such as one to switch tabs when we use a notebook.
msg266221 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-05-24 03:39
I don't have an opinion on the matter, other than a reminder about the current cross-version (and cross-platform) compatibility issues with a user's IDLE configuration files as alluded to in Issue20580.  Perhaps others more familiar with IDLE's internals would have comments?
msg266778 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-05-31 20:53
The version conflict problem is why I reverted changing 'extension names' to match the new file names, before rebase collapsing my patches.  I was really glad most were in a separate patch.

Each time a feature ceases to be an extension, the (old) extension name will have to be added to a set of names ignored when calculating the set of extensions.  I will start with one until I have worked out and tested the details.
msg267575 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-06-07 00:44
The final paragraph of my initial post should have talked about config-main and features, rather than keys.

The following experiment indicates that adding new sections to config-main.def and customizations thereof to config-main.cfg should not be a problem. 

Add the following to config-main.cfg
new = True

Open IDLE 2.7 from console (python, with import).
Open config dialog.
Change item.
Close with [OK]
confirm that change is written and new section is left alone.
Close IDLE.
Repeat with 3.5.

Old versions will not see new config-main.def, so that is not an issue.

Conclusion: adding a new section that old versions ignore is much safer then adding a new value for a current section&item that old versions do read and act on, and which may refer to something that does not exist.  (This was the problem with IDLE Dark theme and would be with Unix New keyset).

The remaining issues:

4. Finding a place on the dialog itself for new customization widgets. On Font, Theme, and General tabs, there is space available either now or with some changes.  The dialog can be enlarged if needed.

5. Adding the behind-the-scenes plumbing.  This is mostly straightforward.

6. Doing minimal refactoring to make better testing easier.
That would include being able to patch in a .idlerc directory or individual user files.  The files are small and could easily fit in memory as StringIOs.

I would like to be able to set user files, open config and idleConf, open config dialog, simulate user actions on the dialog, close, and check the effect on the files.
msg268479 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-06-13 22:09
The first line of the last message should have said 'in addition to' instead of 'rather than', In the following expanded table, Key is the number of pseudoevents with configurable and fixed key bindings.  Gen is the number of General pseudoevents. (Current, from config-main.def.)

File          Menu                            Key  Gen
------------  ------------------------------  ---  ---
autocomplete  Edit - Show Completions         1,2   1 
autoexpand    Edit - Expand Word              1,-   -
calltips      Edit - Show call tip            1,2   -
parenmatch    Edit - Show surrounding parens  1,1   3
paragraph     Format - Format Paragraph       1,-   1
rstrip        Format - Strip trailing white-  -,-   -
runscript     Run - Check Module              2,0   -
runscript     Run - Run Module                same file
codecontext   Options - Code Context          -,-   2
zoomheight    Window - Zoom Height            1,-   -

My initial experiment with adding a fake event and binding in the user config-keys.cfg does not affect existing versions.  It appears to be completely ignored and does not appear on the list of configurable keys in the dialog.
msg268481 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-06-13 22:32
Ned, I reread #20580.  It reminds me that both Control and Alt are problems.  In built-in config-extensions.def, the builtin fixed bindings and one of the configurable bindings are:

autocomplete: <Key-Tab> <KeyRelease-period> <KeyRelease-slash> <KeyRelease-backslash>
calltips: <KeyRelease-parenleft> <KeyRelease-parenright> <KeyRelease-0>
parenmatch: <KeyRelease-parenright> <KeyRelease-bracketright> <KeyRelease-braceright>
runscript: <Key-F5> (configurable)

I presume that these all work fine on Macs.  Every other configurable binding uses 'Control' or 'Alt'.  When I copy configurable bindings into the file, I have to put them in each section.  Should I change each to 'Command' or 'Option' for the 'Mac' and 'Osx' sections?  Also see comment on #20580.

I am adding #27173 as a dependency, for now, because it might be easier to include the new key set before starting to add new bindings.
msg268511 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-06-14 04:25
Ned, from your response on 20580, there appears to be no conversion rule.  Perhap you could go throuch config-main.def and make a list of what you want for Mac and Unix respectively, if different from the current binding.  Serhiy, should anything be different for Linux?, or for your Modern Linux?
msg268517 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-06-14 04:47
Terry, I don't know that much about the history of the config files.  I do know that during framework installs of Python on OS X, the "install_IDLE" recipe in Mac/ (which ./configure uses to produce a configured Mac/Makefile) has some editing steps, using sed, to alter the IDLE .def files as they are being installed.  I'm attaching the two edited .def files as installed by the OS X installer.  And then there's the runtime munging of "Alt-" to "Option-" in GetCurrentKeySet() of idlelib/
msg297450 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-30 22:34
I am thinking that we should maybe add an 'Editor' tab.  Or we could expand the dialog box.  More room on General tab can be obtained by removing the label frames, or at least the labels, which seem a bit redundant.
msg297451 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-30 22:38
This has to wait for at least some of the refactoring and increased testing of configdialog, and maybe for the switch to ttk.
msg297904 - (view) Author: Charles Wohlganger (wohlganger) * Date: 2017-07-07 19:18
Progress Update: I've moved all of the basic functionality of the extensions into the regular parts of IDLE, including menus and keyboard shortcuts. parenmatch and codecontext have all of their settings now in the Highlighting settings tab. I have added theme elements for both, as that was both easier and more in-line with what the rest of the IDLE settings were doing than trying to have a separate color-picker for them. The attached image is what it looks like.

What is left for me to do: add the configuration dialogs and proper setting saving for the remaining extensions/modules. Change help texts. Once the configdialog changes patch (#30779) goes through, I will of course, need to fix anything both touch, but that shouldn't be too difficult.
msg297918 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-08 00:04
30779 was merged and backported today.
msg298160 - (view) Author: Charles Wohlganger (wohlganger) * Date: 2017-07-11 13:48
Pull request won't pass build test, but passes the full idle test on my workstation. 

Primary cause seems to be:
ImportError: cannot import name 'EditorWindow' from 'idlelib.editor'

I can also import EditorWindow by itself on my workstation. None of the changes to EditorWindow seem like they should be causing this error. Any ideas?
msg298176 - (view) Author: Cheryl Sabella (csabella) * Date: 2017-07-11 18:54
When I download the patch and run the tests, I get a lot of warnings (mostly on custom themes and custom keysets).  I also get errors in test_parenmatch like this one:

ERROR: test_paren_styles (idlelib.idle_test.test_parenmatch.ParenMatchTest) (style='expression')
Traceback (most recent call last):
  File "/home/cheryl/cpython/Lib/idlelib/idle_test/", line 64, in test_paren_styles
  File "/home/cheryl/cpython/Lib/idlelib/", line 98, in flash_paren_event
  File "/home/cheryl/cpython/Lib/idlelib/", line 154, in create_tag_expression
    self.text.tag_config("paren", self.HILITE_CONFIG)
  File "/home/cheryl/cpython/Lib/tkinter/", line 3364, in tag_configure
    return self._configure(('tag', 'configure', tagName), cnf, kw)
  File "/home/cheryl/cpython/Lib/tkinter/", line 1470, in _configure, cmd)) + self._options(cnf))
_tkinter.TclError: unknown color name "000000"
msg298183 - (view) Author: Charles Wohlganger (wohlganger) * Date: 2017-07-11 20:57
Thank you,  Cheryl Sabella. I think I've found which tests I've missed running. Now to fix all the bugs...
msg298225 - (view) Author: Charles Wohlganger (wohlganger) * Date: 2017-07-12 15:17
Pull Request is passing build tests. Please check for pushing to master at your convenience.
msg298253 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-13 01:05
I will look at the patch after doing more with config and configdialog.
msg298687 - (view) Author: Charles Wohlganger (wohlganger) * Date: 2017-07-19 20:12
Changes to master have introduced tests with hardcoded values for what extensions are expected to be loaded by IDLE. Given that this patch turn all current extensions into mainlined modules, none of them are loaded as extensions and all of the related tests fail. What should I do about this?
msg298690 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-19 21:54
Either the hard-coding in the config test will have to be changed or it will have to be replaced by something adaptive.  I have not quite decided yet.  Once the new tests are merged, I expect to do some followup improvements.

Equal important for this issue is minimal testing of each extension converted.  They do not have to be complete, but should demonstrate that the extension is present in the menu and basically runs.  Testing for presence in the menu has to be worked out.  There should be one or more separate issues that will be dependencies of this.  Does your patch put menu entries in the same place?  I will want to move some, but that can be a followup issue, which will require change in the test.  Ditto for combining some of the small files into fewer files.
msg298723 - (view) Author: Charles Wohlganger (wohlganger) * Date: 2017-07-20 14:12
Menus items were placed where they appeared as extensions. It would be no difficulty for me to move them around.
msg298740 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-20 17:23
Same place is good for now as testing will be easier.  Moving menu items will be a separate issue later.
msg298982 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-24 15:33
I am still thinking about how to handle user customization of key defs and values.

The current extensions tab does not enable custom key defs, so that would be a new feature, and hence could be handled separately, and normally.  I have not yet looked at the patch to see if it does anything in this regard.

I would like to move custom values to config-main.  But if we do that, existing customizations would be ignored and new customization will not be seen by older versions.  A second possibility is to write changes to both config-main and config-extensions, but reading them from both places would be a nuisance.  A third is to keep those options on config-extensions and ignore the enable fields even though idleConf would continue to read them. Whatever we do, we will need additions to config help and a 'see help' message.

Complicating the decision is that the extension tab was first added, two years ago I think, as a separate dialog with a separate system for handling user changes.  When we moved it to a new tab, we did *not* convert it to using the existing system.  Another existing deficiency is that there is no validity check on entries.

Fortunately, the decision of where to display is separate from where to store.  But if displays are moved, the machinery for extension display will have to be changed to ignore values displayed elsewhere.
msg299011 - (view) Author: Charles Wohlganger (wohlganger) * Date: 2017-07-24 18:15
The patch moves all config variables to config-main, config-highlight (for highlight colors), and config-keys (for keys). Keys and highlights are configurable in their respective tabs. Parens and Code Context options are in the Highlighting tab, Format Paragraph configurables are in the General tab. The sections in the main.def file where the options are saved are respective to their tab.

If you want, I can change the modules so they also load from their old location if an overriding option isn't found in main.def . However, I would recommend cleanly breaking from the old extensions, with a notice somewhere of where the new options are.

Changing the behavior of the extensions Tab / System is a separate issue that I think would be best handled after this, as this patch already makes a lot of changes.
msg299015 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-24 20:00
'Clean break' is easy to say. I won't decide yet.  Other opinions? I think I will post to idle-dev also.  Even if no one responds, there will have been the chance.
msg299025 - (view) Author: Cheryl Sabella (csabella) * Date: 2017-07-24 22:34
I don't think a clean break is possible as there's only one set of config files on the system.  So, if someone runs Idle 3.7 (this version) and Idle 2.7, they would probably want their settings to be the same for both since that's how it would currently work.
msg299074 - (view) Author: Charles Wohlganger (wohlganger) * Date: 2017-07-25 13:12
Unfortunately, it looks like config deletes settings for extensions when they are not found - which is what will happen with this patch. One solution would be to have separate config files for 2 and 3.
msg299153 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-25 22:01
There is a reason that I said, in msg266221, that I would experiment with moving 1 feature ;-). I knew from past experience that changing anything that affected user configuration would have to consider the effect on previous releases.  A total 'clean break' now would mean not a Py2 versus Py3 set of files but a Py3.6.2- versus a Py3.6.3+ set of files and I will not do this.

But it is OK with me that you plunged ahead to learn for yourself.  I expect to use most of what you have done.  I would like to get this into 3.6.3, whose release candidate is scheduled for late September.  But see below. That I one reason why I have focused for the last month on getting config and configdialog ready for changes.

When I said, in msg298982, that the extensions tab has 'a separate system for handling user changes', I meant for putting user changes into idleConf.  IdleConf itself handles the 2 sets of 4 files (or the 4 pairs) uniformly.  I intend to remove that separate system *before* working on this.  The changes instance of the new config.OptionChanges is already initialized with an 'extensions' dictionary that corresponds to the idleConf.userCfg['extensions'] instance of IdleUserConfigParser.

So, *do not change your patch to use the separate system*! I expect that we should just need to replace 'main' with 'extensions' in something like "changes.add_option('main', 'AutoComplete', 'popupwait', value)".

Changes that impact users should have a net user benefit, either directly or indirectly.  Moving menu entries, making the menu different in different releases, will bother some people, but I think a few changes will be justified.

Moving the tab where non-key options are displayed and changed also makes successive releases different.  The (potential) benefit is that it will be easier to select and check entries.  I just discovered that one can currently set editor width, for instance, to the string "nonsense"!. So the other tabs are also deficient. The benefit needs to be that entries, both existing and added, *are* checked.  Moving the file where non-key options are *stored* has no user benefit to counter the severe incompatibility problem.  Satisfying my aesthetic sensibilities is not good enough to justify the breakage.

Extension key-binding options are already displayed on the keys tab.  The issue is saving *all* key changes to the key file.  The user benefit is to make it easier to design a new custom keyset.  Some people do this by copying a section into config-keys.cfg or a separate file and editing it in an editor with enough lines to see everything.  This seems easier than laboriously scanning the box showing 10 bindings at a time and changing one at a time with the popup.  But the editor method misses the extension binding.  In any case, even after adding an error check from a 2009 submission, there are 9 remaining keys issues and as many potential issues on my list.  Improving the keys tab for all bindings is as important as moving the storage of some.

Changing existing non-buggy behavior is much more problemmatical than fixing buggy behavior or adding new behavior.  So we could defer the above changes to 3.7.  But I don't like that either.  It deprives 3.6 users, which there will be for several years, of the benefits.  Diverging the files would make backing other improvements to 3.6 harder and less likely.

This issue is worse for the big change to tabbed windows.  Proposed solution: make tabbed windows a 'beta preview' option for the 3.6.x release after a minimal version is ready.  (There would be the option of leaving the option in 3.7, though perhaps as opt-out.) 

The same could be done with the changes in this issue.  Make them opt-in in 3.6.  Add 'Preview' before 'Help' with two items on the drop down: 'Explanation', and 'Selected'.  Explanation would popup a text view window.  Selected would set a variable.

I am working on removing from ConfigDialog a class for each tab.  When that is done, we put the opt-in changes for this issue in a subclass of each.  Then ConfigDialog
Date User Action Args
2017-07-25 22:01:16terry.reedysetnosy: + louielu
messages: + msg299153
2017-07-25 13:12:32wohlgangersetmessages: + msg299074
2017-07-24 22:34:37csabellasetmessages: + msg299025
2017-07-24 20:00:53terry.reedysetmessages: + msg299015
2017-07-24 18:15:24wohlgangersetmessages: + msg299011
2017-07-24 15:33:00terry.reedysetmessages: + msg298982
2017-07-20 17:23:37terry.reedysetmessages: + msg298740
2017-07-20 14:12:44wohlgangersetmessages: + msg298723
2017-07-19 21:54:03terry.reedysetmessages: + msg298690
2017-07-19 20:12:00wohlgangersetmessages: + msg298687
2017-07-13 01:05:46terry.reedysetmessages: + msg298253
2017-07-12 15:17:25wohlgangersetmessages: + msg298225
2017-07-11 20:57:21wohlgangersetmessages: + msg298183
2017-07-11 18:54:31csabellasetnosy: + csabella
messages: + msg298176
2017-07-11 13:48:44wohlgangersetmessages: + msg298160
2017-07-10 21:15:32wohlgangersetpull_requests: + pull_request2718
2017-07-08 00:04:09terry.reedysetmessages: + msg297918
2017-07-07 19:18:05wohlgangersetfiles: + 2017-07-07 13_58_45-Settings.png
nosy: + wohlganger
messages: + msg297904

2017-06-30 22:38:39terry.reedylinkissue30809 dependencies
2017-06-30 22:38:18terry.reedysetdependencies: + IDLE: configdialog -- factor out Changes class, IDLE: configdialog - add tests for ConfigDialog GUI., IDLE: configdialog -- switch to ttk widgets.
messages: + msg297451
2017-06-30 22:34:08terry.reedysetmessages: + msg297450
versions: + Python 3.7
2017-06-19 18:46:22terry.reedysetcomponents: + IDLE
2016-07-25 00:08:07terry.reedylinkissue27609 dependencies
2016-06-14 04:47:44ned.deilysetfiles: + config-main.def
2016-06-14 04:47:09ned.deilysetfiles: + config-extensions.def

messages: + msg268517
2016-06-14 04:25:03terry.reedysetmessages: + msg268511
2016-06-13 22:32:34terry.reedysetdependencies: + Modern Unix key bindings for IDLE
messages: + msg268481
2016-06-13 22:09:44terry.reedysetmessages: + msg268479
2016-06-07 00:44:46terry.reedysetnosy: + serhiy.storchaka
messages: + msg267575
2016-05-31 20:53:43terry.reedysetmessages: + msg266778
2016-05-24 03:39:50ned.deilysetmessages: + msg266221
2016-05-24 02:04:11terry.reedysetdependencies: + Idlelib: changing file names
2016-05-24 02:02:40terry.reedycreate