Title: IDLE: Fix tab completion after settings and some keys
Type: behavior Stage: patch review
Components: IDLE Versions: Python 3.11, Python 3.10, Python 3.9
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: gvanrossum, rhettinger, taleinat, terry.reedy, tetelevm
Priority: normal Keywords: patch

Created on 2021-03-29 08:40 by terry.reedy, last changed 2021-06-06 07:59 by taleinat.

Pull Requests
URL Status Linked Edit
PR 26403 closed taleinat, 2021-05-27 14:59
PR 26421 open taleinat, 2021-05-28 06:31
Messages (12)
msg389673 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-03-29 08:40
(Original report by Mikhail on #43647, running 3.9 on Linux; verified and extended by me running 3.10 on Windows.)

Normally, "i<tab>" brings up a completion window with 'id', 'if', 'import', etc.  Opening a Settings windows with Options => Configure IDLE and closing with Apply and Cancel or OK (which also applies) disables tab completion.  Other completions (forced with ^<space> or auto with '.' or '/' and waits seem not affected.  The only way to restore is to close and reopen each window.

Tab completions are enabled in with these two lines.
        text.event_add('<<autocomplete>>', '<Key-Tab>')
        text.bind("<<autocomplete>>", autocomplete.autocomplete_event) 
Attribute and path completions, not affected, are enabled with these.
        text.event_add('<<try-open-completions>>', '<KeyRelease-period>',
                       '<KeyRelease-slash>', '<KeyRelease-backslash>
Similarly for some other things.

In configdialog, the relevant method is (179) def apply, who relevant calls are (219) deactivate_current_config and (230) activate_current_config.  The former removes key bindings and the latter rebinds and makes other changes.

What is different about Tab versus '.' is that is tab also used for indents and the indent space is reset by 'activate...'.  I will later add some debug prints to console based on the clues above.
msg389791 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-03-30 05:04
Investigation of this issue is complicated by the fact that the editor test widgets are wrapped by multicall.MulticallCreator.  It intercepts bind and event method calls (other than event_generate) for user pseudoevents.  It keeps its own map of pseudoevent names to handler and key sequences instead of passing the information on to tk.

The custom entry for "<<autocomplete>>" is the same,
 [<bound method AutoComplete.autocomplete_event of <idlelib.autocomplete.AutoComplete object at 0x00000209DA4FBD20>>, [(0, 0, 'Tab')]]
both before deactivate_... and after activate_... .

I determined this by adding calls like the following
          print('ei5 ', instance.text.event_info('<<autocomplete>>'))
and adding the following to the event_info(virtual) override.
          print('mei', virtual, self.__eventinfo.get(virtual))

I will next look at the custom binding data that should map keys to event handlers.
msg390731 - (view) Author: Mikhail (tetelevm) * Date: 2021-04-10 18:50
Hello again!
I found another kind of bug, where the autosupplement window doesn't show up. 
In order for the bug to occur, you need to put a non-string value into `locals()`.
locals()['arg'] = 123  # or locals().setdefault('arg', 123)
# ^ it's okay

locals()[123] = 123  # or locals().setdefault(123, 123)
# ^ completion window is not shown
Of course, I know that "Whether or not updates to this dictionary will affect name lookups in the local scope and vice-versa not covered by any backwards compatibility guarantees" , but take a little bug :)

Here is the traceback of the error
Exception in Tkinter callback
Traceback (most recent call last):
  File "/usr/lib/python3.9/tkinter/", line 1885, in __call__
    return self.func(*args)
  File "/usr/lib/python3.9/idlelib/", line 176, in handler
    r = l[i](event)
  File "/usr/lib/python3.9/idlelib/", line 74, in autocomplete_event
    opened = self.open_completions(TAB)
  File "/usr/lib/python3.9/idlelib/", line 146, in open_completions
    comp_lists = self.fetch_completions(comp_what, mode)
  File "/usr/lib/python3.9/idlelib/", line 171, in fetch_completions
    return rpcclt.remotecall("exec", "get_the_completion_list",
  File "/usr/lib/python3.9/idlelib/", line 219, in remotecall
    return self.asyncreturn(seq)
  File "/usr/lib/python3.9/idlelib/", line 250, in asyncreturn
    return self.decoderesponse(response)
  File "/usr/lib/python3.9/idlelib/", line 270, in decoderesponse
    raise what
TypeError: '<' not supported between instances of 'int' and 'str'
msg390732 - (view) Author: Mikhail (tetelevm) * Date: 2021-04-10 19:05
I did a little experiment, and it turned out that the problem is not in the IDLE, but inside Python itself. With this case, the same behavior remains inside the terminal version of Python, and IPython also produces a similar error.
msg390755 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-04-11 00:22
I presume '^' means 'hit Tab'.

The underlying Python problem is that namespaces are conceptually mappings of names (identifiers), a subset of strings, to objects.  Python specifies that the global namespace is a dict, which I believe could mean a dict subclass restricting keys to strings or maybe even identifier strings.  But CPython, at least, uses built-in dicts, allowing any hashable as a key.  Ditto for attribute dicts and locals.  At toplevel, locals() is globals().

(There are a few semi-same uses to non-identifier string completions.
>>> globals()["3.1459"] = None
>>> 3.1459 # 3<tab>
>>> globals()['itertools.permutations'] = None
>>> itertools.permutations # itert<tab>
But not recommended or, I believe, guaranteed.)

Completion lists are sorted.  In original Python, sorting was universal.  That was broken for a few types by 2.7.  In 3.x, comparisons and hence sorting were restricted.

Decoding the traceback.  Completion requests are handled in the IDLE interface process.  The remotecall in autocomplete.fetch_completions tries to execute fetch_completions in the user execution process.  That fails trying to sort the list of globals.  The exception is returned with the tag "CALLEXC".  In rpc, decoderesponse filters out various failures and raises the returned exception.

IDLE could filter out non-strings from global and attribute dicts or display a special message.  Or do nothing on the basis that putting non-strings in namespaces and making them unsortable is either a user bug or a consenting adults, do at your own risk and deal with the consequences operation.

Built-in dir() sorts keys and hence it and anything using it fails.

>>> globals()[1]=None
>>> dir()
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
TypeError: '<' not supported between instances of 'int' and 'str'

Indeed, fetch_completions uses dir(), and this is where it fails.  

Vars() does not sort, allowing one to see the full contents.

>>> vars()
{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, 1: None}

If one edits a file after putting a non-string in globals, the TypeError is not displayed.  Instead, tab completion is silently disabled and spaces are inserted, just as after applying settings.  This is not good and so a point in favor of doing something.
msg390824 - (view) Author: Mikhail (tetelevm) * Date: 2021-04-12 08:27
I also checked this behavior in Jython and IPython, and there errors are explicitly caused when putting/calling such values (I also checked on MicroPython online version, everything just hangs there). For IPython I created an issue on their Github, I'm thinking of doing the same for Jython.
Since you're my guide to the world of bugs and issues for CPython, any advice on whether to create a separate issue here or leave it as is?

P.S. I like the Python community, everyone here is kind and helpful :)
msg394542 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2021-05-27 14:04
The original issue appears to be caused by <<smart-indent>> also being bound to Tab, but <<autocomplete>> only being bound using event_add() once in EditorWindow.__init__().  Therefore, RemoveKeybindings() does remove the binding for tab due to <<smart-indent>>, but <<autocomplete>> is not bound again by ApplyKeybindings().

That should mean that tab is still bound to <<autocomplete>> and <<smart-indent>>, as expected, but the order has changed: now <<smart-indent>> was the last event added. We need <<autocomplete>> to fire first.

Specifically, commenting out the line for <<smart-indent>> from causes this bug to no longer happen.
msg394545 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2021-05-27 15:05
Indeed, adding the events bound in EditorWindow.__init__ to RemoveKeybindings() and ApplyKeybindings(), so they are bound in the correct order, resolves this issue.

See PR GH-26403.
msg394631 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-05-28 04:40
Tal, I think you had the right solution the first time.  To me, tab is (should be) a fixed key, and the event added just once, like the other fixed keys.  I am quite surprised that it isn't.  Please revise or make a new patch to remove it from both configkeys.def and

Even if I agreed with unfixing other keys, only <<autocomplete>> would need rebinding, not an arbitrary subset of the fixed keys.  We mostly did away with treating some features differently; what remains was mostly an expedient to get a too-big patch merged.  I meant and still mean to finish the job.
msg394883 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-06-02 01:40
Currently, the keys that define certain pseudoevents and invoke the associated event handlers are fixed: tab, '.', and within strings, '/' and '\' for completions, '(' and ')' for calltip open and close, and ')', ']', and '}' for opener matching.  Note that quote matching is indicated by coloring completed strings.

PR-26421 proposes to augment this list with tab for smart indent, backspace for smart backspace, and newline for completing a line and maybe smart indenting. In other words, remove the following 3 lines

  '<<smart-backspace>>': ['<Key-BackSpace>'],
  '<<newline-and-indent>>': ['<Key-Return>', '<Key-KP_Enter>'],
  '<<smart-indent>>': ['<Key-Tab>'],

from config-keys.def and the Settings Keys tab and add them as fixed binding to EditorWindow.__init__ just above the existing fixed pseudoevent -- keys bindings.

Only fixing smart-indent and tab (or unfixing name-completion and tab) is needed to keep name completion working after re-doing setting.  So why all three?  1. These three pairs go together; I don't see anything else that should be fixed.  2. By the standard used to fix some pairs already, these three should be also.  I think it an accident of IDLE's history that they aren't*.  3. It might be that changing either of the other two binding could disable something, or might in the future.  In other words, I consider having this bindings be mutable to be a bit buggy, with this issue being evidence.

* Multiple coders, coding convenience, shift of focus from 'consenting adults to 'beginners'?

The unknown is whether anyone has changed these pseudoevent bindings and if so, how much do we care?  

Guido, do you have any comment on this proposal from a change policy perspective?
msg395063 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2021-06-04 02:31
> The unknown is whether anyone has changed these
> pseudoevent bindings and if so, how much do we care?  

I don't think we care.  Getting tab completion sorted out is the priority.
msg395201 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2021-06-06 07:59
> The unknown is whether anyone has changed these pseudoevent bindings and if so, how much do we care?

Let's ask on idle-dev.

> Guido, do you have any comment on this proposal from a change policy perspective?

I'm not an ex-BDFL, but I can't think of a reason this shouldn't be backported as per our usual policy regarding IDLE.
Date User Action Args
2021-06-06 07:59:50taleinatsetmessages: + msg395201
2021-06-04 02:31:58rhettingersetnosy: + rhettinger
messages: + msg395063
2021-06-02 01:40:18terry.reedysetnosy: + gvanrossum

messages: + msg394883
title: IDLE: Applying settings disables tab completion -> IDLE: Fix tab completion after settings and some keys
2021-05-28 06:31:15taleinatsetpull_requests: + pull_request25016
2021-05-28 04:40:55terry.reedysetmessages: + msg394631
2021-05-27 15:05:51taleinatsetversions: + Python 3.9, Python 3.11
2021-05-27 15:05:28taleinatsetmessages: + msg394545
2021-05-27 14:59:46taleinatsetkeywords: + patch
stage: test needed -> patch review
pull_requests: + pull_request24996
2021-05-27 14:04:00taleinatsetnosy: + taleinat
messages: + msg394542
2021-04-12 08:27:33tetelevmsetmessages: + msg390824
2021-04-11 00:22:36terry.reedysetmessages: + msg390755
versions: + Python 3.10, - Python 3.9
2021-04-10 19:05:53tetelevmsetmessages: + msg390732
2021-04-10 18:50:47tetelevmsetmessages: + msg390731
versions: + Python 3.9, - Python 3.10
2021-03-30 05:04:14terry.reedysetmessages: + msg389791
2021-03-29 09:04:45tetelevmsetnosy: + tetelevm
2021-03-29 08:40:35terry.reedycreate