This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Teach IDLE to Autocomplete dictionary keys
Type: enhancement Stage: patch review
Components: IDLE Versions: Python 3.10, Python 3.9, Python 3.8
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: Eduardo.Seabra, aeros, cdspace, louielu, martin.panter, rhettinger, taleinat, terry.reedy
Priority: normal Keywords: patch

Created on 2014-04-16 18:21 by rhettinger, last changed 2022-04-11 14:58 by admin.

File name Uploaded Description Edit
issue21261.patch Eduardo.Seabra, 2014-06-20 00:34 review
issue21261.patch Eduardo.Seabra, 2014-06-20 17:53 With tests review
Pull Requests
URL Status Linked Edit
PR 1511 closed louielu, 2017-05-09 06:08
PR 15169 closed taleinat, 2019-08-07 20:33
PR 26039 open taleinat, 2021-05-11 20:27
Messages (23)
msg216542 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-04-16 18:21
IDLE can autocomplete global variable, method names, and filenames.  But, it cannot complete dictionary keys.

Here's what we want:

   >>> d = {'long_key': '10, 'short_key': 20}
   >>> d['lo<tab>
msg216813 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-04-18 22:09
Looks sensible.
msg221056 - (view) Author: Eduardo Seabra (Eduardo.Seabra) * Date: 2014-06-20 00:34
From the example, I couldn't know if the patch should also autocomplete int and other types. So here's a patch that autocompletes string dictionary keys.
I'm new contributing so let me know if I made anything wrong and I'll fix as soon as possible.
msg221061 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-06-20 02:08
String keys is what Raymond requested and what looks sensible to me. A week ago, I committed test_idle/ and an incomplete So we can now, for this issue, at least partly follow our standard procedure of requiring tests with patches.

I want at least a test of is_in_subscript_string(self) added to test_hyperparser. cls.code will need another line (or an existing string bracketed and the string test altered) and a new test_is_in_subscript_string added. If you are working from an installation rather that a repository, and therefore cannot write/test an addition to the new file, say so.

The change to auto-complete is trickier. I would have to look the code versus to tests to decide what to do, if anything. I might decide to improve the autocomplete htest (human test, in repository and test_idle/ rather than the unittest.

In any case, we need a signed contributor agreement to accept patches.
msg221102 - (view) Author: Eduardo Seabra (Eduardo.Seabra) * Date: 2014-06-20 17:53
I've added three lines to cls.code to test_hyperparser. So I can test for subscripts with double quotes, single quotes and with no strings at all.

Should I implement try_open_completions_event for COMPLETE_DICTIONARY? Calling this event everytime someone types a string seemed a bit expensive in my opinion.

I'm attaching the new patch.

As fas as the signed contributor, I've already signed last week but still waiting.
msg265475 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-05-13 13:29
I have no idea how IDLE works internally, but I wonder if it is possible to share some of the work with the Readline completer (rlcompleter module). Despite the name, rlcompleter should be usable without also using Readline, though recently (Issue 25660) I think this was broken. Anyway, maybe see Issue 10351 for the beginning of an rlcompleter patch and some potential test cases.
msg265489 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-05-13 21:12
Thanks.  Being on Windows, I never paid attention to rlcompleter.  The big difference is that IDLE completions use IDLE's hyperparser module (which is used for other purposes also).  I will look at the tests.
msg293282 - (view) Author: Louie Lu (louielu) * Date: 2017-05-09 06:11
In this PR, it will complete dictionary key with string, int, and others.

for example:

    d = {'long_key': 10, 'short_key': 20, 30: 40, (((1, 2), 3, 4), 5): 50}
    d['lo<tab>  -> d['long_key'
    d[(((1<tab> -> d[(((1, 2), 3, 4), 5)
    d[3<tab>    -> d[30

The problem is, autocomplete_w can't figure the original key is string or others, so this will be possible:

    d[long<tab> -> d[long_key]
    d[shor<tab> -> d[short_key]
msg293284 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-05-09 06:38
Would it be safer/simpler to just autocomplete string keys.
msg293289 - (view) Author: Louie Lu (louielu) * Date: 2017-05-09 07:59
I'm not sure the "safer" meaning. If it is about for beginner less confuse when mistakenly typing "d[long_<tab>", the answer will be yes for only complete string keys.

Impl complexity between str-only and not-str-only will not have too much different, only need to change the sentinel and some other work.
msg293502 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-05-11 16:03
I think it would be unusual for tab completion to work on non-strings and would create a weird feel to the API.
msg293520 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-05-12 01:58
IDLE currently completes global names, attributes after ., and filename segments after / or \ within a string.  In the later two cases, a box will pop up automatically after a user selected time after typing . or /\ and nothing thereafter.  The filename segments are not quoted in the list box.

These completions work within subscripts.
d[a<tab or wait> pops up global name completion box
d['/<tab or wait> pops up filename completion box

Raymond proposes that IDLE complete 'dictionary [string] keys'.  To properly code and test, we need a more complete specification.  For instance, "a string key box should open after an opening quote that follows '[' that follows a dict expression".  Any opening quote should work, just as for filename completion.

This is similar "a calltip opens after a '(' that follows a callable expresssion".  For calltips, the expression cannot contain a function call, because calls can take an indeterminant amount of time.  If "expression.find('(') != -1", the calltip is aborted and the same should be true here.  Also, calltips.get_entity(expression) should be reused to get the dict object.  (test_calltips should but does not test that 'f()(' is ignored and get_entity not called.  The same should be true for "f()['".)

Nice (?) but not necessary: delayed auto-popup after typing "d[<open quote>".  This seems that it would be more difficult than the current auto popups.  And see the following.

This proposal conflicts with filename completion for subscripts.  When one is accessing an existing value, one would want key completion.  If one is assigning a value to a new filename key, one would want filename completion.  The simplest solution I can think of is to not auto pop up key completion but to require <tab> before typing (/\) and waiting.

Lastly, should the string keys be quoted in the box?
| long key  |
| short key |
|'long key' |
|'short key'|

Selecting key objects by their representation is tempting, but it is conceptually different from completing names.  Objects may have one canonical representation, but many possible representations.  So clicking on a list  (which currently does not work) or using movement keys is more sensible than typing chars that have to match one of many possibilities.  String keys would have to be quoted.

So I would only consider this as a separate issue, depending on a fix for clicks.  It should only be accessed by <tab> immediately after '[', and I might want to disable selection by character matching.

Even then, I would be dubious.  I grepped idlelib for "\w\[".  A majority of subscripts are names, handled by current name completion or not (if the names are local, which they often are).  The rest are either list indexes and slices involving literal ints or string keys, which this proposal would handle for accessible dicts.  I am pretty sure there are no keys other than names and strings.

But the sparsity of use cases is my problem even with this proposal. Calltips are useful because there are many globally accessible callables, including builtins and imports.  But other than class __dicts__, there are few globally accessible dicts, except perhaps in toy beginner code.  Raymond, have I missed something?

The idlelib grep had 763 hits and I believe more that half are for dicts.  But they may all be locals or self attributes.  I would love to be able, for instance, to type "local_dict['<tab>" and fill in 'background', but that will not work.
msg349078 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-08-05 21:27
I had no idea that this was desired... I had this working in a old version of autocomplete back before 2010!  I'm not sure whether I'll be able to find it though.

I can't understand why Louie's PR was closed, it seemed to be going in the right direction... Any explanation?
msg349091 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-05 23:34
The history is confusing. bpo user Louie Lu (louielu), as github user 'noname louisom', open PR 1511 for this existing issue on 2017 May 9.  On May 12, he opened #30348 and PR 1512 with the tests for fetch_completions and get_entity that were part of PR 1511.  (This was a needed separation.)

By June, he had switched to a new github name 'Louie Lu mlouielu'.  On June 12, he opened #30632 and PR 2124, which duplicated #30348 and PR 1512 (which partly duplicated PR 1511).  On June 15, he closed PR 1511 to 'migrate' it to pr 2209.  But the latter only included the tests also in PR 1512, which it replaced on #30348, and PR 2124.  He also closed #1512.  Loiue moved on to other projects in Fall, 2017.

After revisions, I merged PR 2209 for #30348 last March.  I followed up with #36419, PR 15121, now merged. I just opened #37766 to finish preparing autocomplete for new additions such as this.  I was thinking of this issue when I included adding an htest.  (Note: #27609 is the master issue for completions.)

Notes for migrating the dict keys code:
1. In PR 15121, I shrank mode names to ATTRS and FILES.  The new mode name should be KEYS, or maybe SKEYS (for string keys).  Other refactors should not affect KEYS too much.
2. I intend to change 'smalll' and 'bigl' to 'small' and 'big' and might make other changes to fetch_completions.
3. I intend to split test_fetch_completions into separate methods for each mode. The new KEYS tests should be a separate method 'test_fetch_keys'.

The questions of function calls in the entity expression is more nuanced than I know when I wrote msg293520.  For force-open-completions, control-space, function calls are allowed.  But I do not think that this affects the new mode.
msg349099 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-08-06 04:37
Thanks for going through the history of this issue.  It was surprisingly convoluted.  I still hope this feature comes to fruition.
msg349198 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-08-07 20:36
Raymond, your with may just come true!

I've just created PR GH-15169 with a new implementation of my own, complete with tests.  I have not yet thoroughly tested it though, and would like some feedback on it.
msg349216 - (view) Author: Kyle Stanley (aeros) * (Python committer) Date: 2019-08-08 05:25
> I have not yet thoroughly tested it though, and would like some feedback on it.

I performed some testing on Linux and it looks good as far as I can tell. I added a few minor suggestions, but the auto-complete seems to be functioning as desired.
msg349221 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-08-08 07:11
Many thanks for the review, Kyle!
msg377209 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-09-20 07:08
The current patch completes both strings keys and bytes keys mixed together.  I want to challenge this.

0. What is the use case?  The current patch nearly doubles the autocomple code, which handle user actions up to creation of a completion list.  I would prefer smaller.

1. We agreed on completing 'string keys'. Bytes are not strings.  Bytes might represent an encoded string, but might instead be anything else.  Some bytes are represented by ascii chars, but the majority need hex escapes.

2. Including bytes forces quoting keys in the list box in order to add the 'b' prefix for bytes.  I said above that we should quote anyway, but that was before working with the implementation and discovering the resulting usage issues.

3. Including bytes makes it harder to select a key by typing a partial key.  Suppose d had hundreds of key, a to z.  One want to select 'zeta13  c3'.
>>> d[ <pause> opens a list box with starting with the 'a' words.
One enters 'z'.  Nothing happens because no entry starts with 'a'.
Backspace, enter "'z".  Nothings happens because the default quote used in the listbox is '"'.  If 'b' did not possibly represent a bytes prefix, we could either remove surrounding quotes or ignore them when matching.
msg377211 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-09-20 08:43
People still get confused between bytes and strings, even with Python 3, especially novice programmers who are a major target audience of IDLE.

I simply thought it would be  confusing if string dict keys did appear in the completion list, but bytes keys did not.

You're certainly right that removing support for completing bytes keys would simplify the code.
msg377212 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-09-20 08:44
Also, please note that the PR allows using either type of quote when typing completions.
msg377213 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-09-20 09:19
> the PR allows using either type of quote when typing completions.

Only if one types a quote before the box pops up.  In this case, there is no auto popup and one must request completions.  The box then uses the quote typed.  Thereafter, there is no choice. Backspacing to delete the quote and typing the another one disables matching.

If one type >>> d[ and pauses long enough for the auto-popup, (I have 'wait' set to 200 milleseconds), one must similarly match the existing quote, because jumping ahead requires a string match.  Because of the bytes inclusion, the match much include the opening quote, because otherwise an initial 'b' would be ambiguous.  Without bytes, we could either removed the quotes from the box or automatically add an open quote.
msg393471 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2021-05-11 20:31
Note that I've created a new version of the latest PR, with support for dict keys of type bytes removed.
Date User Action Args
2022-04-11 14:58:02adminsetgithub: 65460
2021-05-11 20:39:47taleinatsetkeywords: + patch
2021-05-11 20:39:38taleinatsetkeywords: - patch, easy
2021-05-11 20:31:07taleinatsetmessages: + msg393471
2021-05-11 20:27:17taleinatsetpull_requests: + pull_request24684
2020-09-20 09:19:27terry.reedysetmessages: + msg377213
2020-09-20 08:44:00taleinatsetmessages: + msg377212
2020-09-20 08:43:55taleinatsetmessages: + msg377211
2020-09-20 07:08:04terry.reedysetmessages: + msg377209
versions: + Python 3.10, - Python 3.7
2019-08-08 07:11:52taleinatsetmessages: + msg349221
2019-08-08 05:25:21aerossetnosy: + aeros
messages: + msg349216
2019-08-07 20:36:24taleinatsetmessages: + msg349198
2019-08-07 20:33:25taleinatsetpull_requests: + pull_request14901
2019-08-06 04:37:43rhettingersetmessages: + msg349099
2019-08-05 23:34:25terry.reedysetassignee: terry.reedy
messages: + msg349091
2019-08-05 21:27:29taleinatsetnosy: + taleinat

messages: + msg349078
versions: + Python 3.8, Python 3.9, - Python 3.6
2017-05-12 01:58:55terry.reedysetversions: - Python 2.7, Python 3.5
2017-05-12 01:58:37terry.reedysetstage: test needed -> patch review
messages: + msg293520
versions: + Python 3.6, Python 3.7, - Python 3.4
2017-05-11 16:03:28rhettingersetmessages: + msg293502
2017-05-09 07:59:14louielusetmessages: + msg293289
2017-05-09 06:38:11rhettingersetmessages: + msg293284
2017-05-09 06:11:20louielusetnosy: + louielu
messages: + msg293282
2017-05-09 06:08:04louielusetpull_requests: + pull_request1612
2016-07-25 00:08:07terry.reedylinkissue27609 dependencies
2016-05-13 21:12:44terry.reedysetmessages: + msg265489
2016-05-13 13:29:49martin.pantersetnosy: + martin.panter
messages: + msg265475
2016-05-13 12:03:26terry.reedylinkissue27013 superseder
2014-06-20 17:53:06Eduardo.Seabrasetfiles: + issue21261.patch

messages: + msg221102
2014-06-20 02:08:35terry.reedysetmessages: + msg221061
stage: test needed
2014-06-20 00:34:28Eduardo.Seabrasetfiles: + issue21261.patch

nosy: + Eduardo.Seabra
messages: + msg221056

keywords: + patch
2014-05-22 22:11:49cdspacesetnosy: + cdspace
2014-04-18 22:09:53terry.reedysetnosy: + terry.reedy
messages: + msg216813
2014-04-16 20:03:59rhettingersetkeywords: + easy
2014-04-16 18:21:12rhettingercreate