This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: IDLE: make Ctrl,Alt + IME non-ascii letter work on Windows
Type: behavior Stage: test needed
Components: IDLE, Tkinter Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: anton.bryl, epaine, eryksun, serhiy.storchaka, terry.reedy
Priority: normal Keywords:

Created on 2021-12-11 22:12 by anton.bryl, last changed 2022-04-11 14:59 by admin.

Messages (13)
msg408345 - (view) Author: Anton Bryl (anton.bryl) Date: 2021-12-11 22:12
Ctrl+C and Ctrl+V key combinations in IDLE on Windows do not work with Cyrillic keyboard layout. It is unexpected, as well as inconvenient when editing string constants.
msg408357 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-12-12 01:30
IDLE is a tkinter application and tkinter wraps the tcl/tk GUI framework. Your problem is mostly in the interaction between Windows, your Cyrillic input method, and tk.  As a test, run

import tkinter as tk
r = tk.Tk()
t = tk.Text(r)
t.pack()

Click in the box, type something, and try Ctrl-X, -C, -V.  If they work, we can add explicit bindings somewhat similar to those IDLE makes.

Also, how do you make your keyboard a Cyrillic keyboard.

EP, are you aware of any related tk issues?  Serhiy, do you know of any specific problems with Cyrillic and tkinter/tk?
msg408381 - (view) Author: Anton Bryl (anton.bryl) Date: 2021-12-12 13:07
Tried the tkinter example.
The exact same problem occurs there as well: when a Cyrillic layout is on, Ctrl+Letter combinations do not work (it's in fact not just Ctrl+C and Ctrl+V, but apparently all of them).
msg408382 - (view) Author: Anton Bryl (anton.bryl) Date: 2021-12-12 13:10
> Also, how do you make your keyboard a Cyrillic keyboard.

On Windows, just install a keyboard layout for e.g. Russian. As soon as you switch to it, all Ctrl+Letter combinations stop working. Switch back to EN, and everything's working again.

As the code is written in Latin alphabet, it's only rarely a problem, but, as I mentioned, sometimes one needs to edit string constants, and then it's an inconvenience (normally the exact same key combinations work).
msg408391 - (view) Author: E. Paine (epaine) * Date: 2021-12-12 15:31
I have reproduced the behaviour described in Wish (from the Tk head). Having tried other applications, I can also confirm that it is normal for inputs to be treated in this manner (e.g. using Ctrl-C on my Latin keyboard with the input device set to Russian copies to the clipboard). I'll take it up with the Tk team.
msg408392 - (view) Author: E. Paine (epaine) * Date: 2021-12-12 15:58
Actually, doing a bit more research, issue31244 came up (and more specifically msg300716). It was concluded there that the issue should not be fixed, though there was some C/C++ code reported in the last message that the OP claims could potentially solve the issue if added to either Tk or Python.
msg408395 - (view) Author: E. Paine (epaine) * Date: 2021-12-12 16:27
Sorry for the spam...

OK, making the corresponding ctypes calls to the commands reported in issue31244 succeeds in the Python REPL, but the Tk text's behaviour doesn't change. In IDLE, the ctypes calls fail.

ctypes.windll.Kernel32.SetConsoleCP(1251)
ctypes.windll.Kernel32.SetConsoleOutputCP(1251)
locale.setlocale(locale.LC_ALL, 'Russian')

Hence, I don't think this a viable option. And it would not be viable for either this project, nor Tk, to write our own codepages, so I'm not sure whether it is worth raising it with the Tk team.
msg408400 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-12-12 16:55
It appears that your particular keyboard program is translating Ctrl + letter key combinations to something other than the default Ascii Control-letter code.  Do you see the same problem with Notepad?  To test what tcl/tk and hence tkinter see, expand the test code to

import tkinter as tk
r = tk.Tk()
t = tk.Text(r)
t.pack()

def keyevent(e):
    if c := e.char:
        print(f'char: {c}, ord: {ord(c)}, ', end='')
    print(f'code: {e.keycode}, sym: {e.keysym}, num: {e.keysym_num}.')
t.bind('<Key>', keyevent)

If I type c and ctrl + c in the tk box, I see the following in either the IDLE Shell or Command Prompt.

char: c, ord: 99, code: 67, sym: c, num: 99.
code: 17, sym: Control_L, num: 65507.
char: , ord: 3, code: 67, sym: c, num: 99.

I expect the third line will be different for you when you switch to Russian.

Your immediate fix is to use either the IDLE Edit menu or the right-click context menu to access copy and paste functions.  A longer term fix might be to get a different Russian keyboard program.

Assuming that I am correct above, I will make this an IDLE doc issue to add something about non-ascii keyboard issues, and include the test program above.
msg408414 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-12-13 00:10
This is a duplicate of #31244, but I continue here with more experiments using the code above.  I first confirmed that on Windows, CapsLock is really 'caps inversion.  Keycode 67 is the keycode for 'C' given in https://www.tcl.tk/man/tcl/TkCmd/keysyms.html (Ascii keyboards have the capitals on keycaps), but the chars 'c' and 'C', without and with shift, are 4 different events.

char: c, ord: 99, code: 67, sym: c, num: 99.
code: 16, sym: Shift_L, num: 65505.
char: C, ord: 67, code: 67, sym: C, num: 67.
code: 20, sym: Caps_Lock, num: 65509.
char: C, ord: 67, code: 67, sym: C, num: 67.
code: 16, sym: Shift_L, num: 65505.
char: c, ord: 99, code: 67, sym: c, num: 99.

Next, the same keypresses with Ctrl added.  At least on Windows, Ctrl+c, Shift+Ctrl+c, Ctrl+C, and Shift+Ctrl+C are different events that have the same Ascii code but are differentiated by the key sym that would have been generated without the control modifier and the state of the Shift key.  (Note, for instance, that on the file menu, the addition of 'Shift' modifies 'Save' to 'Save as').

code: 17, sym: Control_L, num: 65507.
char: , ord: 3, code: 67, sym: c, num: 99.
code: 17, sym: Control_L, num: 65507.
code: 16, sym: Shift_L, num: 65505.
char: , ord: 3, code: 67, sym: C, num: 67.
code: 20, sym: Caps_Lock, num: 65509.
code: 17, sym: Control_L, num: 65507.
char: , ord: 3, code: 67, sym: C, num: 67.
code: 17, sym: Control_L, num: 65507.
code: 16, sym: Shift_L, num: 65505.
char: , ord: 3, code: 67, sym: c, num: 99.

I loaded the Win 10 Russian package.  Repeating the no-ctrl block, we get the Cyrillic с and С.  The keysym field is '??' instead of the char because non-ascii letters are not valid as keysyms.  The keysym_num field is correct as if the keysym were the non-ascii letter.

char: с, ord: 1089, code: 67, sym: ??, num: 1089.
code: 16, sym: Shift_L, num: 65505.
char: С, ord: 1057, code: 67, sym: ??, num: 1057.
code: 20, sym: Caps_Lock, num: 65509.
char: С, ord: 1057, code: 67, sym: ??, num: 1057.
code: 16, sym: Shift_L, num: 65505.
char: с, ord: 1089, code: 67, sym: ??, num: 1089.

With Ctrl added, the generated character is still ascii 3, control-C and the keycode is still 67.  But the keysym and keysym_num are changed and they no longer match neither ascii 'c' or 'C'.  So the event matches neither the control-c or control-C events and the copy event is not invoked.

code: 17, sym: Control_L, num: 65507.
char: , ord: 3, code: 67, sym: ??, num: 1089.
code: 17, sym: Control_L, num: 65507.
code: 16, sym: Shift_L, num: 65505.
char: , ord: 3, code: 67, sym: ??, num: 1057.
code: 20, sym: Caps_Lock, num: 65509.
code: 17, sym: Control_L, num: 65507.
char: , ord: 3, code: 67, sym: ??, num: 1057.
code: 17, sym: Control_L, num: 65507.
code: 16, sym: Shift_L, num: 65505.
char: , ord: 3, code: 67, sym: ??, num: 1089.

The workaround considered in #31244 was to add key-x bindings specific to a particular Windows IME.  But this does not work as key sequences only allow ascii alphanumerics as keysyms.  (See https://www.tcl.tk/man/tcl8.6/TkCmd/bind.html, event details.)

There are non-ascii letter descriptions, such as 'Cyrillic_es' listed in https://www.tcl.tk/man/tcl/TkCmd/keysyms.html, but I suspect that these only work for native keyboards, with the non-ascii chars on the keycaps, that generate the keycodes specific to each key as listed in that doc.  Cyrillic_es has keycode 1747, not 67.  When I tried binding '<Key-Cyrillic_es>' there was no TclError, but a Russian es, 'с' or 'С', did not invoke the handler.

Instead of expanding keybindings, we should consider collapsing events to undo the IME translation.  '??' is not a valid keysym, so there seems to be no generic 'non-Ascii IME letter' event.  But we could try replacing the keysym with the ascii char that would have been there if there were no IME.  The problem is that modifying the python event will not change the tk event.  Nor can it be used with event_generate.  Rather it has to be turn back into 'sequence' string, taking into account the .state attribute.

Automated testing would be done by calling the key event handler with a synthesized Event instance.

EP: Since Key-X, where x is a non-ascii char, in not a legal sequence and cannot be bound to anything, it seems reasonable to ask that tk itself translate control key events into bindable control sequences.  They might object that this would break any code that catches '??' events in generic key handlers to do something language specific.  But perhaps they could add a non-default option.
msg408417 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-12-13 01:28
I closed #31244 as a duplicate of this.

The modifier code for tkinter.Event.__repr__ can be used to construct the modifier part of the reconstructed event sequence.  Just join with '-' instead of '|'.
msg408421 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-12-13 02:58
The alternate keyboard shortcuts for the clipboard work fine with a Russian keyboard layout:

   copy: Ctrl+Insert
    cut: Shift+Delete
  paste: Shift+Insert

Some programs also support Alt+Backspace for undo (Ctrl+Z). Watch out with Shift+Delete in GUI shells, since it physically deletes files, bypassing the recycle bin (trash), instead of cutting the selected files to the clipboard.
msg408422 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-12-13 03:01
I think the following wiki article still applies even though it was first discussed in 2003: "KeySyms on platforms other than X11" [1]. In particular, it states the following:

    On Windows and MacOS X Tk only supports keysyms correctly for a 
    limited number of keys, namely special keys, and the ranges of ASCII
    and ISO-8859-1 (support of ISO-8859-1 on MacOS X since 8.4.2).

I can confirm the result for the Russian keyboard mapping. The value of keysym is "??", even when combined with the control key. 

x:
        char: 'ч', ord: 0447, code: 0058, sym: '??', num: 0447.
Ctrl+x:
        char: '\x18', ord: 0018, code: 0058, sym: '??', num: 0447.
c:
        char: 'с', ord: 0441, code: 0043, sym: '??', num: 0441.
Ctrl+c:
        char: '\x03', ord: 0003, code: 0043, sym: '??', num: 0441.
v:
        char: 'м', ord: 043c, code: 0056, sym: '??', num: 043c.
Crl+v:
        char: '\x16', ord: 0016, code: 0056, sym: '??', num: 043c.

(I modified the keyevent function from msg408400 to use hexadecimal and repr formatting.)

I checked Ubuntu 20.04 with a Russian keyboard layout. It seems in Linux the combination with the control key changes keysym and keysym_num to use the ASCII characters "x", "c", and "v": 

x:
        char: 'ч', ord: 0447, code: 0035, sym: 'Cyrillic_che', num: 06de.
Ctrl+x:
        char: '\x18', ord: 0018, code: 0035, sym: 'x', num: 0078.
c:
        char: 'с', ord: 0441, code: 0036, sym: 'Cyrillic_es', num: 06d3.
Ctrl+c:
        char: '\x03', ord: 0003, code: 0036, sym: 'c', num: 0063.
v:
        char: 'м', ord: 043c, code: 0037, sym: 'Cyrillic_em', num: 06cd.
Crl+v:
        char: '\x16', ord: 0016, code: 0037, sym: 'v', num: 0076.

---
[1] https://wiki.tcl-lang.org/page/KeySyms+on+platforms+other+than+X11
msg408464 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-12-13 16:45
Thank you Eryk.  The X/Ubuntu behavior is what I want to simulate.  I can only wonder why tk has not fixed that for non-X systems.  Are Alt+letter combinations also fixed on Ubuntu, so that, for instance, Alt+M-key is seen as Alt-M even with Russian layout?
History
Date User Action Args
2022-04-11 14:59:53adminsetgithub: 90210
2021-12-13 16:45:36terry.reedysetmessages: + msg408464
2021-12-13 03:01:59eryksunsetmessages: + msg408422
2021-12-13 02:58:57eryksunsetnosy: + eryksun
messages: + msg408421
2021-12-13 01:28:55terry.reedysetmessages: + msg408417
title: Ctrl+C, C+V in tk.Text on Windows do not work with Cyrillic keys -> IDLE: make Ctrl,Alt + IME non-ascii letter work on Windows
2021-12-13 01:20:32terry.reedylinkissue31244 superseder
2021-12-13 00:10:38terry.reedysetassignee: terry.reedy
stage: test needed
messages: + msg408414
versions: + Python 3.11, - Python 3.9
2021-12-12 16:55:04terry.reedysetmessages: + msg408400
title: Ctrl+C, C+V in IDLE on Windows do not work with Cyrillic keys -> Ctrl+C, C+V in tk.Text on Windows do not work with Cyrillic keys
2021-12-12 16:27:18epainesetmessages: + msg408395
2021-12-12 15:58:33epainesetmessages: + msg408392
2021-12-12 15:31:40epainesetmessages: + msg408391
2021-12-12 13:10:20anton.brylsetmessages: + msg408382
2021-12-12 13:07:13anton.brylsetmessages: + msg408381
2021-12-12 01:30:46terry.reedysettitle: Ctrl+C, Ctrl+V in IDLE on Windows do not work with Cyrillic keyboard layout -> Ctrl+C, C+V in IDLE on Windows do not work with Cyrillic keys
nosy: + serhiy.storchaka, epaine

messages: + msg408357

assignee: terry.reedy -> (no value)
components: + Tkinter
2021-12-11 22:12:16anton.brylcreate