Tkinter (and IDLE specially) can use only UCS-2 characters.
In PyShell IDLE tries to escape non-ascii.
To better result we should to escape only non-BMP chars leaving BMP characters untouched.

The solution outlined in the issue title ("utf-8-bmp codec") sounds like a rather dubious idea.

pitrou: can you elaborate?

''.join(c if ord(c) < 0x10000 else escape(c) for c in s)

What is this codec? What do you mean by "escpe non-ascii"?

This codec is one that is equal to UTF-8, but restricted to the BMP. For non-BMP character, the error handler is called. It will be the stdout codec for the IDLE interactive shell, causing non-BMP results to be ascii() escaped.

Tkinter (as Tcl itself) has no support of non-BMP characters in any form.
It looks like support of UTF-16 without surrogates.
I like to implement codec for that which will process different error modes (strict, replace, ignore etc) as well as others codecs does.

It will allow to support BMP well and control processing of non-BMP in IDLE.

About your second question.
IDLE has interactive shell. This shell in REPL will try to print expression result. It it contains non-BMP whole result is converted to ASCII with escaping. It's different from standard python console. From my perspective expected behavior is to pass BMP chars and escape only non-BMP.

Example:

>>> '\u0100'
'Ā'
>>> '\u0100\U00010000'
'\u0100\U00010000'
>>> print('\u0100')
Ā
>>> print('\u0100\U00010000')
Traceback (most recent call last):
  File "<pyshell#33>", line 1, in <module>
    print('\u0100\U00010000')
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 1-1: Non-BMP character not supported in Tk

But I think that it is too specific problem and too specific solution. It would be better if IDLE itself escapes the string in the most appropriate way.

def utf8bmp_encode(s):
    return ''.join(c if ord(c) <= 0xffff else '\\U%08x' % ord(c) for c in s).encode('utf-8')

or

def utf8bmp_encode(s):
    return re.sub('[^\x00-\uffff]', lambda m: '\\U%08x' % ord(m.group()), s).encode('utf-8')

The way is named 'codec'.

But I think that it is too specific problem and too specific
solution. It would be better if IDLE itself escapes the string in the
most appropriate way.

That is not implementable correctly. If you think otherwise, please
submit a patch. If not, please trust me on that judgment.

May be I did not correctly understand the problem, but I can assume,
that this patch solves it.

'Агов!\U00010000'

Sorry, the mail daemon has eaten a piece of example.

>>> '\u0410\u0433\u043e\u0432!\U00010000'
'Агов!\U00010000'

Andrew, the patch solves your issue?

The patch is incorrect, i.e. it deviates from what the command line interface does. When you try to write to sys.stdout, and the characters are not supported you get UnicodeError. Only when it is interactive mode, and tries to represent some result, ascii escaping happens.

I don't see what the patch worse than the current behavior.

Unpatched:
>>> ''.join(map(chr, [76, 246, 119, 105, 115]))
'Löwis'
>>> ''.join(map(chr, [76, 246, 119, 105, 115, 65536]))
'L\xf6wis\U00010000'

Patched:
>>> ''.join(map(chr, [76, 246, 119, 105, 115]))
'Löwis'
>>> ''.join(map(chr, [76, 246, 119, 105, 115, 65536]))
'Löwis\U00010000'

In the case of the Cyrillic alphabet all text becomes unreadable, if there are some non-bmp characters in it.

In the case of the Cyrillic alphabet all text becomes unreadable, if
there are some non-bmp characters in it.

And indeed, that's the correct, desired behavior, as it models what the
interactive shell does.

If you want to change this, you need to also change the interactive console,
which is an issue independent of this one.

I take that back; the interactive shell uses the backslashescape error handler.

Still, I don't think IDLE should setup a displayhook in the first place. What if an application replaces the displayhook?

Still, I don't think IDLE should setup a displayhook in the first place. What if an application replaces the displayhook?

IDLE *is* the application.

If another application that uses the idlelib, replace displayhook, it
must itself to worry about the correct encoding and escaping.

Serhiy, I like to fix tkinter itself, not only IDLE.
There are other problems like idle is crashing if non-bmp char will be pasted from clipboard.
Moreover, non-bmp behavior is different from one Tk widget to other.
I still want to make codec for it and then try to solve tk problems.
Maybe solution will force to extend tkinter interface for process codec errors with reasonable well specified default behavior.
Sorry for my silence. I hope to make some progress next weeks.

IDLE *is* the application.

No, IDLE is the development environment. The application is
whatever is being developed with IDLE.

I don't understand how the utf-8-bmp codec will help to fix the tkinter. To fix the tkinter, you need to fix the Tcl/Tk, but it is outside of Python. While Tcl does not support non-bmp characters, correct and non-ambiguous working with non-bmp characters is not possible. You should choose the method of encoding of non-bmp characters and these methods will be different for different applications.

No, IDLE is the development environment. The application is
whatever is being developed with IDLE.

If the application replaces the displayhook, than it is the development
environment too.

Andrew, imagine that the utf-8-bmp codec is already there (I will do it
for you, if I see its necessity). How are you going to use it? Show a
patch that fixes IDLE and tkinter using this codec. It seems to me that
any result can be achieved without the codec, and not higher cost. And
that's not counting cost of the codec itself.

Any chance to commit the patch before final feature freeze?

Pending doing some experiments with current and patched code, and reading the rpc code, I believe I would like to see the patch applied. I don't care about whether the patch defines a 'codec' or what its name would be. What i do want is for the Idle Shell to display unicode strings produced by python code as faithfully as possible, without raising an exception, given the limitations of tk and the selected font.

Tkinter (and IDLE specially) can use only UCS-2 characters.

Is it always the case, or does depend on a compilation flag of Tcl or Tk?

In theory Tcl/Tk can be built with 32-bit Tcl_Char. But I doubt that this option is well tested. In any case on Linux Python depends on system Tcl/Tk.

In theory Tcl/Tk can be built with 32-bit Tcl_Char.

Would it make sense to compile Tcl/Tk with 32-bit Tcl_Char on Windows? I think that we embed our own build ot Tcl/Tk, right?

In 3.6, Python's use of the Windows console was changed to work much better with unicode. As a result, IDLE is now worse rather than better than the console on Windows. I plan to do something before 3.7.0.

October 2019, Serhiy solved the display issue with a _tkinter patch for python/cpython#57362.
bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)
https://github.com/python/cpython/commit/06cb94bc8419b9a24df6b0d724fcd8e40c6971d6
In Windows IDLE Shell
>>> ''.join(map(chr, [76, 246, 119, 105, 115, 0x1F40D]))
'Löwis🐍'
except that the snake is black and white.  (Many astral chars have no glyph and appear as a box.) In console REPL, the snake shows as box box space box.

Pasting astral characters into edited code 'works' except that editing following code is messy because the astral char is multiple chars internally and the visible cursor no longer matches the internal index. (But pasting such no longer crashes IDLE.)