I was playing with some unicode chars on Python 3.2 (x64 on Windows 7), but when pasted a char bigger than 0xFFFF, IDLE crashes without any error message.

Example (works fine):
>>> '\U000104a2'
'𐒢'

But, if I try to paste the above char, the window will instantly close.

The interpreter uses 2-bytes per char (UTF-16) and I don't know if that's causing the problem (as side note, why don't the default Windows build uses 4-bytes char?).

I can't check now with my Ubuntu install (UTF-32) if the problem persists.

This is related to Issue12342.  The problem is that Tcl/Tk 8.5 (and earlier) do not support Unicode code points outside the BMP range as in this example.  So IDLE will be unable to display such characters but it should not crash either.

@Ned

That looks like a bit different case. IDLE *can* print the char after you entered the '\Uxxxxxxxx' version of it.

It doesn't accept you to paste those caracters...

The current Windows build used 2-byte unicode chars because that is what Windows does. In 3.3, all builds will use a new unicode implementation that uses 1,2,or4 bytes as needed. But I suspect we will still have the paste problem unless we can somehow bypass the tk limitation.

Printing a Python string to the screen does not seem to involve conversion to a tk string. Or else tk blindly copies surrogate pairs to Windows even though it cannot create them.

In any case, true window-closing crashes (as opposed to an error traceback) are obnoxious bugs that we try to fix if possible. I verified this on my 64-bit Win 7 system. Thanks for the report. Feel free to look into the code if you can.

Just for comparison, on Python 2.7.1 (x32 on Windows 7) it's possible to paste the char (but can't use it) and a nice error is given. 

>>> u'𐒢'
Unsupported characters in input

So the problem was partially solved but something might have happened with the 3.x port...

Searching on both source codes, I can see the following block was commented on Python3.2 but not on Python2.7 (Maybe someone removed someone else's bug fix?) and an `assert` was added.

#--- Lines 605 to 613 of PyShell.py

assert isinstance(source, str)
#                       v-- on Python2.7 it is types.UnicodeType instead
#if isinstance(source, str):
#    from idlelib import IOBinding
#    try:
#        source = source.encode(IOBinding.encoding)
#    except UnicodeError:
#        self.tkconsole.resetoutput()
#        self.write("Unsupported characters in input\n")
#        return

I uncommented those lines, removed the `assert` and deleted __pycache__ for fresh bytecode but the error keeps happening.

This function `runsource()` is only called after the return key is pressed so the bug was introduced on another part of the program.

I'll search further but it's hard to do that without traceback of the error.

(Maybe `runit()` is the problem because it seems to build the line and call `runsource(line)`)

------
PS: @Terry Reedy
That looks nice to have different lengths for chars but what will be the impact on performance? Indexing will still be in constant time?

Just to complete my monologue:
Here's the traceback from running IDLE in cmd line.


C:\Python32\Lib\idlelib>python -i idle.py
Traceback (most recent call last):
  File "idle.py", line 11, in <module>
    idlelib.PyShell.main()
  File "C:\Python32\Lib\idlelib\PyShell.py", line 1429, in main
    root.mainloop()
  File "C:\Python32\Lib\tkinter\__init__.py", line 1009, in mainloop
    self.tk.mainloop(n)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-2: invalid continuation byte


Not much meaningful but is better than nothing... Looks like some traceback is missing, and this one points to tkinter.

[Yes, indexing will still be O(1), though I personally consider that less important than most make it to be. Consistency across platforms and total time and space performance of typical apps should be the concern. There is ongoing work on improving the new implementation. Some operations already take less space and run faster.]

The traceback may very well be helpful. It implies that copying a supplemental char does not produce proper utf-8 encoded bytes. Or if it does, tkinter (or tk underneath it) does not recognize them. But then the problem should be the initial byte, not the continuation bytes, which are the same for all chars and which all have 10 for their two high order bits. See
https://secure.wikimedia.org/wikipedia/en/wiki/Utf-8
for a fuller explanation.

Line 1009 is the definition of Misc.mainloop(). I believe self.tk represents the embedded tcl interpreter, which is a black box from Python's viewpoint. Perhaps we should wrap the call with

try:
  self.tk.mainloop(n)
except Exception as e:
  <print error message with all info attached to e before exiting>

This should catch any miscellaneous crashes which are not otherwise caught and maybe turn the crash issues into bug reports -- the same way that running from the command line did. (It will still be good to catch what we can at error sites and give better, more specific messages.) (What I am not familiar with is how the command line interpreter might turn a tcl error into a python exception and why IDLE does not.)

When I copy '𐒢' and paste into the command line interpreter or Notepad++, I get '??'. I am guessing that ?? represent a surrogate pair and that Windows separately encodes each. The result would be 'illegal' utf-8 with an illegal continuation chars. An application can choose to decode the 'illegal' utf-8 -- or not. Python can when "errors='surrogate_escape" (or something like that) is specified. It might be possible to access the raw undecoded bytes of the clipboard with the third party pythonwin module. I do not know if there is anyway to do so with tk.

I wonder if tcl is calling back to Python for decoding and whether there was a change in the default for errors or the callback specification that would explain a change from 2.7 to 3.2.

Ezio, do you know anything about these speculations?

Thanks for the additional investigation.  You don't see more in the traceback because the exception is occurring in the _tkinter C glue layer.  I am able to reproduce the problem on some other platforms as well (e.g. Python 3.x on OS X with Carbon Tk 8.4).  More later.

> Ezio, do you know anything about these speculations?

Assuming that the non-BMP character is represented with two surrogates (\ud801\udca2) and that _tkinter tries to decode them independently, the error message ("invalid continuation byte") would be correct.

Python 2 UTF-8 codec is more permissive and allows encoding/decoding of surrogates (this might also explain why it works on Python 2): 
>>> u'\ud801'.encode('utf-8')
'\xed\xa0\x81'
>>> '\xed\xa0\x81'.decode('utf-8')
u'\ud801'

But on Python 3, trying to decode that results in an error:
>>> b'\xed\xa0\x81'.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid continuation byte

> But then the problem should be the initial byte, not the continuation
> bytes, which are the same for all chars and which all have 10 for
> their two high order bits.

While it's true that all continuation bytes have the first two bits equal to '10', the opposite is not always true.  Some start bytes have additional restrictions on the continuation bytes.  For example, even if the first two bits of 0xA0 (0b10100000) are '10', the valid continuation bytes for a sequence starting with 0xED are restricted to the range 80..9F.

The fact that
>>> '\U000104a2'
'𐒢'
works is because the input is all ASCII, so the decoding doesn't fail.


> [...]
> This should catch any miscellaneous crashes which are not otherwise
> caught and maybe turn the crash issues into bug reports -- the same
> way that running from the command line did.

Having some "safe net" to catch all the unhandled exceptions seems like a good idea.  This won't work in case of segfaults, but it's still better than nothing.  I'm not sure what you mean with "turn them into bug reports" though.

This can also be reproduced by doing:
>>> print('\U000104a2'[0])
���
and then copy/pasting the lone surrogate.
The traceback is:
  [...]
  File "C:\Programs\Python32\Lib\tkinter\__init__.py", line 1009, in mainloop
    self.tk.mainloop(n)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid continuation byte

> I'm not sure what you mean with "turn them into bug reports" though.

In about the last month, there have been, I think, 4 reports about IDLE crashing (quitting unexpectedly with no error traceback). I would consider it preferable if it quit with an error traceback that gave as much info as available, or if there is none, just said "IDLE has met an unexpected problem.", perhaps followed by something like "Pleaee note the circumstances and make a report of the tracker is there is none already."

> I would consider it preferable if it quit 

Note that if we catch the error there might be no reason for IDLE to quit (unless the error left IDLE in some invalid state).

> with an error traceback that gave as much info as available,

That might scare newbies away.

> or if there is none, just said "IDLE has met an unexpected problem.", 

So this might be better for all the cases.

> perhaps followed by something like "Pleaee note the circumstances and
> make a report of the tracker is there is none already."

The first message could offer a "Report the problem" option that links to the tracker.  In theory we could also have a way to auto-fill the tracker issue, but that might lead to duplicates.

Just to be sure we're talking about the same thing here, my understanding is that the "missing traceback" issues referred to here are only an issue when IDLE is run as a stand-alone GUI program, such as can be done on Windows and with the OS X IDLE.app.  In that case, the standard Python tracebacks from the interpreter written to stderr are not readily visible to the user.  In the OS X IDLE.app case it does get captured in a system log. I'm not sure if that happens anywhere in the Windows cases.  If IDLE is started from a terminal window or console window where stderr is displayed, this is not an issue.

But I think further discussion about proposed improvements to IDLE diagnostics could be useful but it is not germane to the specific bug here.  It should be carried out elsewhere, possibly resulting in a feature request.

Reassigning to Andrew to investigate solution similar to the one used in Issue14200.

(Oops, wrong assignment!)

Issue13582 deals with the IDLE error feedback.

Issue14200 has a patch to fix this problem.

Not sure. Let me to investigate the problem deeper.

I'm experiencing a similar issue. Fresh install of 3.3 today from the .msi installer on the web site, identifies itself as:
Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32

To reproduce: Copy and paste this character into IDLE. 𝐇

C:\Python33>.\python -m idlelib.idle
Traceback (most recent call last):
  File "C:\Python33\lib\runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python33\lib\runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "C:\Python33\lib\idlelib\idle.py", line 11, in <module>
    idlelib.PyShell.main()
  File "C:\Python33\lib\idlelib\PyShell.py", line 1477, in main
    root.mainloop()
  File "C:\Python33\lib\tkinter\__init__.py", line 1038, in mainloop
    self.tk.mainloop(n)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-2: invalid continuation byte


(Incidentally, there appears to be a slight difference depending on whether I copy the character in Chrome or Firefox. IDLE terminates the same way, but a Latin-1 app sees the character from Firefox as a letter, but the same thing from Chrome is two question marks (presumably the surrogates).)

Same on 64-bit 3.3. Changed title since 3.3 is no longer '16-bit build'.

A simplest solution is to raise a TclError instead of ValueError for non-BMP characters. This should not break any existing code, because a user code should be ready to catch a TclError in any case. Here is a patch.

A more complicated solution is to add ValueError to any catch of TclError. And this will fix only IDLE, user programs should fix self every.

Also we can silently encode non-BMP characters for Tcl with UTF-16 (and decode a result back). This can cause some subtle errors with shifted indices however.

Ping.

Ping.

LGTM.  The patch does prevent the crash in IDLE which is certainly an improvement until such time as someone investigates having Tk/tkinter fully support non-BMP characters.

@Ned Deily
Tk, at least on my system, doesn't render Unicode characters, even within BMP correctly but the characters are kept (cut-and-paste works correctly)
What you mean by "support".

The characters tk can render depends on the font you tell it to use. On my Windows IDLE, I have Options Font Face set to Lucida Sans Unicode, though I am not sure what has the widest coverage. This page
https://www.microsoft.com/typography/fonts/font.aspx?FMID=1263
only mentions West Asian, but I seem to get more than that.

The font used shouldn't affect the errors.  Usually if a glyph is missing in the current font, either a placeholder (usually a box) is showed instead or the missing glyph is taken from another font (if possible).

If you still want to do some tests, you can take a look at http://en.wikipedia.org/wiki/List_of_Unicode_fonts#Unicode_fonts

Also, there are differences in behavior among the various flavors of Tk.  I know of at least four main flavors in use by current Python builds:  Unix X11-based Tk 8.5, Windows Tk 8.5, OS X Cocoa Tk 8.5, OS X Carbon Tk 8.4.  Some third-party distributors are starting to supply Tk 8.6, in its various flavors, now that 8.6 has been released. Each flavor has various build options and features to fit in with its host o/s environment.  This makes testing tkinter issues *interesting*.

Tkinter is not compatible with Tcl/Tk 8.6 yet (issue16809).

Serhiy, I'm aware of that; regardless, Tk 8.6 is starting to be used out in the field with tkinter.

I have set it to "Ubuntu", which supports the Unicode characters. Maybe
Tkinter doesn't work properly with all the fonts.

On 16 February 2013 01:57, Terry J. Reedy <report@bugs.python.org> wrote:

>
> Terry J. Reedy added the comment:
>
> The characters tk can render depends on the font you tell it to use. On my
> Windows IDLE, I have Options Font Face set to Lucida Sans Unicode, though I
> am not sure what has the widest coverage. This page
> https://www.microsoft.com/typography/fonts/font.aspx?FMID=1263
> only mentions West Asian, but I seem to get more than that.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue13153>
> _______________________________________
>

Serhiy, I think your patch is ready to commit and close this issue as it prevents the crash.  A test would be nice if a reliable test could be devised without too much effort but it's not mandatory, IMO.  Any tangential issues or more complex solutions can be pursued in other issues.

New changeset bb5a8564e186 by Serhiy Storchaka in branch '2.7':
Issue #13153: Tkinter functions now raise TclError instead of ValueError when
http://hg.python.org/cpython/rev/bb5a8564e186

New changeset 9904f245c3f0 by Serhiy Storchaka in branch '3.2':
Issue #13153: Tkinter functions now raise TclError instead of ValueError when
http://hg.python.org/cpython/rev/9904f245c3f0

New changeset 38bb2a46692e by Serhiy Storchaka in branch '3.3':
Issue #13153: Tkinter functions now raise TclError instead of ValueError when
http://hg.python.org/cpython/rev/38bb2a46692e

New changeset 61993bb9ab0e by Serhiy Storchaka in branch 'default':
Issue #13153: Tkinter functions now raise TclError instead of ValueError when
http://hg.python.org/cpython/rev/61993bb9ab0e

Looks like this issue is closed, but I got IDLE to crash.

On Python 3.3.2, Windows 7, and Tk version 8.5, IDLE crashes when pasting \U0001F382 (Unicode birthday cake character). Below is the version string for the Python I'm running.

Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit (AMD64)] on win32.

According to http://docs.python.org/3.3/whatsnew/changelog.html this issue was fixed in Python 3.3.1 RC 1. Indeed the patch discussed above exists in the cpython 3.3 branch: http://hg.python.org/cpython/file/910ec3471d55/Modules/_tkinter.c. But IDLE still crashes, at least for me.

Unfortunately I can't reproduce a crash on Linux. Perhaps this is Windows only issue.

I verified problem continuing problem windows. There is only a 'crash' when running under pythonw, which does not happenon linux, as far as I know. When running on a console, the error traceback is the same as in msg145581 (with line numbers altered). Changing some ValueError to a TclError did not affect either UnicodeDecodeError or the problem of error reports crashing Idle because there is no stderr to send them to.

This type of crash is a generic problem addressed by #13582. The solution to that would solve the crash part of this issue, but not the message. I would like to see the 2.x message, possibly improved. Perhaps we should resurrect the 2.x codec for processing text pasted into the shell.

As for text windows,
# -*- coding: utf-8 -*-
print('𐒢')
# ðﾐﾒﾢ
which is not good either.

In 3.3.2, 3.4.0 the traceback says that the invalid continuation byte (immediately, when '𐒢' is pasted) is ED. Snipped version is
  File "F:\Python\dev\py33\lib\tkinter\__init__.py", line 1071, 
    self.tk.mainloop(n)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 1:

As I understand utf-8 from, for instance, the wikipedia article, continuation bytes are 0b10xxxxxx, or A0 to BF and definitely not ED.

0xed is the start byte of a 3 bytes sequence (i.e. a BMP char), and it should be followed by two continuation bytes.

For some reason the traceback you pasted is missing the last part, that might provide some insight.  It could be one of these:
>>> b'\xed'.decode('utf-8') # not enough continuation bytes
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: unexpected end of data
>>> b'\xed\x7f'.decode('utf-8') # not a valid continuation byte
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte

Byte 0, not byte 1, is the start byte, and it should be F0, as in output below. However, I now see "invalid continuation byte'.
In 2.7.5,
# -*- coding: utf-8 -*-
s = b'𐒢'  # output same if uncomment following lines
#s = u'𐒢'.encode('utf-8')  # '𐒢' pasted in from 1st post
#s = u'\U000104a2'.encode('utf-8')  
print(len(s))
for c in s: print(ord(c), hex(ord(c)))
>>> 
4
(240, '0xf0')
(144, '0x90')
(146, '0x92')
(162, '0xa2')

I have no idea how the second pasted byte becomes ED in 3.x.

Attempting to open the file in 3.x results in a broken* 'Untitled' edit window and the following error message in the console.
_tkinter.TclError: character U+104a2 is above the range (U+0000-U+FFFF) allowed by Tcl

* Attempting to close the window either immediately or after entering text results in
AttributeError: 'PyShellEditorWindow' object has no attribute 'extensions'
I have to close the initial python process to get rid of it.

u'\U000104a2' == u'\ud801\udca2' on narrow build.

u'\ud801'.encode('utf-8', 'surrogatepass') == b'\xed\xa0\x81'
u'\udca2'.encode('utf-8', 'surrogatepass') == b'\xed\xb2\xa2'

Hope it will help.

Seems as Tk stores pasted "\U000104a2" as surrogate pair "\ud801\udca2". Then it encoded in UTF-8 as "\xed\xa0\x81\xed\xb2\xa2" end passed to Python. Python converts char* to Unicode object with PyUnicode_FromString() which forbids invalid UTF-8 including encoded surrogates.

Please test proposed patch on Windows.

This bug can be reproduced on Linux too. Just copy and paste illegal UTF-8 sequence. I.e. b'\xed\xb2\x80' or b'\xc0\x80'. My patch works with first example but failed with second. When change the error handler in fromTclStringAndSize() to "replace" it works with all illegal sequences.

Previous patch has a regression, it breaks decoding NUL which Tcl encodes in "modified" UTF-8 as \xc0\x80. However this part of code already broken, because it handles only singular NUL and not a NUL embedded in larger string.

Here is a patch which also fixes decoding NULs from "modified" UTF-8.

Martin, could you please review this patch? This bug affects not only IDLE, but any Tkinter application which uses callbacks with arguments. Encoding/decoding error during converting arguments from Tcl to Python causes immediate finishing Tcl main loop (and normally closing an application).

As far as the patch becomes too complicated, I propose minimalist patch which fixes only this issue. I.e. IDLE will no longer silently closed when paste some unusual text (non-BMP on Windows or illegal UTF-8 on Linux). It fixes converting Tcl string to Python string for arguments of Python callback. \xc0\x80 is translated to the NUL character (U+0000) because Tcl uses "modified" UTF-8. All other illegal UTF-8 codes are replaced by the replacement character (U+FFFD).

I completely lost track which problem is being solved here. Is it still "IDLE crashes when pasting non-BMP unicode char on Py3"? If so, how is this patch solving it?

IMO, the issue shouhldn't have been reopened. Instead, a new issue should have started.

Yes, it is still the same issue. The root of issue is in converting strings when passed to Python-implemented callbacks. When a text is pasted in IDLE window, the callback is called (for highlighting). The callback is a command created by Tcl_CreateCommand from PythonCmd. PythonCmd is a wrapper which converts arguments (char*) to Python strings and then pass them to Python command. Arguments are encoded in "modified UTF-8" [1], i.e. the NUL character is represented as \xc0\x80, they can contains other invalid UTF-8 sequences (such as encoded surrogates). When decoding arguments to Python strings are failed, main Tcl loop is broken and IDLE silently closed.

When astral character is pasted on Windows, it first encoded to UTF-16 by Windows, then Tcl encodes every 16-bit surrogate to modified UTF-8. The result is not valid UTF-8. On X Window systems the X selection value usually is UTF-8 encoded (the type is UTF8_STRING), but can contains invalid UTF-8 sequences.

I will open separate issue to fix other bugs related to Tcl <-> Python string conversions. The last patch fixes only initial issue which is most important.

[1] http://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8

Here is updated patch (after committing issue20368 which made main refactoring). It makes PythonCmd never fail due to arguments decoding error.

I am puzzled at the following.  Some reported today that IDLE crashed when pasting the Snake emoji U+1F40D 🐍.  I copied from Thunderbird and pasted in IDLE on Win10, with same UnicodeDecodeError as before.  I then ran this simple code

from tkinter import *  # 3.4, 3.5, 2.7 with Tkinter
root = Tk()
text = Text(root)
text.pack()
text.focus_set()  # required to work
root.mainloop()

pasted the char there, and to my surprise, a black & white version of the snake appeared. How?  I thought tk does not support astral chars? I copied from the Text window to paste above, where it is green for me.

There is no the Snake emoji in my font, I use the Cat Face emoji U+1F431 🐱 (\xf0\x9f\x90\xb1 in UTF-8, \x3d\xd8\x31\xdc in UTF-16LE).

Move cursor or press Backspace. I had needed to press Left 2 times to move cursor to the begin of the line, press Right 4 times to move cursor back to the end of line, and press Backspace 4 times to remove all stuff. What is called "Tk doesn't support astral characters".

Get the text programmically.

>>> text.get('1.0', '1.end')
'ðﾟﾐﾱ'
>>> print(ascii(text.get('1.0', '1.end')))
'\xf0\uff9f\uff90\uffb1'

On Linux the clipboard uses UTF-8, and this symbol is represented by 4-bytes bytestring b'\xf0\x9f\x90\xb1' (that is why Tk sometimes interpret it as 4 characters). When you request the text content as a Unicode, Tcl fails to decode the string from UTF-8 and falls back to Latin1. Due to other bug it extends the sign of some bytes. When you programmically insert the same string back, it will be encoded to b'\xc3\xb0\xef\xbe\x9f\xef\xbe\x90\xef\xbe\xb1' and displayed as 'ðﾟﾐﾱ'.

On Windows the clipboard uses UTF-16LE and you can see different results.

The underlying graphical system can support astral characters, but Tk fails to handle them correctly.