When open a file with characters above the range (U+0000-U+FFFF), IDLE quit without any report. For example, open this file \Lib\test\test_re.py

The below is Traceback info, the last line tells the reason. I just hope IDLE say something before quit, so we can know what happend.

I have checked Python 3.3.5 and 3.4.0, they have the same problem. I didn't find a 3.5 build, so I can't test this problem under 3.5.

=============================================
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Python33\lib\tkinter\__init__.py", line 1489, in __call__
    return self.func(*args)
  File "C:\Python33\lib\idlelib\IOBinding.py", line 186, in open
    flist.open(filename)
  File "C:\Python33\lib\idlelib\FileList.py", line 36, in open
    edit = self.EditorWindow(self, filename, key)
  File "C:\Python33\lib\idlelib\PyShell.py", line 126, in __init__
    EditorWindow.__init__(self, *args)
  File "C:\Python33\lib\idlelib\EditorWindow.py", line 288, in __init__
    if io.loadfile(filename):
  File "C:\Python33\lib\idlelib\IOBinding.py", line 236, in loadfile
    self.text.insert("1.0", chars)
  File "C:\Python33\lib\idlelib\Percolator.py", line 25, in insert
    self.top.insert(index, chars, tags)
  File "C:\Python33\lib\idlelib\UndoDelegator.py", line 81, in insert
    self.addcmd(InsertCommand(index, chars, tags))
  File "C:\Python33\lib\idlelib\UndoDelegator.py", line 116, in addcmd
    cmd.do(self.delegate)
  File "C:\Python33\lib\idlelib\UndoDelegator.py", line 219, in do
    text.insert(self.index1, self.chars, self.tags)
  File "C:\Python33\lib\idlelib\ColorDelegator.py", line 85, in insert
    self.delegate.insert(index, chars, tags)
  File "C:\Python33\lib\idlelib\WidgetRedirector.py", line 104, in __call__
    return self.tk_call(self.orig_and_operation + args)
_tkinter.TclError: character U+1d518 is above the range (U+0000-U+FFFF) allowed by Tcl

When open a file with characters above the range (U+0000-U+FFFF), IDLE quit without any report. For example, open this file C:\Python33\lib\test\test_re.py

The below is Traceback info, the last line tells the reason. I just hope IDLE say something before quit, so we can know what happend.

I have checked Python 3.3.5 and 3.4.0, they have the same problem. I didn't find a 3.5 build, so I can't test this problem under 3.5.

=============================================
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Python33\lib\tkinter\__init__.py", line 1489, in __call__
    return self.func(*args)
  File "C:\Python33\lib\idlelib\IOBinding.py", line 186, in open
    flist.open(filename)
  File "C:\Python33\lib\idlelib\FileList.py", line 36, in open
    edit = self.EditorWindow(self, filename, key)
  File "C:\Python33\lib\idlelib\PyShell.py", line 126, in __init__
    EditorWindow.__init__(self, *args)
  File "C:\Python33\lib\idlelib\EditorWindow.py", line 288, in __init__
    if io.loadfile(filename):
  File "C:\Python33\lib\idlelib\IOBinding.py", line 236, in loadfile
    self.text.insert("1.0", chars)
  File "C:\Python33\lib\idlelib\Percolator.py", line 25, in insert
    self.top.insert(index, chars, tags)
  File "C:\Python33\lib\idlelib\UndoDelegator.py", line 81, in insert
    self.addcmd(InsertCommand(index, chars, tags))
  File "C:\Python33\lib\idlelib\UndoDelegator.py", line 116, in addcmd
    cmd.do(self.delegate)
  File "C:\Python33\lib\idlelib\UndoDelegator.py", line 219, in do
    text.insert(self.index1, self.chars, self.tags)
  File "C:\Python33\lib\idlelib\ColorDelegator.py", line 85, in insert
    self.delegate.insert(index, chars, tags)
  File "C:\Python33\lib\idlelib\WidgetRedirector.py", line 104, in __call__
    return self.tk_call(self.orig_and_operation + args)
_tkinter.TclError: character U+1d518 is above the range (U+0000-U+FFFF) allowed by Tcl

See #13153.

Accidentally set to pending I take it.

Yes, this is very similar to issue13153. Both these issues can have same solution or can have different solutions. This issue relates to more realistic situation and therefore is more important.

Here is simple and almost working solution for this issue. Unfortunately it works incorrectly when astral characters are encountered in raw string literals. More mature solution should parse sources and convert raw string literals containing astral characters to non-raw string literals. But this will not work with invalid Python files and non-Python files.

I afraid this issue has not perfect solution. The question is which imperfect solution and compromise we will decided enough acceptable.

I suggest don't change the content of file, just give a message such as:

IDLE can't display non-BMP character (codepoint above 0xFFFF).
A non-BMP character found in Line 23, position 8 of aaaa.py, please open this file with other editor.

I wrote this code, but I don't know how to make a patch.

Insert these codes in C:\Python34\Lib\idlelib\IOBinding.py
Around line 234, before this line:
self.text.delete("1.0", "end")


        # check non-bmp characters
        line_count = 1
        position_count = 1
        for char in chars:
            if char == '\n':
                line_count += 1
                position_count = 1
            if ord(char) > 0xFFFF:
                nonbmp_msg = ("IDLE can't display non-BMP characters "
                              "(codepoint above 0xFFFF).\n"
                              "A non-BMP character found at line %d, "
                              "position %d of file %s, codepoint 0x%X.\n"
                              "Please open this file with another editor.")
                tkMessageBox.showerror("non-BMP character",
                                        nonbmp_msg %
                                       (line_count, position_count,
                                        filename, ord(char)),
                                       parent=self.text)
                return False
            position_count += 1

Changing the second "if" to "elif" is better.

I'm sorry, I have never submitted patch.
If somebody gives a hand, feel free to modify those codes.

See https://docs.python.org/devguide/patch.html

Feel free to modify this patch.

nonbmp_except_check_v2.patch changes character numbers to 0-based, same as IDLE.

Quote from www.tkdocs.com :
"for historical conventions related to how programmers normally refer to lines and characters, line numbers are 1-based, and character numbers are 0-based."

Tk Text (and other widgets, but Text is the main issue) has two display problems: astral chars and long lines (over a thousand chars, say).  These problems can manifest in various places: file names, shell input (keyboard or clipboard), shell output, editor input (keyboard, clipboard, or file).  IDLE needs to take more control over what is displayed to work around both problems.

Tk Text also has a display feature: substring tagging.  I have been heistant to simple replace astral chars with their \U000hhhhh expansion because of the aliasing problem: in shell output, for instance, the user would not know if the program wrote 1 char or 10.  It would also be impossible to know if a reverse transformation might be needed.  Tagging astral expansions would solve both problems.

import re

astral = re.compile(r'([^\x00-\uffff])')
s = 'X\U00011111Y\U00011112\U00011113Z'
for i, ss in enumerate(re.split(astral, s)):
    if not i%2:
        print(ss, end='')
    else:
        print(r'\\U%08x' % ord(ss), end='')
# prints
X\\U00011111Y\\U00011112\\U00011113Z

Now replace print with test.insert, with an 'astral' tag for the second.  tk will not double '\'s.  Astral tag could switch, for instance, to underline version of current font.  This should work with any color scheme.

[Separate but related issue: augment Format or context menu with functions to convert between literal char, escape string, and name representation (using unicodedatabase).]

Fixed by PR 16545 (see issue13153).

As noted on #13153, files with astral chars can now be read without an exception, but the presence of astral chars messes up editing text that follows at least on the same line by misplacing the cursor.  I will open a new issue about replacing such with \U escapes.