This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Shell restart when error message contains non-BMP characters
Type: behavior Stage: resolved
Components: IDLE Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder: IDLE shows traceback when printing non-BMP character
View: 22742
Assigned To: terry.reedy Nosy List: TheMathsGod, miss-islington, serhiy.storchaka, terry.reedy
Priority: normal Keywords: patch

Created on 2019-04-22 17:15 by TheMathsGod, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 16583 merged serhiy.storchaka, 2019-10-04 15:09
PR 16649 merged miss-islington, 2019-10-08 11:32
PR 16650 merged miss-islington, 2019-10-08 11:32
Messages (7)
msg340662 - (view) Author: (TheMathsGod) Date: 2019-04-22 17:15
When attempting to raise an error with a message containing non-BMP characters (Unicode ordinals above U+0xFFFF), the python shell displays a long traceback including several UnicodeEncodeErrors and then restarts. Example:

>>> raise Exception('\U0001f603')
Traceback (most recent call last):
  File "<pyshell#84>", line 1, in <module>
    raise Exception('\U0001f603')
Traceback (most recent call last):
  File "<pyshell#84>", line 1, in <module>
    raise Exception('\U0001f603')
Traceback (most recent call last):
  File "D:\Python37\lib\idlelib\run.py", line 474, in runcode
    exec(code, self.locals)
  File "<pyshell#84>", line 1, in <module>
Traceback (most recent call last):
  File "D:\Python37\lib\idlelib\run.py", line 474, in runcode
    exec(code, self.locals)
  File "<pyshell#84>", line 1, in <module>
Exception: 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Python37\lib\idlelib\run.py", line 144, in main
    ret = method(*args, **kwargs)
  File "D:\Python37\lib\idlelib\run.py", line 486, in runcode
    print_exception()
  File "D:\Python37\lib\idlelib\run.py", line 234, in print_exception
    print_exc(typ, val, tb)
  File "D:\Python37\lib\idlelib\run.py", line 232, in print_exc
    print(line, end='', file=efile)
  File "D:\Python37\lib\idlelib\run.py", line 362, in write
    return self.shell.write(s, self.tags)
  File "D:\Python37\lib\idlelib\rpc.py", line 608, in __call__
    value = self.sockio.remotecall(self.oid, self.name, args, kwargs)
  File "D:\Python37\lib\idlelib\rpc.py", line 220, in remotecall
    return self.asyncreturn(seq)
  File "D:\Python37\lib\idlelib\rpc.py", line 251, in asyncreturn
    return self.decoderesponse(response)
  File "D:\Python37\lib\idlelib\rpc.py", line 271, in decoderesponse
    raise what
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 11-11: Non-BMP character not supported in Tk

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Python37\lib\idlelib\run.py", line 158, in main
    print_exception()
  File "D:\Python37\lib\idlelib\run.py", line 234, in print_exception
    print_exc(typ, val, tb)
  File "D:\Python37\lib\idlelib\run.py", line 220, in print_exc
    print_exc(type(context), context, context.__traceback__)
  File "D:\Python37\lib\idlelib\run.py", line 232, in print_exc
    print(line, end='', file=efile)
  File "D:\Python37\lib\idlelib\run.py", line 362, in write
    return self.shell.write(s, self.tags)
  File "D:\Python37\lib\idlelib\rpc.py", line 608, in __call__
    value = self.sockio.remotecall(self.oid, self.name, args, kwargs)
  File "D:\Python37\lib\idlelib\rpc.py", line 220, in remotecall
    return self.asyncreturn(seq)
  File "D:\Python37\lib\idlelib\rpc.py", line 251, in asyncreturn
    return self.decoderesponse(response)
  File "D:\Python37\lib\idlelib\rpc.py", line 271, in decoderesponse
    raise what
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 11-11: Non-BMP character not supported in Tk

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\Python37\lib\idlelib\run.py", line 162, in main
    traceback.print_exception(type, value, tb, file=sys.__stderr__)
  File "D:\Python37\lib\traceback.py", line 105, in print_exception
    print(line, file=file, end="")
  File "D:\Python37\lib\idlelib\run.py", line 362, in write
    return self.shell.write(s, self.tags)
  File "D:\Python37\lib\idlelib\rpc.py", line 608, in __call__
    value = self.sockio.remotecall(self.oid, self.name, args, kwargs)
  File "D:\Python37\lib\idlelib\rpc.py", line 220, in remotecall
    return self.asyncreturn(seq)
  File "D:\Python37\lib\idlelib\rpc.py", line 251, in asyncreturn
    return self.decoderesponse(response)
  File "D:\Python37\lib\idlelib\rpc.py", line 271, in decoderesponse
    raise what
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 11-11: Non-BMP character not supported in Tk

=============================== RESTART: Shell ===============================
>>>

I presume the error is caused by Tk being unable to display the characters in the error message, but being forced to anyway by the traceback, causing a series of UnicodeEncodeErrors. Perhaps the error handler should use repr() or similar methods to convert the message into a displayable form?
msg340677 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-04-22 19:09
You are correct about reason.  This is essentially a duplicate of #22742, but I am leaving it open for now because the behavior is worse and there is additional information in the tracebacks.  The lines for IDLE internals are normally removed, but the exception during the exception prevented that.
msg353932 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-10-04 12:00
Mostly fixed by PR 16545 (see issue13153). The original example is now passed.

>>> raise Exception('\U0001f603')
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    raise Exception('\U0001f603')
Exception: 😃
>>> 

But you can still cause a restart of the shell by using surrogate characters:

>>> raise Exception('\ud800')
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    raise Exception('\ud800')
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    raise Exception('\ud800')

================================ RESTART: Shell ================================
>>>
msg353968 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-10-04 18:28
As with #22742, the problem, without PR 16583, only occurs with *nix.  On Windows, the traceback ends with "Exception: �" and a prompt is printed.
msg354187 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-10-08 11:32
New changeset b690a2759e62d9ee0b6ea1b20e8f7e4b2cdbf8bb by Serhiy Storchaka in branch 'master':
bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583)
https://github.com/python/cpython/commit/b690a2759e62d9ee0b6ea1b20e8f7e4b2cdbf8bb
msg354189 - (view) Author: miss-islington (miss-islington) Date: 2019-10-08 11:51
New changeset a1f45008f190b0d340ec0eac623f05ef98e6b4c9 by Miss Islington (bot) in branch '3.8':
bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583)
https://github.com/python/cpython/commit/a1f45008f190b0d340ec0eac623f05ef98e6b4c9
msg354191 - (view) Author: miss-islington (miss-islington) Date: 2019-10-08 11:52
New changeset 4f962ecfa1aadc45facc250841208f6dd2ce690f by Miss Islington (bot) in branch '3.7':
bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583)
https://github.com/python/cpython/commit/4f962ecfa1aadc45facc250841208f6dd2ce690f
History
Date User Action Args
2022-04-11 14:59:14adminsetgithub: 80879
2019-10-08 12:03:42serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-10-08 11:52:52miss-islingtonsetmessages: + msg354191
2019-10-08 11:51:19miss-islingtonsetnosy: + miss-islington
messages: + msg354189
2019-10-08 11:32:46miss-islingtonsetpull_requests: + pull_request16233
2019-10-08 11:32:39miss-islingtonsetpull_requests: + pull_request16232
2019-10-08 11:32:28serhiy.storchakasetmessages: + msg354187
2019-10-04 18:28:33terry.reedysetmessages: + msg353968
versions: + Python 3.9
2019-10-04 15:09:06serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request16174
2019-10-04 12:00:20serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg353932
2019-04-22 19:09:13terry.reedysetsuperseder: IDLE shows traceback when printing non-BMP character
messages: + msg340677
versions: + Python 3.8
2019-04-22 17:15:57TheMathsGodcreate