classification
Title: CLI error when .python_history contains unicode characters
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.7
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: armandas, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2018-12-20 06:06 by armandas, last changed 2018-12-20 07:12 by serhiy.storchaka. This issue is now closed.

Messages (2)
msg332179 - (view) Author: Armandas (armandas) Date: 2018-12-20 06:05
OS: Windows 10
Python version: Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:05:16) [MSC v.1915 32 bit (Intel)] on win32

Traceback:
Failed calling sys.__interactivehook__
Traceback (most recent call last):
  File "C:\Users\owner\AppData\Local\Programs\Python\Python37-32\lib\site.py", line 439, in register_readline
    readline.read_history_file(history)
  File "C:\Users\owner\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyreadline\rlmain.py", line 165, in read_history_file
    self.mode._history.read_history_file(filename)
  File "C:\Users\owner\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyreadline\lineeditor\history.py", line 82, in read_history_file
    for line in open(filename, 'r'):
  File "C:\Users\owner\AppData\Local\Programs\Python\Python37-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 2: character maps to <undefined>

How to reproduce:
On a Windows machine, add the following line to your .python_history:

"あ".isalpha()

I believe the issue stems from the fact that the history file is opened with the "default" encoding, which on windows is cp1252.
msg332185 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-12-20 07:09
pyreadline is a third-party package. Refer to issue 55 at its GitHub repo:

https://github.com/pyreadline/pyreadline/issues/55
History
Date User Action Args
2018-12-20 07:12:16serhiy.storchakasetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2018-12-20 07:09:07eryksunsetstatus: open -> closed

nosy: + eryksun
messages: + msg332185

resolution: third party
stage: resolved
2018-12-20 06:06:01armandascreate