classification
Title: Chinese characters issue with input() function
Type: behavior Stage: patch review
Components: macOS Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Valentin Zhao, vstinner, xiang.zhang
Priority: normal Keywords: patch

Created on 2018-05-30 09:36 by Valentin Zhao, last changed 2018-06-01 10:29 by vstinner.

Files
File name Uploaded Description Edit
input_test.py Valentin Zhao, 2018-05-30 09:36 a file makes you input Chinese characters
Pull Requests
URL Status Linked Edit
PR 7244 merged vstinner, 2018-05-30 14:13
PR 7246 merged vstinner, 2018-05-30 14:50
PR 7293 merged vstinner, 2018-05-31 22:57
PR 7293 merged vstinner, 2018-05-31 22:57
PR 7298 merged vstinner, 2018-06-01 09:09
PR 7301 merged vstinner, 2018-06-01 09:13
PR 7304 merged vstinner, 2018-06-01 09:20
Messages (16)
msg318160 - (view) Author: Valentin Zhao (Valentin Zhao) Date: 2018-05-30 09:36
DIRECTLY run(i.e., python xxx/xxx/input_test.py) the file below and input other Chinese characters, then press backspace to delete them. You will find that you cannot delete them all, and even the cursor in the terminal misplaces and overlaps with the characters.

This issue is only reproduced under Mac OS X, everything is fine with Windows.
msg318169 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-30 11:26
input() uses readline. On macOS, readline is provided by libedit. It may be a bug in libedit which is an external dependency, and so not maintained by Python.
msg318177 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2018-05-30 12:29
I could reproduce the behavior that console output doesn't match what Python gets. I agree with Victor this may not be a Python problem. I could also reproduce the behavior with Ruby reading Chinese characters from console.
msg318179 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-30 12:35
FTR:
> On macOS, readline is provided by libedit.

It *may* be provided by libedit. GNU readline is also commonly used on macOS, although it is not provided by Apple.  The easiest way to tell which is in use is to examine the __doc__ attribute of the readline module:

$ /usr/local/bin/python3.6 -c 'import readline;print(readline.__doc__)'
Importing this module enables command line editing using libedit readline.
$ /macports/bin/python3.6 -c 'import readline;print(readline.__doc__)'
Importing this module enables command line editing using GNU readline.

https://docs.python.org/3/library/readline.html#module-readline
msg318180 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-30 12:38
Xiang Zhang, was that on macOS?  If so, can someone try on one or more Linux systems with a Chinese locale?  I see the same behavior on Linux but I'm not sure that I'm testing it properly.
msg318185 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2018-05-30 13:25
Yes, it's on MacOS. libedit and GNU deadline both have the behavior.  I can see the same behavior on Ubuntu, LANG=en_US.UTF-8.
msg318189 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-30 14:44
So it's not a macOS-specific issue: editing the title et al accordingly
msg318190 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-30 14:49
New changeset f7a19c28bda7b0ed3a440db80202013a86af6850 by Victor Stinner in branch '3.6':
bpo-33692: pythoninfo detect libedit on Python 3.6 (GH-7244)
https://github.com/python/cpython/commit/f7a19c28bda7b0ed3a440db80202013a86af6850
msg318208 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-30 21:36
New changeset 9994eff17f943fb2c190708b12c96fef9838a425 by Victor Stinner in branch '2.7':
bpo-33692: pythoninfo detect libedit on Python 2.7 (#7246)
https://github.com/python/cpython/commit/9994eff17f943fb2c190708b12c96fef9838a425
msg318235 - (view) Author: Valentin Zhao (Valentin Zhao) Date: 2018-05-31 02:07
Hello everyone, do we have a walkaround for this issue now? I barely read your discussion.
msg318363 - (view) Author: Valentin Zhao (Valentin Zhao) Date: 2018-06-01 06:38
Hey there! I saw your pull requests to python3.6 and have been merged, so what should we do now? Just update the python to higher a version?
msg318371 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-01 09:03
> Hey there! I saw your pull requests to python3.6 and have been merged, so what should we do now? Just update the python to higher a version?

Oh. My changes are just to add debug traces in the "python3 -m test.pythoninfo" command which is used on buildbots, Travis CI and AppVeyor to ease debug when a test fails.

It's not a fix.
msg318372 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-01 09:04
New changeset ef634b5b26f14d942768069fa23f9791a06ff37e by Victor Stinner in branch 'master':
bpo-33692: pythoninfo detect libedit on Python 3.6 (GH-7293)
https://github.com/python/cpython/commit/ef634b5b26f14d942768069fa23f9791a06ff37e
msg318383 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-01 10:10
New changeset b9afe645c4926b1f086f30ba9bd00671b0b8b55e by Victor Stinner in branch '3.6':
bpo-33692: Update pythoninfo from master (GH-7301)
https://github.com/python/cpython/commit/b9afe645c4926b1f086f30ba9bd00671b0b8b55e
msg318385 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-01 10:10
New changeset 466aae545786d2db5ac27f243c5d77aeb7a5958e by Victor Stinner in branch '3.7':
bpo-33692: Update pythoninfo from master (GH-7298)
https://github.com/python/cpython/commit/466aae545786d2db5ac27f243c5d77aeb7a5958e
msg318390 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-01 10:29
New changeset 0f642620a8a85ef0b5482f253205baf1865e05b2 by Victor Stinner in branch '2.7':
bpo-33692: Update pythoninfo from master (GH-7304)
https://github.com/python/cpython/commit/0f642620a8a85ef0b5482f253205baf1865e05b2
History
Date User Action Args
2018-06-01 10:29:49vstinnersetmessages: + msg318390
2018-06-01 10:10:10vstinnersetmessages: + msg318385
2018-06-01 10:10:01vstinnersetmessages: + msg318383
2018-06-01 09:20:30vstinnersetpull_requests: + pull_request6935
2018-06-01 09:13:16vstinnersetpull_requests: + pull_request6931
2018-06-01 09:09:30vstinnersetpull_requests: + pull_request6927
2018-06-01 09:04:38vstinnersetmessages: + msg318372
2018-06-01 09:03:59vstinnersetmessages: + msg318371
2018-06-01 06:38:59Valentin Zhaosetmessages: + msg318363
2018-05-31 22:57:55vstinnersetpull_requests: + pull_request6921
2018-05-31 22:57:55vstinnersetpull_requests: + pull_request6920
2018-05-31 02:07:07Valentin Zhaosetmessages: + msg318235
2018-05-30 21:36:14vstinnersetmessages: + msg318208
2018-05-30 14:50:52vstinnersetpull_requests: + pull_request6872
2018-05-30 14:49:10vstinnersetmessages: + msg318190
2018-05-30 14:44:35ned.deilysetnosy: - ned.deily
2018-05-30 14:44:14ned.deilysetnosy: - ronaldoussoren

messages: + msg318189
title: Chinese characters issue with input() function under Mac OSX -> Chinese characters issue with input() function
2018-05-30 14:13:43vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request6871
2018-05-30 13:25:24xiang.zhangsetmessages: + msg318185
2018-05-30 12:38:42ned.deilysetmessages: + msg318180
2018-05-30 12:35:31ned.deilysetmessages: + msg318179
2018-05-30 12:29:37xiang.zhangsetnosy: + xiang.zhang
messages: + msg318177
2018-05-30 11:26:56vstinnersetnosy: + vstinner, ronaldoussoren, ned.deily
messages: + msg318169
components: + macOS, - 2to3 (2.x to 3.x conversion tool)
2018-05-30 09:38:58Valentin Zhaosettitle: Chinese characters issue with input() function -> Chinese characters issue with input() function under Mac OSX
2018-05-30 09:36:41Valentin Zhaocreate