classification
Title: lib2to3: support non-ASCII identifiers
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, miss-islington, monson
Priority: normal Keywords: patch

Created on 2018-08-27 06:47 by monson, last changed 2018-09-15 17:53 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 8950 merged monson, 2018-08-27 06:54
PR 9333 merged miss-islington, 2018-09-15 17:32
Messages (3)
msg324148 - (view) Author: monson (monson) * Date: 2018-08-27 06:47
Python 3.0 introduces additional characters from outside the ASCII range (see PEP 3131). see https://docs.python.org/3/reference/lexical_analysis.html#identifiers

But lib2to3 can't tokenize them corretly.
```
$ echo '中 = 1' | python3.7 -m lib2to3.pgen2.tokenize
1,0-1,1:	ERRORTOKEN	'中'
1,2-1,3:	OP	'='
1,4-1,5:	NUMBER	'1'
1,5-1,6:	NEWLINE	'\n'
2,0-2,0:	ENDMARKER	''
```
'中' should be tokenized as NAME instead of ERRORTOKEN.
msg325451 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2018-09-15 17:32
New changeset 10a428b64b3f224e2ccd40ff2afb141b9b3425b1 by Benjamin Peterson (Monson Shao) in branch 'master':
closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950)
https://github.com/python/cpython/commit/10a428b64b3f224e2ccd40ff2afb141b9b3425b1
msg325452 - (view) Author: miss-islington (miss-islington) Date: 2018-09-15 17:53
New changeset 51dbae867e82014f9af89662977e4981463c51e8 by Miss Islington (bot) in branch '3.7':
closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950)
https://github.com/python/cpython/commit/51dbae867e82014f9af89662977e4981463c51e8
History
Date User Action Args
2018-09-15 17:53:02miss-islingtonsetnosy: + miss-islington
messages: + msg325452
2018-09-15 17:32:49miss-islingtonsetpull_requests: + pull_request8756
2018-09-15 17:32:33benjamin.petersonsetstatus: open -> closed

nosy: + benjamin.peterson
messages: + msg325451

resolution: fixed
stage: patch review -> resolved
2018-08-27 08:39:07monsonsetversions: + Python 3.8, - Python 3.7
2018-08-27 06:54:19monsonsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8426
2018-08-27 06:47:47monsoncreate