Title: make reindent failed.
Type: behavior Stage: resolved
Components: Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Mariatta, cheryl.sabella, corona10, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2017-04-20 03:17 by corona10, last changed 2018-02-12 18:17 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1207 merged Mariatta, 2017-04-20 05:22
PR 1208 merged Mariatta, 2017-04-20 06:00
PR 1209 merged Mariatta, 2017-04-20 06:01
PR 5637 merged serhiy.storchaka, 2018-02-12 12:29
Messages (10)
msg291935 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2017-04-20 03:17
When I try to `make reindent`

It was failed with this messages.

./python.exe ./Tools/scripts/ -r ./Lib
Traceback (most recent call last):
  File "/Users/corona10/cpython/Lib/", line 404, in find_cookie
    codec = lookup(encoding)
LookupError: unknown encoding: uft-8

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./Tools/scripts/", line 329, in <module>
  File "./Tools/scripts/", line 101, in main
  File "./Tools/scripts/", line 115, in check
  File "./Tools/scripts/", line 115, in check
  File "./Tools/scripts/", line 121, in check
    encoding, _ = tokenize.detect_encoding(f.readline)
  File "/Users/corona10/cpython/Lib/", line 433, in detect_encoding
    encoding = find_cookie(first)
  File "/Users/corona10/cpython/Lib/", line 412, in find_cookie
    raise SyntaxError(msg)
SyntaxError: unknown encoding for './Lib/test/': uft-8
make: *** [reindent] Error 1

msg291948 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2017-04-20 05:59
New changeset 58f3c9dc8f5626abe09ac9738c34f6ba99ce2972 by Mariatta in branch 'master':
bpo-30109: Fix (GH-1207)
msg291949 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-04-20 06:13
I don't know whether the fix for 2.7 is needed. The interface of tokenize is different in 2.7. Just run the new test and see if it fails.
msg291973 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2017-04-20 14:12
New changeset 430020500c56259d31e07d32771f017952bad4e0 by Mariatta in branch '3.6':
[3.6] bpo-30109: Fix (GH-1207) (GH-1208)
msg291974 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2017-04-20 14:12
New changeset 60d27f42a07d0b9d4a2c3fb160cac1683e91f196 by Mariatta in branch '3.5':
[3.5] bpo-30109: Fix (GH-1207) (GH-1209)
msg311551 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2018-02-03 13:42
Does this just need to be cherry-picked to 2.7 to be able to close it?
msg311555 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2018-02-03 14:12
Thanks for the ping. Almost forgotten about this issue.

This can't be cherry-picked cleanly. I need to first backport the new testcase to 2.7, and see if it passes. If the test failed in 2.7 then I need to work on actually backporting the change.

I should have time to do it later tonight.

msg312008 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2018-02-11 18:38
make reindent is still failing on 2.7 branch, but I haven't had chance to look into the fix. Not really sure if we want to bother?
Anyways, I've unassigned myself so if someone else want to look into it, they can. Not sure if it's worth the effort. (not a lot of people use it?)
msg312042 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-12 12:14
2.7 needs different solution.

The side effect of issue10639 in 2.7 was changing input and output files for the reindenter to Unicode files. This was not correct. Standard streams stdin/stdout in 2.7 are binary files, and the tokenizer needs the stream of bytes.

The proposed PR makes input and output files for the reindenter binary files again, and implements detecting line separators for issue10639.

After fixing Tools/scripts/ running `make reindent` fixed Lib/email/
msg312068 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-12 18:16
New changeset 17cec70a38b297779b8fd3f081fb041d45ae1dff by Serhiy Storchaka in branch '2.7':
bpo-30109: Fix for non-ASCII files. (#5637)
Date User Action Args
2018-02-12 18:17:44serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-02-12 18:16:45serhiy.storchakasetmessages: + msg312068
2018-02-12 12:29:06serhiy.storchakasetkeywords: + patch
stage: backport needed -> patch review
pull_requests: + pull_request5437
2018-02-12 12:14:51serhiy.storchakasetmessages: + msg312042
2018-02-11 20:30:26serhiy.storchakasetassignee: serhiy.storchaka
2018-02-11 18:38:41Mariattasetassignee: Mariatta -> (no value)
messages: + msg312008
2018-02-03 14:12:12Mariattasetmessages: + msg311555
2018-02-03 13:42:56cheryl.sabellasetnosy: + cheryl.sabella
messages: + msg311551
2017-04-20 14:12:53Mariattasetmessages: + msg291974
2017-04-20 14:12:41Mariattasetmessages: + msg291973
2017-04-20 06:13:19serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg291949
2017-04-20 06:04:52Mariattasetstage: patch review -> backport needed
versions: + Python 2.7, Python 3.5, Python 3.6
2017-04-20 06:01:10Mariattasetpull_requests: + pull_request1331
2017-04-20 06:00:57Mariattasetpull_requests: + pull_request1330
2017-04-20 05:59:22Mariattasetmessages: + msg291948
2017-04-20 05:24:33Mariattasetassignee: Mariatta

nosy: + Mariatta
stage: patch review
2017-04-20 05:22:24Mariattasetpull_requests: + pull_request1329
2017-04-20 03:24:40corona10settype: behavior
versions: + Python 3.7
2017-04-20 03:17:36corona10create