classification
Title: idle test fails at least on the 3.6 branch
Type: behavior Stage: resolved
Components: IDLE Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: cheryl.sabella, doko, miss-islington, serhiy.storchaka, terry.reedy
Priority: normal Keywords: easy, patch

Created on 2018-02-12 10:43 by doko, last changed 2018-02-12 22:28 by terry.reedy. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5639 merged terry.reedy, 2018-02-12 16:08
PR 5642 merged miss-islington, 2018-02-12 19:58
PR 5643 merged miss-islington, 2018-02-12 19:59
Messages (10)
msg312037 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2018-02-12 10:43
seen with the 3.6 branch 20180212, last known succeeding test is the 3.6.4 release.

test test_idle failed -- Traceback (most recent call last):
  File "/home/packages/python/3.6/python3.6-3.6.4/Lib/idlelib/idle_test/test_help_about.py", line 78, in test_file_buttons
    self.assertEqual(f.readline().strip(), get('1.0', '1.end'))
  File "/home/packages/python/3.6/python3.6-3.6.4/build-debug/../Lib/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1540: ordinal not in range(128)
msg312059 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-02-12 15:35
What system?  test_idle passes on Windows.  Unfortunately, none of the *nix or Mac buildbots run tkinter and hence no tests that require a live tk window, and this one does.  Cheryl, what do you see on your linux?

Byte 0xc3 in position 1540 is the first byte of the utf-8 encoding of the 'ö' in Löwis on line 27 of CREDITS.txt.  So on your system, readline must be reading *and decoding* more of the file.

I am puzzled that the test fails now if it passed with the 3.6.4 release, as it and the textfile have not been touched since last summer.
The files are the same on all 3 python versions.

In any case, I will add encoding='utf-8'to open.  I will also use subtest to identify failing cases.  I will need you to verify the fix on your system.
msg312060 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-12 15:54
The failure is reproduced on Posix locale. Files are open with bare open() which uses the locale encoding by default. Which is ASCII in this case.
msg312064 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-02-12 17:15
I just discovered that the default for me is some version of latin1, not ascii, so that utf-8 'ö' is read as 'ö'.  Hence, no UnicodeDecodeError even if 'ö' is being read and decoded by the initial readline call.  I presume the same is true on current Windows buildbots.  Hence, it was a bug to not include an encoding in the test file open calls.  (The widget itself *does* use explicit 'ascii' or 'utf-8' encodings for open().)
msg312065 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2018-02-12 17:42
I've been able to run the IDLE test suite on Linux.

As far as this issue, there's also a readline in test_textview that you may want to add the encoding to, even though it's only reading __file__.
msg312066 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-02-12 18:04
I reproduced the exact failure by temporarily replacing 'utf-8' with 'ascii' in my clone.  So I don't need anyone else to verify the fix.
msg312073 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-02-12 19:58
New changeset f34e03ec0ea6a4cef8d966087c77e616c4a5893b by Terry Jan Reedy in branch 'master':
bpo-32826: Add "encoding=utf-8" to open() in idle_test/test_help_about. (GH-5639)
https://github.com/python/cpython/commit/f34e03ec0ea6a4cef8d966087c77e616c4a5893b
msg312075 - (view) Author: miss-islington (miss-islington) Date: 2018-02-12 20:35
New changeset 38b4dd7f83fbdce6fe970088cd89b80f074a13f6 by Miss Islington (bot) in branch '3.7':
bpo-32826: Add "encoding=utf-8" to open() in idle_test/test_help_about. (GH-5639)
https://github.com/python/cpython/commit/38b4dd7f83fbdce6fe970088cd89b80f074a13f6
msg312077 - (view) Author: miss-islington (miss-islington) Date: 2018-02-12 20:53
New changeset 46daf39453023767eef3b6876d4a34996e124397 by Miss Islington (bot) in branch '3.6':
bpo-32826: Add "encoding=utf-8" to open() in idle_test/test_help_about. (GH-5639)
https://github.com/python/cpython/commit/46daf39453023767eef3b6876d4a34996e124397
msg312086 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-02-12 22:28
Serhiy, thanks for comment and review.
Cheryl, #32837 expands upon your comment.
History
Date User Action Args
2018-02-12 22:28:44terry.reedysetstatus: open -> closed
resolution: fixed
messages: + msg312086

stage: patch review -> resolved
2018-02-12 20:53:53miss-islingtonsetmessages: + msg312077
2018-02-12 20:35:00miss-islingtonsetnosy: + miss-islington
messages: + msg312075
2018-02-12 19:59:35miss-islingtonsetpull_requests: + pull_request5443
2018-02-12 19:58:34miss-islingtonsetstage: commit review -> patch review
pull_requests: + pull_request5442
2018-02-12 19:58:31terry.reedysetmessages: + msg312073
2018-02-12 18:04:23terry.reedysetmessages: + msg312066
2018-02-12 17:42:26cheryl.sabellasetmessages: + msg312065
2018-02-12 17:15:20terry.reedysetmessages: + msg312064
components: - Tests
stage: patch review -> commit review
2018-02-12 16:08:19terry.reedysetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request5439
2018-02-12 15:54:11serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg312060

components: + Tests
keywords: + easy
2018-02-12 15:35:00terry.reedysetversions: + Python 3.7, Python 3.8
nosy: + cheryl.sabella

messages: + msg312059

type: behavior
stage: needs patch
2018-02-12 10:43:23dokocreate