Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_codecs currently failing on several Windows buildbots #64770

Closed
larryhastings opened this issue Feb 9, 2014 · 11 comments
Closed

test_codecs currently failing on several Windows buildbots #64770

larryhastings opened this issue Feb 9, 2014 · 11 comments
Labels
OS-windows release-blocker tests Tests in the Lib/test dir topic-unicode type-bug An unexpected behavior, bug, or error

Comments

@larryhastings
Copy link
Contributor

BPO 20571
Nosy @malemburg, @loewis, @doerwalter, @birkenfeld, @pitrou, @vstinner, @larryhastings, @benjaminp, @ezio-melotti, @serhiy-storchaka

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-02-09.13:18:55.753>
created_at = <Date 2014-02-09.09:16:48.960>
labels = ['type-bug', 'tests', 'expert-unicode', 'OS-windows', 'release-blocker']
title = 'test_codecs currently failing on several Windows buildbots'
updated_at = <Date 2014-02-23.09:37:34.286>
user = 'https://github.com/larryhastings'

bugs.python.org fields:

activity = <Date 2014-02-23.09:37:34.286>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2014-02-09.13:18:55.753>
closer = 'vstinner'
components = ['Tests', 'Unicode', 'Windows']
creation = <Date 2014-02-09.09:16:48.960>
creator = 'larry'
dependencies = []
files = []
hgrepos = []
issue_num = 20571
keywords = []
message_count = 11.0
messages = ['210732', '210733', '210741', '210743', '210744', '210747', '210751', '210761', '211983', '211984', '211986']
nosy_count = 11.0
nosy_names = ['lemburg', 'loewis', 'doerwalter', 'georg.brandl', 'pitrou', 'vstinner', 'larry', 'benjamin.peterson', 'ezio.melotti', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20571'
versions = ['Python 3.4']

@larryhastings
Copy link
Contributor Author

The Windows buildbots are currently broken due to a codec issue. I populated the "nosy" list based on the "unicode" experts from the Experts Index.

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/4040

test_streamreaderwriter (test.test_codecs.WithStmtTest) ... test test_codecs failed
ok

======================================================================
ERROR: test_readline (test.test_codecs.CP65001Test)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_codecs.py", line 157, in test_readline
    self.assertEqual(readalllines("".join(vw), True), "|".join(vw))
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_codecs.py", line 136, in readalllines
    line = reader.readline(size=size, keepends=keepends)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\codecs.py", line 548, in readline
    data = self.read(readsize, firstline=True)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\codecs.py", line 494, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'CP_UTF8' codec can't decode bytes in position 0--1: No mapping for the Unicode character exists in the target code page.

@larryhastings larryhastings added release-blocker tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Feb 9, 2014
@larryhastings
Copy link
Contributor Author

Note that this appears to be in Windows-specific code ("CP_UTF8"), rather than being cross-platform code which happens to only fail on Windows. So we need someone who does both Windows and Unicode.

@vstinner
Copy link
Member

vstinner commented Feb 9, 2014

It looks to be related to changeset e988661e458c5402c0236cd1084a8671249a760d
Issue bpo-20538: UTF-7 incremental decoder produced inconsistant string when
input was truncated in BASE64 section.

@larryhastings
Copy link
Contributor Author

Serhiy said on IRC that he doesn't have a Windows development environment, so he didn't think he could help.

@serhiy-storchaka
Copy link
Member

UTF-7 decoder is not related to this test.

The test_readline test was broken from the born, and a part of this test was do nothing. After fixing it in bpo-20520, new bugs were exposed: bpo-20538 and this. This bug was hidden until fixing bpo-20538.

Note that there is no test_partial in CP65001Test. Perhaps it is related.

The simplest solution would be to temporary skip test_readline in CP65001Test:

    test_readline = unittest.expectedFailure(ReadTest.test_readline)

@vstinner
Copy link
Member

vstinner commented Feb 9, 2014

The test tries to decode a partial UTF-8 bytes string. The problem is that codecs.code_page_decode() doesn't implement fully partial decoders. The decoder only supports partial decoding for a few code pages: 932, 936, 949, 950, and 1361. The partial decoding is currently based on IsDBCSLeadByteEx():
http://msdn.microsoft.com/en-us/library/windows/desktop/dd318667%28v=vs.85%29.aspx

It may be possible to enhance decoders, but it's not a regression from Python 3.3 and so can be done in Python 3.5.

Please just skip failing tests for CP_UTF8 (cp 65001) and maybe other Windows code pages in test_codecs.

(I don't have time to write a patch to skip, sorry.)

@python-dev
Copy link
Mannequin

python-dev mannequin commented Feb 9, 2014

New changeset 4f6499fc2f09 by Victor Stinner in branch 'default':
Issue bpo-20571: skip test_readline() of test_codecs for Windows code page 65001.
http://hg.python.org/cpython/rev/4f6499fc2f09

@vstinner
Copy link
Member

vstinner commented Feb 9, 2014

I opened bpo-20574 to implement the missing feature for cp65001.

@vstinner vstinner closed this as completed Feb 9, 2014
@python-dev
Copy link
Mannequin

python-dev mannequin commented Feb 23, 2014

New changeset d8f48717b74e by Victor Stinner in branch '3.3':
Issue bpo-20571: skip test_readline() of test_codecs for Windows code page 65001.
http://hg.python.org/cpython/rev/d8f48717b74e

@birkenfeld
Copy link
Member

Would have been nice to do this also on 3.3 branch...

@vstinner
Copy link
Member

Ah yes, sorry. I forgot that the utf-7 change was also applied to 3.3.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OS-windows release-blocker tests Tests in the Lib/test dir topic-unicode type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

4 participants