classification
Title: Readline test in test_codecs is broken
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: 20538 Superseder:
Assigned To: serhiy.storchaka Nosy List: benjamin.peterson, doerwalter, georg.brandl, larry, ned.deily, python-dev, serhiy.storchaka, vajrasky
Priority: release blocker Keywords: patch

Created on 2014-02-05 13:09 by serhiy.storchaka, last changed 2014-02-08 19:11 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
test_codecs_readline.patch serhiy.storchaka, 2014-02-05 13:09 review
fix_linetests.diff doerwalter, 2014-02-05 14:07 review
fix_linetests2.diff doerwalter, 2014-02-05 16:30 review
Messages (10)
msg210309 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-05 13:09
The ReadTest.test_readline test in Lib/test/test_codecs.py has two loops on "\n \r\n \r \u2028".split(). But as far as "\n \r\n \r \u2028".split() is empty list (because newline characters are whitespace characters), the bodies of these loops are never executed.

After fixing this bug, a number of other tests were exposed. I tried to fix them. The purpose of these tests is not entirely clear, so I'm not sure that it is properly grasped the idea of the author.

Test was added in issue1076985 and modified in issue1175396.
msg210315 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2014-02-05 14:07
"\n \r\n \r \u2028".split() should have been "\n \r\n \r \u2028".split(" "), i.e. a list of different line ends.

> The purpose of these tests is not entirely clear, so I'm not sure that it is properly grasped the idea of the author.

I wrote the tests nearly 10 years ago, so it's no longer entirely clear to me either! ;)

Anyway, here's a patch that fixes the resulting test failures.
msg210329 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-05 15:54
> "\n \r\n \r \u2028".split() should have been "\n \r\n \r \u2028".split(" "),
> i.e. a list of different line ends.

Yes, this is obvious fix, but explicit tuple of line ends looks more clear and 
error-proof to me.
msg210332 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2014-02-05 16:30
True, here's an updated patch.
msg210362 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-02-06 07:28
New changeset 305b6a5da852 by Serhiy Storchaka in branch '2.7':
Issue #20520: Fixed readline test in test_codecs.
http://hg.python.org/cpython/rev/305b6a5da852

New changeset 82d374a9bbc7 by Serhiy Storchaka in branch '3.3':
Issue #20520: Fixed readline test in test_codecs.
http://hg.python.org/cpython/rev/82d374a9bbc7

New changeset 1457fa0da200 by Serhiy Storchaka in branch 'default':
Issue #20520: Fixed readline test in test_codecs.
http://hg.python.org/cpython/rev/1457fa0da200
msg210432 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-02-07 02:16
The change has broken the builds, for example, http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/1197.

[ 29/389] test_codecs
Assertion failed: (maxchar >= 0x100), function _PyUnicode_CheckConsistency, file Objects/unicodeobject.c, line 410.
Fatal Python error: Aborted

Current thread 0x00007fff71296cc0 (most recent call first):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/codecs.py", line 494 in read
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/codecs.py", line 548 in readline
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_codecs.py", line 169 in test_readline
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/case.py", line 574 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/case.py", line 622 in __call__
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 125 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 87 in __call__
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 125 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 87 in __call__
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 125 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 87 in __call__
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/runner.py", line 168 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/support/__init__.py", line 1685 in _run_suite
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/support/__init__.py", line 1719 in run_unittest
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/regrtest.py", line 1277 in <lambda>
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/regrtest.py", line 1278 in runtest_inner
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/regrtest.py", line 967 in runtest
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/regrtest.py", line 532 in main
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/regrtest.py", line 1565 in main_in_temp_cwd
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/regrtest.py", line 1590 in <module>
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/runpy.py", line 86 in _run_code
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/runpy.py", line 171 in _run_module_as_main
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/runpy.py", line 171, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/__main__.py", line 3, in <module>
    regrtest.main_in_temp_cwd()
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/regrtest.py", line 1565, in main_in_temp_cwd
    main()
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/regrtest.py", line 738, in main
    raise Exception("Child error on {}: {}".format(test, result[1]))
Exception: Child error on test_codecs: Exit code -6
make: *** [buildbottest] Error 1
msg210440 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-07 08:12
Aha! It looks as a bug in UTF-7 codec.
msg210443 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2014-02-07 09:14
The culprit:

diff -r b4e99bec0c8a Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py	Fri Feb 07 10:10:55 2014 +0200
+++ b/Lib/test/test_codecs.py	Fri Feb 07 17:14:12 2014 +0800
@@ -164,6 +164,8 @@
                 s = 10*(size*"a" + lineend + "xxx\n")
                 reader = getreader(s)
                 for i in range(10):
+                    if size==64 and self.encoding == 'utf-7' and lineend=='\u2028' and i == 3:
+                        import pdb; pdb.set_trace()
                     self.assertEqual(
                         reader.readline(keepends=True),
                         size*"a" + lineend,
msg210445 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-07 09:34
Opened separate issue20538 for this bug.
msg210446 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-07 09:35
Thank you Vajrasky. I got the same result.
History
Date User Action Args
2014-02-08 19:11:05serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: needs patch -> resolved
2014-02-07 09:35:29serhiy.storchakasetmessages: + msg210446
2014-02-07 09:34:17serhiy.storchakasetdependencies: + Segfault in UTF-7 incremental decoder
messages: + msg210445
2014-02-07 09:14:59vajraskysetnosy: + vajrasky
messages: + msg210443
2014-02-07 08:12:39serhiy.storchakasetmessages: + msg210440
2014-02-07 02:16:03ned.deilysetstatus: closed -> open
priority: normal -> release blocker

assignee: serhiy.storchaka

nosy: + ned.deily, benjamin.peterson, georg.brandl, larry
messages: + msg210432
resolution: fixed -> (no value)
stage: resolved -> needs patch
2014-02-06 07:54:02serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2014-02-06 07:28:01python-devsetnosy: + python-dev
messages: + msg210362
2014-02-05 16:30:07doerwaltersetfiles: + fix_linetests2.diff

messages: + msg210332
2014-02-05 15:54:46serhiy.storchakasetmessages: + msg210329
2014-02-05 14:07:39doerwaltersetfiles: + fix_linetests.diff

messages: + msg210315
2014-02-05 13:09:34serhiy.storchakacreate