Message210299
This morning I noticed that I had forgotten to update the library
reference, and I also noticed two more problems to add to the list
above:
6. Although Lib/test/test_tokenize.py looks like it contains tests for
backslash-newline handling, these tests are ineffective. Here they
are:
>>> roundtrip("x=1+\\\\n"
... "1\\n"
... "# This is a comment\\\\n"
... "# This also\\n")
True
>>> roundtrip("# Comment \\\\nx = 0")
True
There are two problems here: (i) because of the double string
escaping, these are not backslash-newline, they are backslash-n.
(ii) the roundtrip() test is too weak to detect this problem:
tokenize() outputs an ERRORTOKEN for the backslash and untokenize()
restores it. So the round-trip property is satisfied.
7. Problem 6 shows the difficulty of using doctests for this kind of
test. It would be easier to ensure the correctness of these tests
if the docstring was read from a separate file, so that at least
the tests only need one level of string escaping.
I fixed problem 6 by updating these tests to use dump_tokens() instead
of roundtrip(). I have not fixed problem 7 (like 4 and 5, I can leave
it for another issue). Revised patch attached. |
|
Date |
User |
Action |
Args |
2014-02-05 10:59:04 | gdr@garethrees.org | set | recipients:
+ gdr@garethrees.org, benjamin.peterson, ezio.melotti, meador.inge, daniel.urban, sandro.tosi, eric.snow, takluyver, serhiy.storchaka, yselivanov |
2014-02-05 10:59:04 | gdr@garethrees.org | set | messageid: <1391597944.21.0.140849623525.issue12691@psf.upfronthosting.co.za> |
2014-02-05 10:59:04 | gdr@garethrees.org | link | issue12691 messages |
2014-02-05 10:59:04 | gdr@garethrees.org | create | |
|