This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author gdr@garethrees.org
Recipients benjamin.peterson, daniel.urban, eric.snow, ezio.melotti, gdr@garethrees.org, meador.inge, sandro.tosi, serhiy.storchaka, takluyver, yselivanov
Date 2014-02-05.10:59:03
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1391597944.21.0.140849623525.issue12691@psf.upfronthosting.co.za>
In-reply-to
Content
This morning I noticed that I had forgotten to update the library
reference, and I also noticed two more problems to add to the list
above:

6. Although Lib/test/test_tokenize.py looks like it contains tests for
   backslash-newline handling, these tests are ineffective. Here they
   are:

      >>> roundtrip("x=1+\\\\n"
      ...           "1\\n"
      ...           "# This is a comment\\\\n"
      ...           "# This also\\n")
      True
      >>> roundtrip("# Comment \\\\nx = 0")
      True

  There are two problems here: (i) because of the double string
  escaping, these are not backslash-newline, they are backslash-n.
  (ii) the roundtrip() test is too weak to detect this problem:
  tokenize() outputs an ERRORTOKEN for the backslash and untokenize()
  restores it. So the round-trip property is satisfied.

7. Problem 6 shows the difficulty of using doctests for this kind of
   test. It would be easier to ensure the correctness of these tests
   if the docstring was read from a separate file, so that at least
   the tests only need one level of string escaping.

I fixed problem 6 by updating these tests to use dump_tokens() instead
of roundtrip(). I have not fixed problem 7 (like 4 and 5, I can leave
it for another issue). Revised patch attached.
History
Date User Action Args
2014-02-05 10:59:04gdr@garethrees.orgsetrecipients: + gdr@garethrees.org, benjamin.peterson, ezio.melotti, meador.inge, daniel.urban, sandro.tosi, eric.snow, takluyver, serhiy.storchaka, yselivanov
2014-02-05 10:59:04gdr@garethrees.orgsetmessageid: <1391597944.21.0.140849623525.issue12691@psf.upfronthosting.co.za>
2014-02-05 10:59:04gdr@garethrees.orglinkissue12691 messages
2014-02-05 10:59:04gdr@garethrees.orgcreate