This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author terry.reedy
Recipients Brian.Bossé, DLitz, akuchling, eric.snow, kristjan.jonsson, meador.inge, nick.caruso, python-dev, rhettinger, sfllaw, terry.reedy
Date 2014-02-24.04:55:48
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1393217749.25.0.542093424089.issue9974@psf.upfronthosting.co.za>
In-reply-to
Content
I added 5-tuple mode to roundtrip() in #20750. I solved the ENDMARKER problem by breaking out of the token loop if and when it appears. Reconstructing trailing whitespace other than \n is hopeless. The roundtrip test currently only tests equality of token sequences. But my own tests show that code with backslash-newline is reconstructed correctly as long as there is no space before it and something other than ENDMARKER after it.

I discovered that tokenize will tokenize '\\' but not '\\\n'. So the latter will never appear as tokenizer output. Even if we did use ENDMARKER to create the latter, it would fail the current roundtrip test.
History
Date User Action Args
2014-02-24 04:55:49terry.reedysetrecipients: + terry.reedy, akuchling, rhettinger, kristjan.jonsson, sfllaw, meador.inge, nick.caruso, Brian.Bossé, python-dev, eric.snow, DLitz
2014-02-24 04:55:49terry.reedysetmessageid: <1393217749.25.0.542093424089.issue9974@psf.upfronthosting.co.za>
2014-02-24 04:55:49terry.reedylinkissue9974 messages
2014-02-24 04:55:48terry.reedycreate