Author terry.reedy
Recipients Arfrever, jaraco, terry.reedy
Date 2014-02-02.08:30:17
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1391329817.53.0.82097480776.issue20387@psf.upfronthosting.co.za>
In-reply-to
Content
I read the manual more carefully and noticed that the guarantee is that tokenizing the result of untokenize matches the input to untokenize. " The result is guaranteed to tokenize back to match the input so that the conversion is lossless and round-trips are assured."

I ran the test in 3.4 and noticed the exactness and direction of the matching does not matter because only the 2nd \t is replaced by a space, making the invalid code that raises IndentationError.

So this is definitely a bug and I withdraw the close suggestion.

I believe the test should be that both lines of the body begin with the same indent.

from tokenize import tokenize, untokenize
import io

    def test_tab_indent(self):
        code = b"if False:\n\tx=3\n\ty=3\n"
        codelines = untokenize(tokenize(io.BytesIO(s).readline)).split(\n)
        assertEqual(codelines[1], codelines[2])

Of course, the test of tokenize and untokenize should be separated.
History
Date User Action Args
2014-02-02 08:30:17terry.reedysetrecipients: + terry.reedy, jaraco, Arfrever
2014-02-02 08:30:17terry.reedysetmessageid: <1391329817.53.0.82097480776.issue20387@psf.upfronthosting.co.za>
2014-02-02 08:30:17terry.reedylinkissue20387 messages
2014-02-02 08:30:17terry.reedycreate