Author jaraco
Recipients jaraco
Date 2014-01-24.22:51:14
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1390603875.19.0.598751073352.issue20387@psf.upfronthosting.co.za>
In-reply-to
Content
Consider this simple unit test:

def test_tokenize():
    input = "if False:\n\tx=3\n\ty=3\n"
    g = list(generate_tokens(io.StringIO(input).readline))
    assert untokenize(g) == input

According to the docs, untokenize guarantees the output equals the input to tokenize: http://docs.python.org/2/library/tokenize.html#tokenize.untokenize

As this test fails on Python 2 and Python 3 (use io.BytesIO on Python 2), I believe the test illustrates a violation of this guarantee. The second line in the if block gets its tab replaced by a space.

As the input is valid Python Syntax, this behavior is particularly surprising.
History
Date User Action Args
2014-01-24 22:51:15jaracosetrecipients: + jaraco
2014-01-24 22:51:15jaracosetmessageid: <1390603875.19.0.598751073352.issue20387@psf.upfronthosting.co.za>
2014-01-24 22:51:15jaracolinkissue20387 messages
2014-01-24 22:51:14jaracocreate