Author gumblex
Recipients Arfrever, gumblex, jaraco, terry.reedy
Date 2015-06-22.02:33:12
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1434940393.68.0.733290402414.issue20387@psf.upfronthosting.co.za>
In-reply-to
Content
The new patch should now pass all tests correctly.

The main idea is:
* if the token is INDENT, push it on the `indents` stack and continue
* if a new line starts, AND the position of the first token >= the length of the last indent level, we assume the indent is contained in the leading whitespaces.

The new test_tokenize.py fails:

https://bitbucket.org/jaraco/cpython-issue20387/src/b7fe3c865b8dbdb33d26f4bc5cbb6096f5445fb2/Lib/test/test_tokenize.py?at=3.4#cl-1244

Line 1244 should be:
codelines = self.roundtrip(code).split(b'\n')

It seems that the tokens generated by tokenize.tokenize don't contain enough information to restore the original file.

* Tabs between tokens are not preserved.
* Space before backslash as line continuation are not preserved.

(From test/tokenize_tests.txt)
 # Backslash means line continuation:
-x = 1 \
+x = 1\
 + 1

My roundtrip test code copied here from #24447:

python2 -c 'import sys, tokenize; sys.stdout.write(tokenize.untokenize(tokenize.generate_tokens(sys.stdin.readline)))'
python3 -c 'import sys, tokenize; sys.stdout.buffer.write(tokenize.untokenize(tokenize.tokenize(sys.stdin.buffer.readline)))'
History
Date User Action Args
2015-06-22 02:33:13gumblexsetrecipients: + gumblex, terry.reedy, jaraco, Arfrever
2015-06-22 02:33:13gumblexsetmessageid: <1434940393.68.0.733290402414.issue20387@psf.upfronthosting.co.za>
2015-06-22 02:33:13gumblexlinkissue20387 messages
2015-06-22 02:33:13gumblexcreate