Message245613
The new patch should now pass all tests correctly.
The main idea is:
* if the token is INDENT, push it on the `indents` stack and continue
* if a new line starts, AND the position of the first token >= the length of the last indent level, we assume the indent is contained in the leading whitespaces.
The new test_tokenize.py fails:
https://bitbucket.org/jaraco/cpython-issue20387/src/b7fe3c865b8dbdb33d26f4bc5cbb6096f5445fb2/Lib/test/test_tokenize.py?at=3.4#cl-1244
Line 1244 should be:
codelines = self.roundtrip(code).split(b'\n')
It seems that the tokens generated by tokenize.tokenize don't contain enough information to restore the original file.
* Tabs between tokens are not preserved.
* Space before backslash as line continuation are not preserved.
(From test/tokenize_tests.txt)
# Backslash means line continuation:
-x = 1 \
+x = 1\
+ 1
My roundtrip test code copied here from #24447:
python2 -c 'import sys, tokenize; sys.stdout.write(tokenize.untokenize(tokenize.generate_tokens(sys.stdin.readline)))'
python3 -c 'import sys, tokenize; sys.stdout.buffer.write(tokenize.untokenize(tokenize.tokenize(sys.stdin.buffer.readline)))' |
|
Date |
User |
Action |
Args |
2015-06-22 02:33:13 | gumblex | set | recipients:
+ gumblex, terry.reedy, jaraco, Arfrever |
2015-06-22 02:33:13 | gumblex | set | messageid: <1434940393.68.0.733290402414.issue20387@psf.upfronthosting.co.za> |
2015-06-22 02:33:13 | gumblex | link | issue20387 messages |
2015-06-22 02:33:13 | gumblex | create | |
|