This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author gumblex
Recipients Arfrever, gumblex, jaraco, terry.reedy
Date 2015-06-22.02:33:12
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
The new patch should now pass all tests correctly.

The main idea is:
* if the token is INDENT, push it on the `indents` stack and continue
* if a new line starts, AND the position of the first token >= the length of the last indent level, we assume the indent is contained in the leading whitespaces.

The new fails:

Line 1244 should be:
codelines = self.roundtrip(code).split(b'\n')

It seems that the tokens generated by tokenize.tokenize don't contain enough information to restore the original file.

* Tabs between tokens are not preserved.
* Space before backslash as line continuation are not preserved.

(From test/tokenize_tests.txt)
 # Backslash means line continuation:
-x = 1 \
+x = 1\
 + 1

My roundtrip test code copied here from #24447:

python2 -c 'import sys, tokenize; sys.stdout.write(tokenize.untokenize(tokenize.generate_tokens(sys.stdin.readline)))'
python3 -c 'import sys, tokenize; sys.stdout.buffer.write(tokenize.untokenize(tokenize.tokenize(sys.stdin.buffer.readline)))'
Date User Action Args
2015-06-22 02:33:13gumblexsetrecipients: + gumblex, terry.reedy, jaraco, Arfrever
2015-06-22 02:33:13gumblexsetmessageid: <>
2015-06-22 02:33:13gumblexlinkissue20387 messages
2015-06-22 02:33:13gumblexcreate