This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Blake Hilliard
Recipients Blake Hilliard
Date 2015-11-18.21:13:27
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1447881207.88.0.934848299376.issue25661@psf.upfronthosting.co.za>
In-reply-to
Content
Here's a simple test case that shows the bug:

# python
Python 2.7.9 (default, Apr  2 2015, 14:49:18) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import StringIO, tokenize
>>> code = "if False:\n\tx=3\n \ty=3\n"
>>> g = list(tokenize.generate_tokens(StringIO.StringIO(code).readline))
>>> assert tokenize.untokenize(g) == code
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError


This is very similar to: https://bugs.python.org/issue20387

I'm using the patch that was provided to fix that bug, but with the patch this slightly different test case still fails. On the last line of "code" from my example, where the line begins with a single space then a tab, the untokenized string reverses the order, and the tab precedes the space. This generates code that will throw an IndentationError if run.
History
Date User Action Args
2015-11-18 21:13:27Blake Hilliardsetrecipients: + Blake Hilliard
2015-11-18 21:13:27Blake Hilliardsetmessageid: <1447881207.88.0.934848299376.issue25661@psf.upfronthosting.co.za>
2015-11-18 21:13:27Blake Hilliardlinkissue25661 messages
2015-11-18 21:13:27Blake Hilliardcreate