This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author eric.snow
Recipients eric.snow
Date 2012-10-14.06:04:52
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1350194693.62.0.576275883555.issue16224@psf.upfronthosting.co.za>
In-reply-to
Content
When tokenize.untokenize() encounters a 2-tuple, it moves to compatibility mode, where only the token type and string are used from that point forward.  There are two closely related problems:

* when the iterable is a sequence, the portion of the sequence prior to the 2-tuple is traversed a second time under compatibility mode.
* when the iterable is an iterator, the first 2-tuple encountered is essentially gobbled up (see issue16221).

Either an explicit "iterable = iter(iterable)" or "iterable = list(iterable)" should happen at the very beginning of Untokenizer.untokenize().  If the former, Untokenizer.compat() should be fixed to not treat that first token differently.  If the latter, self.tokens should be cleared at the beginning of Untokenizer.compat().

I'll put up a patch with the second option when I get a chance.
History
Date User Action Args
2012-10-14 06:04:53eric.snowsetrecipients: + eric.snow
2012-10-14 06:04:53eric.snowsetmessageid: <1350194693.62.0.576275883555.issue16224@psf.upfronthosting.co.za>
2012-10-14 06:04:53eric.snowlinkissue16224 messages
2012-10-14 06:04:52eric.snowcreate