Message172851
When tokenize.untokenize() encounters a 2-tuple, it moves to compatibility mode, where only the token type and string are used from that point forward. There are two closely related problems:
* when the iterable is a sequence, the portion of the sequence prior to the 2-tuple is traversed a second time under compatibility mode.
* when the iterable is an iterator, the first 2-tuple encountered is essentially gobbled up (see issue16221).
Either an explicit "iterable = iter(iterable)" or "iterable = list(iterable)" should happen at the very beginning of Untokenizer.untokenize(). If the former, Untokenizer.compat() should be fixed to not treat that first token differently. If the latter, self.tokens should be cleared at the beginning of Untokenizer.compat().
I'll put up a patch with the second option when I get a chance. |
|
Date |
User |
Action |
Args |
2012-10-14 06:04:53 | eric.snow | set | recipients:
+ eric.snow |
2012-10-14 06:04:53 | eric.snow | set | messageid: <1350194693.62.0.576275883555.issue16224@psf.upfronthosting.co.za> |
2012-10-14 06:04:53 | eric.snow | link | issue16224 messages |
2012-10-14 06:04:52 | eric.snow | create | |
|