This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: tokenize.untokenize() "compat" mode misses the encoding when using an iterator
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
Status: closed Resolution: out of date
Dependencies: Superseder: tokenize.untokenize() misbehaves when moved to "compatiblity mode"
View: 16224
Assigned To: Nosy List: eric.snow, iritkatriel
Priority: normal Keywords: patch

Created on 2012-10-14 05:26 by eric.snow, last changed 2022-04-11 14:57 by admin. This issue is now closed.

File name Uploaded Description Edit
untokenize_compat.diff eric.snow, 2012-10-14 05:29 review
Messages (6)
msg172848 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-10-14 05:26
While traversing the passed iterable, untokenize() will go to "compatibility mode" one it hits a 2-tuple instead of a 5-tuple (coming from the iterable).  That token will not go through the normal steps that other tokens do.  Most critically, if that token is the ENCODING token, the source is not encoded to that encoding at the end (even though it should have been).  I'll have a patch up in a moment (for the tip of default).

There are a couple of other issues at hand that I will bring up separately.
msg172852 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-10-14 06:05
issue16224 _may_ supercede this ticket.
msg172854 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-10-14 06:20
The patch that I have in #16224 takes care of this issue.  If that issue goes in another direction however...
msg222818 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-12 01:09
#16224 has been closed as a duplicate of #8478.
msg380519 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-11-07 19:41
I think this is fixed now - the tests in the patch are passing.
msg391264 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-04-16 23:18
This was fixed here under issue719888.
Date User Action Args
2022-04-11 14:57:37adminsetgithub: 60425
2021-04-16 23:18:16iritkatrielsetstatus: open -> closed
resolution: out of date
messages: + msg391264

stage: patch review -> resolved
2020-11-07 19:41:40iritkatrielsetnosy: + iritkatriel
messages: + msg380519
2016-09-09 18:56:13BreamoreBoysetnosy: - BreamoreBoy
2016-09-09 18:25:27eric.snowsetassignee: eric.snow ->
2014-07-12 01:09:15BreamoreBoysetstatus: pending -> open
nosy: + BreamoreBoy
messages: + msg222818

2012-10-14 06:20:58eric.snowsetstatus: open -> pending
superseder: tokenize.untokenize() misbehaves when moved to "compatiblity mode"
messages: + msg172854
2012-10-14 06:05:59eric.snowsetmessages: + msg172852
components: + Library (Lib)
2012-10-14 05:29:31eric.snowsetfiles: + untokenize_compat.diff
keywords: + patch
stage: test needed -> patch review
2012-10-14 05:26:56eric.snowcreate