Message 63618 - Python tracker

➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author	loewis
Recipients	barry, loewis, mark.dickinson
Date	2008-03-17.04:39:05
SpamBayes Score	0.07301519
Marked as misclassified	No
Message-id	<1205728746.74.0.894937501179.issue719888@psf.upfronthosting.co.za>
In-reply-to

Content
In 3k, the tokenize module should definitely return strings, and, in doing so, it should definitely consider the encoding declaration (and also the default encoding in absence of the encoding declaration). For 2.6, I wouldn't mind if it were changed incompatibly so that it returns Unicode strings, or else that it parses in Unicode, and then encodes back to the source encoding before returning anything.

In 3k, the tokenize module should definitely return strings, and, in 
doing so, it should definitely consider the encoding declaration (and 
also the default encoding in absence of the encoding declaration).

For 2.6, I wouldn't mind if it were changed incompatibly so that it 
returns Unicode strings, or else that it parses in Unicode, and then 
encodes back to the source encoding before returning anything.

History
Date	User	Action	Args
2008-03-17 04:39:07	loewis	set	spambayes_score: 0.0730152 -> 0.07301519 recipients: + loewis, barry, mark.dickinson
2008-03-17 04:39:06	loewis	set	spambayes_score: 0.0730152 -> 0.0730152 messageid: <1205728746.74.0.894937501179.issue719888@psf.upfronthosting.co.za>
2008-03-17 04:39:05	loewis	link	issue719888 messages
2008-03-17 04:39:05	loewis	create