This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author michael.foord
Recipients meador.inge, michael.foord
Date 2010-09-28.14:04:57
SpamBayes Score 4.083938e-06
Marked as misclassified No
Message-id <1285682699.31.0.82798494256.issue9969@psf.upfronthosting.co.za>
In-reply-to
Content
Note from Nick Coghlan from the Python-dev discussion:

A very quick scan of _tokenize suggests it is designed to support
detect_encoding returning None to indicate the line iterator will
return already decoded lines. This is confirmed by the fact the
standard library uses it that way (via generate_tokens).

An API that accepts a string, wraps a StringIO around it, then calls
_tokenise with an encoding of None would appear to be the answer here.
A feature request on the tracker is the best way to make that happen.
History
Date User Action Args
2010-09-28 14:04:59michael.foordsetrecipients: + michael.foord, meador.inge
2010-09-28 14:04:59michael.foordsetmessageid: <1285682699.31.0.82798494256.issue9969@psf.upfronthosting.co.za>
2010-09-28 14:04:57michael.foordlinkissue9969 messages
2010-09-28 14:04:57michael.foordcreate