Author vstinner
Recipients Devin Jeanpierre, eric.araujo, petri.lehtinen, terry.reedy, vstinner
Date 2011-07-09.09:03:45
SpamBayes Score 3.07523e-05
Marked as misclassified No
Message-id <1310202226.54.0.882751009246.issue12486@psf.upfronthosting.co.za>
In-reply-to
Content
The compiler has a PyCF_SOURCE_IS_UTF8 flag: see compile() builtin. The parser has a flag to ignore the coding cookie: PyPARSE_IGNORE_COOKIE.

Patch tokenize to support Unicode is simple: use PyCF_SOURCE_IS_UTF8 and/or PyPARSE_IGNORE_COOKIE flags and encode the strings to UTF-8.

Rewrite the parser to work directly on Unicode is much more complex and I don't think that we need that.
History
Date User Action Args
2011-07-09 09:03:46vstinnersetrecipients: + vstinner, terry.reedy, Devin Jeanpierre, eric.araujo, petri.lehtinen
2011-07-09 09:03:46vstinnersetmessageid: <1310202226.54.0.882751009246.issue12486@psf.upfronthosting.co.za>
2011-07-09 09:03:45vstinnerlinkissue12486 messages
2011-07-09 09:03:45vstinnercreate