This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Drekin
Recipients Drekin, benjamin.peterson, brett.cannon, eric.araujo, georg.brandl, gvanrossum, ncoghlan, paul.moore, pitrou, steve.dower, tshepang
Date 2016-08-14.16:58:20
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1471193900.65.0.876512632022.issue17620@psf.upfronthosting.co.za>
In-reply-to
Content
> Unfortunately, it looks like detecting when a readline hook has been added is going to involve significant changes to the tokenizer, which I really don't want to do.

We don't need to detect the presence of readline hook, it may be so that there is always a readline hook. Whenever we have interactive stdio, and so PyOS_Readline is called, the new proposed API PyIO_Readline would be called instead. This would return Unicode str Py_Object*, so the result can be directly returned by input() and should be somehow encoded afterwards by the tokenizer (these are the only consumers of PyOS_Readline). 

We may even leave the tokenizer alone and redefine PyOS_Readline as a wrapper of PyIO_Readline, having full control of the encoding process there. So it would be enough to set up the tokenizer with UTF-8 encoding despite the fact that sys.std*.encoding would be UTF-16.

(I hope that if the tokenizer was desiged nowdays, it would operate on strings rather than bytes so there won't be any encoding problems at all.)

Also, third parties would benefit from sys.readlinehook – at least win_unicode_console and pyreadline would just set the attribute rather than messing with ctypes.
History
Date User Action Args
2016-08-14 16:58:20Drekinsetrecipients: + Drekin, gvanrossum, brett.cannon, georg.brandl, paul.moore, ncoghlan, pitrou, benjamin.peterson, eric.araujo, tshepang, steve.dower
2016-08-14 16:58:20Drekinsetmessageid: <1471193900.65.0.876512632022.issue17620@psf.upfronthosting.co.za>
2016-08-14 16:58:20Drekinlinkissue17620 messages
2016-08-14 16:58:20Drekincreate