Logged In: YES 
user_id=89016

It isn't the buffering support per se that breaks the
tokenizer. This problem exists even in Python 2.3.x (Simply
try the test scripts from http://www.python.org/sf/1089395
with Python 2.3.5 and you'll get a segfault). Applications
that rely on len(readline(x)) == x or anything similar are
broken anyway. Supporting buffered and unbuffered reading
would mean keeping the 2.3 mode of doing things around
indefinitely, and we'd loose readline() support for UTF-16
again.

BTW, applying Greg Chapman's patch
(http://www.python.org/sf/1101726, which fixes the
tokenizer) together with this one seems to fix the problem
from my previous post. So if you could give
http://www.python.org/sf/1101726 a third look, so we can get
it into 2.4.2, this would be great.