Author r.david.murray
Recipients Joshua.Landau, benjamin.peterson, loewis, python-dev, r.david.murray, steve.dower, wim.glenn
Date 2015-04-15.16:02:21
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
My impression is that it was fixed the way it was because it makes the internal tokenizer match the what the tokenize module does.  See also issue 3353.  As for changing it in a point release, it turns something that was an error into something that isn't, so it was unlikely to break existing working code.  Going the other way in the tokenize module *would* have been a backward compatibility issue.  If we wanted to change this, it would require a deprecation process, and it hardly seems worth it.  I hear you about other tokenizers, though, and that is indeed unfortunate.
Date User Action Args
2015-04-15 16:02:21r.david.murraysetrecipients: + r.david.murray, loewis, benjamin.peterson, python-dev, Joshua.Landau, steve.dower, wim.glenn
2015-04-15 16:02:21r.david.murraysetmessageid: <>
2015-04-15 16:02:21r.david.murraylinkissue21642 messages
2015-04-15 16:02:21r.david.murraycreate