Author terry.reedy
Recipients Anthony Sottile, ammar2, benjamin.peterson, gregory.p.smith, meador.inge, miss-islington, ned.deily, taleinat, terry.reedy
Date 2018-10-24.14:53:02
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1540392782.68.0.788709270274.issue33899@psf.upfronthosting.co.za>
In-reply-to
Content
#33766 was about documenting the C tokenizer change, some years ago, that made end-of-file EOF and end-of-string EOS generate the NEWLINE token required to properly terminate statements.  "The end of input also serves
as an implicit terminator for the final physical line."

Although the tokenizer module intentionally does not exactly mirror the C tokenizer (it adds COMMENT tokens), it plausibly seems like a bug that it was not changed along with the C tokenizer, as it has since been tokenizing valid code as grammatically invalid.  But I agree that this fix is too disruptive for 2.7.
History
Date User Action Args
2018-10-24 14:53:02terry.reedysetrecipients: + terry.reedy, gregory.p.smith, taleinat, benjamin.peterson, ned.deily, meador.inge, Anthony Sottile, ammar2, miss-islington
2018-10-24 14:53:02terry.reedysetmessageid: <1540392782.68.0.788709270274.issue33899@psf.upfronthosting.co.za>
2018-10-24 14:53:02terry.reedylinkissue33899 messages
2018-10-24 14:53:02terry.reedycreate