This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author terry.reedy
Recipients Anthony Sottile, ammar2, benjamin.peterson, gregory.p.smith, meador.inge, miss-islington, ned.deily, taleinat, terry.reedy
Date 2018-10-24.14:53:02
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
#33766 was about documenting the C tokenizer change, some years ago, that made end-of-file EOF and end-of-string EOS generate the NEWLINE token required to properly terminate statements.  "The end of input also serves
as an implicit terminator for the final physical line."

Although the tokenizer module intentionally does not exactly mirror the C tokenizer (it adds COMMENT tokens), it plausibly seems like a bug that it was not changed along with the C tokenizer, as it has since been tokenizing valid code as grammatically invalid.  But I agree that this fix is too disruptive for 2.7.
Date User Action Args
2018-10-24 14:53:02terry.reedysetrecipients: + terry.reedy, gregory.p.smith, taleinat, benjamin.peterson, ned.deily, meador.inge, Anthony Sottile, ammar2, miss-islington
2018-10-24 14:53:02terry.reedysetmessageid: <>
2018-10-24 14:53:02terry.reedylinkissue33899 messages
2018-10-24 14:53:02terry.reedycreate