Tokenize module does not mirror "end-of-input" is newline behavior #78080

ammaraskar · 2018-06-19T07:41:52Z

BPO	33899
Nosy	@terryjreedy, @gpshead, @taleinat, @benjaminp, @ned-deily, @meadori, @asottile, @ammaraskar, @miss-islington, @asmeurer
PRs	bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior #7891 [3.7] bpo-33899: Make tokenize module mirror end-of-file is end-of-li… #8132 [2.7] bpo-33899: Make tokenize module mirror end-of-file is end-of-li… #8133 [3.6] bpo-33899: Make tokenize module mirror end-of-file is end-of-li… #8134 [2.7] bpo-33899: Revert tokenize module adding an implicit final NEWLINE #10072 bpo-33899: add mention of this change in What's New #10073 [3.7] bpo-33899: Mention tokenize behavior change in What's New (GH-10073) #10074 [3.6] bpo-33899: Mention tokenize behavior change in What's New (GH-10073) #10075

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ammaraskar'
closed_at = <Date 2018-07-06.10:24:50.699>
created_at = <Date 2018-06-19.07:41:52.217>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'Tokenize module does not mirror "end-of-input" is newline behavior'
updated_at = <Date 2019-03-22.11:46:47.154>
user = 'https://github.com/ammaraskar'

bugs.python.org fields:

activity = <Date 2019-03-22.11:46:47.154>
actor = 'brechtm'
assignee = 'ammar2'
closed = True
closed_date = <Date 2018-07-06.10:24:50.699>
closer = 'taleinat'
components = ['Library (Lib)']
creation = <Date 2018-06-19.07:41:52.217>
creator = 'ammar2'
dependencies = []
files = []
hgrepos = []
issue_num = 33899
keywords = ['patch']
message_count = 27.0
messages = ['319934', '321154', '321162', '321163', '321164', '321165', '328220', '328222', '328226', '328227', '328238', '328283', '328318', '328324', '328353', '328354', '328355', '328356', '328357', '328358', '328359', '328360', '328369', '328383', '328877', '330213', '338601']
nosy_count = 11.0
nosy_names = ['terry.reedy', 'gregory.p.smith', 'taleinat', 'benjamin.peterson', 'ned.deily', 'meador.inge', 'brechtm', 'Anthony Sottile', 'ammar2', 'miss-islington', 'asmeurer']
pr_nums = ['7891', '8132', '8133', '8134', '10072', '10073', '10074', '10075']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33899'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

ammaraskar · 2018-06-19T07:41:52Z

As was pointed out in https://bugs.python.org/issue33766 there is an edge case in the tokenizer whereby it will implicitly treat the end of input as a newline. The tokenize module in stdlib does not mirror the C code's behavior in this case.

tokenizer.c:

~/cpython $ echo -n 'x' | ./python
----------
NAME ("x")
NEWLINE
ENDMARKER

tokenize module:

~/cpython $ echo -n 'x' | ./python -m tokenize
1,0-1,1: NAME 'x'
2,0-2,0: ENDMARKER ''

The instrumentation to have the C tokenizer dump out its tokens is mine, can provide a diff to produce that output if needed.

taleinat · 2018-07-06T07:19:12Z

New changeset c4ef489 by Tal Einat (Ammar Askar) in branch 'master':
bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891)
c4ef489

taleinat · 2018-07-06T10:21:08Z

New changeset ab75d9e by Tal Einat (Ammar Askar) in branch '3.7':
[3.7] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891) (GH-8132)
ab75d9e

taleinat · 2018-07-06T10:22:28Z

New changeset 11c36a3 by Tal Einat (Ammar Askar) in branch '3.6':
[3.6] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891) (GH-8134)
11c36a3

taleinat · 2018-07-06T10:23:16Z

New changeset 7829bba by Tal Einat (Ammar Askar) in branch '2.7':
[2.7] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891) (bpo-8133)
7829bba

taleinat · 2018-07-06T10:24:51Z

Thanks for all of your work on this, Ammar!

asottile · 2018-10-21T16:10:27Z

This change in behaviour is breaking pycodestyle: PyCQA/pycodestyle#786

Perhaps it shouldn't have been backported (especially all the way to python2.7?)

taleinat · 2018-10-21T18:05:38Z

This was backported since it was considered a bug, but you are right that it broke backwards compatibility, and perhaps shouldn't have been backported.

Still, with 3.6.6 and 3.7.1 now released, that ship has sailed.

We could perhaps revert this on the 2.7 branch, but I feel that reverting this change only on 2.7 would just cause even more confusion.

asottile · 2018-10-21T23:42:04Z

I'm surprised this was classified as a bug! Though that's subjective so I get that it's difficult to decide what is and what isn't ¯\(ツ)/¯

ned-deily · 2018-10-22T00:45:22Z

Apparently this change also affected IPython. Perhaps we should add an entry to the whatsnew documents for 3.7.1 and 3.7.6:

https://docs.python.org/3/whatsnew/3.7.html#notable-changes-in-python-3-7-1

https://docs.python.org/3.6/whatsnew/3.6.html#notable-changes-in-python-3-6-7

taleinat · 2018-10-22T06:34:52Z

I'm sorry to have caused this mess, it was bad judgement on my part.

Adding mention in What's is a good idea, Ned, I'll do that.

taleinat · 2018-10-23T06:42:05Z

Ned, should this also be added to the 2.7 What's New? Or perhaps reverted on the 2.7 branch?

ned-deily · 2018-10-23T14:14:58Z

I don't have a strong opinion about 2.7 here. Ultimately, it's Benjamin's call. But it might make sense to revert for 2.7 since it hasn't been released yet.

benjaminp · 2018-10-23T15:59:31Z

Please revert in 2.7.

taleinat · 2018-10-24T06:33:29Z

See PR #54281 for reverting in 2.7.

gpshead · 2018-10-24T06:50:53Z

FYI, An example of other fallout from this change - patsy broke and needed this fix:

pydata/patsy@4f53bba#diff-53c70e68c6dfd4fe9b08427792cb2bd6

taleinat · 2018-10-24T06:57:35Z

See PR #54282 adding mention in "What's New".

gpshead · 2018-10-24T07:17:51Z

some pylint fallout appears to be addressed in pylint-dev/pylint@2698cbe

gpshead · 2018-10-24T07:20:14Z

New changeset dfba1f6 by Gregory P. Smith (Tal Einat) in branch 'master':
bpo-33899: Mention tokenize behavior change in What's New (GH-10073)
dfba1f6

miss-islington · 2018-10-24T07:32:42Z

New changeset 9a04762 by Miss Islington (bot) (Tal Einat) in branch '3.6':
[3.6] bpo-33899: Mention tokenize behavior change in What's New (GH-10073) (GH-10075)
9a04762

miss-islington · 2018-10-24T07:33:04Z

New changeset b4c9874 by Miss Islington (bot) (Tal Einat) in branch '3.7':
[3.7] bpo-33899: Mention tokenize behavior change in What's New (GH-10073) (GH-10074)
b4c9874

taleinat · 2018-10-24T07:40:29Z

Thanks for helping with the fallout from this, Gregory.

terryjreedy · 2018-10-24T14:53:03Z

bpo-33766 was about documenting the C tokenizer change, some years ago, that made end-of-file EOF and end-of-string EOS generate the NEWLINE token required to properly terminate statements. "The end of input also serves
as an implicit terminator for the final physical line."

Although the tokenizer module intentionally does not exactly mirror the C tokenizer (it adds COMMENT tokens), it plausibly seems like a bug that it was not changed along with the C tokenizer, as it has since been tokenizing valid code as grammatically invalid. But I agree that this fix is too disruptive for 2.7.

benjaminp · 2018-10-24T17:32:28Z

New changeset a1f45ec by Benjamin Peterson (Tal Einat) in branch '2.7':
bpo-33899: Revert tokenize module adding an implicit final NEWLINE (GH-10072)
a1f45ec

gpshead · 2018-10-29T22:27:03Z

https://bugs.python.org/issue35107 filed to track further fallout from this API change.

asmeurer · 2018-11-21T19:21:46Z

Is it expected behavior that comments produce NEWLINE if they don't have a newline and don't produce NEWLINE if they do (that is, '# comment' produces NEWLINE but '# comment\n' does not)?

brechtm · 2019-03-22T11:46:47Z

In order to adapt code to this change, can we assume that a NEWLINE token with an empty string only occurs right before the ENDMARKER?

ammaraskar added the 3.8 only security fixes label Jun 19, 2018

ammaraskar self-assigned this Jun 19, 2018

ammaraskar added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Jun 19, 2018

taleinat added the 3.7 (EOL) end of life label Jul 6, 2018

taleinat closed this as completed Jul 6, 2018

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tokenize module does not mirror "end-of-input" is newline behavior #78080

Tokenize module does not mirror "end-of-input" is newline behavior #78080

ammaraskar commented Jun 19, 2018

ammaraskar commented Jun 19, 2018

taleinat commented Jul 6, 2018

taleinat commented Jul 6, 2018

taleinat commented Jul 6, 2018

taleinat commented Jul 6, 2018

taleinat commented Jul 6, 2018

asottile mannequin commented Oct 21, 2018

taleinat commented Oct 21, 2018

asottile mannequin commented Oct 21, 2018

ned-deily commented Oct 22, 2018

taleinat commented Oct 22, 2018

taleinat commented Oct 23, 2018

ned-deily commented Oct 23, 2018

benjaminp commented Oct 23, 2018

taleinat commented Oct 24, 2018

gpshead commented Oct 24, 2018

taleinat commented Oct 24, 2018

gpshead commented Oct 24, 2018

gpshead commented Oct 24, 2018

miss-islington commented Oct 24, 2018

miss-islington commented Oct 24, 2018

taleinat commented Oct 24, 2018

terryjreedy commented Oct 24, 2018

benjaminp commented Oct 24, 2018

gpshead commented Oct 29, 2018

asmeurer mannequin commented Nov 21, 2018

brechtm mannequin commented Mar 22, 2019

Tokenize module does not mirror "end-of-input" is newline behavior #78080

Tokenize module does not mirror "end-of-input" is newline behavior #78080

Comments

ammaraskar commented Jun 19, 2018

ammaraskar commented Jun 19, 2018

taleinat commented Jul 6, 2018

taleinat commented Jul 6, 2018

taleinat commented Jul 6, 2018

taleinat commented Jul 6, 2018

taleinat commented Jul 6, 2018

asottile mannequin commented Oct 21, 2018

taleinat commented Oct 21, 2018

asottile mannequin commented Oct 21, 2018

ned-deily commented Oct 22, 2018

taleinat commented Oct 22, 2018

taleinat commented Oct 23, 2018

ned-deily commented Oct 23, 2018

benjaminp commented Oct 23, 2018

taleinat commented Oct 24, 2018

gpshead commented Oct 24, 2018

taleinat commented Oct 24, 2018

gpshead commented Oct 24, 2018

gpshead commented Oct 24, 2018

miss-islington commented Oct 24, 2018

miss-islington commented Oct 24, 2018

taleinat commented Oct 24, 2018

terryjreedy commented Oct 24, 2018

benjaminp commented Oct 24, 2018

gpshead commented Oct 29, 2018

asmeurer mannequin commented Nov 21, 2018

brechtm mannequin commented Mar 22, 2019