classification
Title: tokenize.py: missing EXACT_TOKEN_TYPES
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Aaron.Meurer, Jim Fasarakis-Hilliard, meador.inge, ned.deily, skrah, terry.reedy, veky
Priority: normal Keywords: patch

Created on 2015-07-12 19:29 by skrah, last changed 2018-06-08 20:55 by skrah.

Files
File name Uploaded Description Edit
tokenize.diff skrah, 2015-07-12 19:29 review
Pull Requests
URL Status Linked Edit
PR 666 merged Jim Fasarakis-Hilliard, 2017-03-14 19:50
Messages (14)
msg246663 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-07-12 19:29
ELLIPSIS and RARROW are missing in EXACT_TOKEN_TYPES. Patch attached.
msg246692 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2015-07-13 15:31
This looks good to me.  A few questions.

The patch is fixing an obvious bug, but there could be code that
depends on the current broken behavior (the table has been incorrect
since Python 3.3).  How do we feel about such breaking changes?
I am OK with this one.

Will you add tests?
msg246693 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-07-13 17:04
I think with the ASYNC/AWAIT changes any code using tokenize.py
will have to be updated anyway in 3.5 and in 3.7 (when ASYNC/AWAIT
will finally be real keywords).

So this is probably an ideal time for the change.  I'll add tests.
msg246694 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2015-07-13 17:11
On Mon, Jul 13, 2015 at 12:04 PM, Stefan Krah <report@bugs.python.org> wrote:

> I think with the ASYNC/AWAIT changes any code using tokenize.py
> will have to be updated anyway in 3.5 and in 3.7 (when ASYNC/AWAIT
> will finally be real keywords).

Agreed.

> So this is probably an ideal time for the change.  I'll add tests.

Thanks.
msg289568 - (view) Author: Vedran Čačić (veky) * Date: 2017-03-14 06:05
We obviously missed this, at least for 3.5... any chance of having it for 3.7?
msg289610 - (view) Author: Jim Fasarakis-Hilliard (Jim Fasarakis-Hilliard) * Date: 2017-03-14 17:17
Stefan, do you have time to make a PR for this?
msg289613 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2017-03-14 17:46
Not right now, but if anyone makes a PR I can click the merge button. :)
msg289615 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2017-03-14 20:19
Thanks for the PR! -- Merged.
msg318855 - (view) Author: Aaron Meurer (Aaron.Meurer) Date: 2018-06-06 17:33
I would suggest adding this to the what's new document https://docs.python.org/3.7/whatsnew/3.7.html. The change affects user-facing code (the exact_type attribute of TokenInfo is OP for ... and -> tokens prior to this patch). 

I would also point out that this directly contradicts the documentation (from https://docs.python.org/3/library/tokenize.html, "To simplify token stream handling, all operator and delimiter tokens and Ellipsis are returned using the generic OP token type. The exact type can be determined by checking the exact_type property on the named tuple returned from tokenize.tokenize()."), so I don't see why it can't be backported.
msg318856 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2018-06-06 18:23
I would not mind backporting this. Let's think about it for a while.
msg319087 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-06-08 17:35
The 3.6.6rc is next Monday.  3.6.8 will be the last 3.6 maintenance release.
msg319103 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2018-06-08 20:08
Yikes, I wasn't aware that we are that late in the 3.6 release cycle.

Perhaps it would be unusual to change it now, even though it is a bug fix.

What do others think?
msg319106 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-08 20:50
> What do others think?

+1 for adding a mention in the 3.7 What's New doc (anyone willing to produce a PR for (master Doc/whatsnew/3.7.rst ?)

-1 for backporting to 3.6.  Since 3.6 has been out there so long and is nearing the end of its maintenance life, it seems to me to not be a good idea to make a change like this that is more likely to break existing code than to fix something that a small number of 3.6 users may have been waiting for.
msg319107 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2018-06-08 20:55
I agree, it would be strange to backport now.
History
Date User Action Args
2018-06-08 20:55:26skrahsetstage: backport needed ->
messages: + msg319107
versions: + Python 3.7, - Python 3.6
2018-06-08 20:50:05ned.deilysetnosy: + ned.deily
messages: + msg319106
2018-06-08 20:08:07skrahsetmessages: + msg319103
2018-06-08 17:35:53terry.reedysetnosy: + terry.reedy
messages: + msg319087
2018-06-06 18:23:39skrahsetstatus: closed -> open
versions: + Python 3.6, - Python 3.7
messages: + msg318856

assignee: skrah ->
resolution: fixed ->
stage: resolved -> backport needed
2018-06-06 17:33:02Aaron.Meurersetnosy: + Aaron.Meurer
messages: + msg318855
2017-03-14 20:21:57skrahsetstatus: open -> closed
2017-03-14 20:19:53skrahsetversions: + Python 3.7
messages: + msg289615

assignee: skrah
components: + Library (Lib)
resolution: fixed
stage: patch review -> resolved
2017-03-14 19:50:08Jim Fasarakis-Hilliardsetpull_requests: + pull_request550
2017-03-14 17:46:55skrahsetmessages: + msg289613
2017-03-14 17:17:14Jim Fasarakis-Hilliardsetnosy: + Jim Fasarakis-Hilliard
messages: + msg289610
2017-03-14 06:05:21vekysetnosy: + veky
messages: + msg289568
2015-07-13 17:11:48meador.ingesetmessages: + msg246694
2015-07-13 17:04:23skrahsetmessages: + msg246693
2015-07-13 15:31:24meador.ingesettype: behavior
messages: + msg246692
stage: patch review
2015-07-12 19:29:37skrahcreate