classification
Title: Deprecating / removing token.ISTERMINAL/ISNONTERMINAL
Type: Stage:
Components: Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: BTaskaya, gvanrossum, lys.nikolaou, pablogsal
Priority: normal Keywords:

Created on 2020-10-28 20:57 by BTaskaya, last changed 2020-10-29 00:54 by gvanrossum.

Messages (4)
msg379844 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python committer) Date: 2020-10-28 20:57
Since the parser & symbol modules are removed, there is no real use case for ISTERMINAL/ISNONTERMINAL since the only input that contains grammar symbols is the input of the tokenize, which all its outputs are terminals. Which restricts the practicality of token.ISTERMINAL/ISNONTERMINAL. I'd propose we either deprecate them or remove them completely.

Search over the most populer PyPI packages (a set of 4k) showed there are only 2 projects that uses these functions, which all cases coupled with the parser module.
pyglet-1.5.7/pyglet/text/formats/attributed.py
133:        if token.ISNONTERMINAL(node[0]):
gdata-2.0.18/tests/coverage.py
328:        if token.ISNONTERMINAL(tree[0]):

If we are going through a deprecation period, I will also propose to remove ISEOF, since it can be easily replaced and very under used. The only project that uses token.ISEOF in my dataset is asttokens, which I would say all cases there can be easily replaced tok.type == token.EOF

asttokens-2.0.4/asttokens/mark_tokens.py
105:           not token.ISEOF(t.type)):

Also, these functions are originally reflections of C macros used in the past, and since they are not used right now, we can also remove those macros.

I have no opinions about the token.NT_OFFSET though, but there are a couple of occurrences so we might keep that? 
Paste-3.4.4/paste/util/PySourceColor.py
205:ARGS = token.NT_OFFSET + 1
206:DOUBLECOMMENT = token.NT_OFFSET + 2

ipython-7.18.1/IPython/utils/PyColorize.py
54:_KEYWORD = token.NT_OFFSET + 1
55:_TEXT    = token.NT_OFFSET + 2
msg379847 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-10-28 23:03
Since there are so few projects, maybe you can just contact them?
msg379848 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python committer) Date: 2020-10-28 23:09
> Since there are so few projects, maybe you can just contact them?

For NT_OFFSET?
msg379852 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-10-29 00:54
I thought for all?
History
Date User Action Args
2020-10-29 00:54:11gvanrossumsetmessages: + msg379852
2020-10-28 23:09:12BTaskayasetmessages: + msg379848
2020-10-28 23:03:31gvanrossumsetmessages: + msg379847
2020-10-28 23:00:25BTaskayasetkeywords: - 3.4regression
2020-10-28 20:57:30BTaskayacreate