Message379844
Since the parser & symbol modules are removed, there is no real use case for ISTERMINAL/ISNONTERMINAL since the only input that contains grammar symbols is the input of the tokenize, which all its outputs are terminals. Which restricts the practicality of token.ISTERMINAL/ISNONTERMINAL. I'd propose we either deprecate them or remove them completely.
Search over the most populer PyPI packages (a set of 4k) showed there are only 2 projects that uses these functions, which all cases coupled with the parser module.
pyglet-1.5.7/pyglet/text/formats/attributed.py
133: if token.ISNONTERMINAL(node[0]):
gdata-2.0.18/tests/coverage.py
328: if token.ISNONTERMINAL(tree[0]):
If we are going through a deprecation period, I will also propose to remove ISEOF, since it can be easily replaced and very under used. The only project that uses token.ISEOF in my dataset is asttokens, which I would say all cases there can be easily replaced tok.type == token.EOF
asttokens-2.0.4/asttokens/mark_tokens.py
105: not token.ISEOF(t.type)):
Also, these functions are originally reflections of C macros used in the past, and since they are not used right now, we can also remove those macros.
I have no opinions about the token.NT_OFFSET though, but there are a couple of occurrences so we might keep that?
Paste-3.4.4/paste/util/PySourceColor.py
205:ARGS = token.NT_OFFSET + 1
206:DOUBLECOMMENT = token.NT_OFFSET + 2
ipython-7.18.1/IPython/utils/PyColorize.py
54:_KEYWORD = token.NT_OFFSET + 1
55:_TEXT = token.NT_OFFSET + 2 |
|
Date |
User |
Action |
Args |
2020-10-28 20:57:30 | BTaskaya | set | recipients:
+ BTaskaya, gvanrossum, lys.nikolaou, pablogsal |
2020-10-28 20:57:30 | BTaskaya | set | messageid: <1603918650.53.0.755514465217.issue42187@roundup.psfhosted.org> |
2020-10-28 20:57:30 | BTaskaya | link | issue42187 messages |
2020-10-28 20:57:30 | BTaskaya | create | |
|