classification
Title: _PyParser_TokenNames does not match up with the token.h numbers
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: georg.brandl, loewis, meador.inge, python-dev
Priority: low Keywords: easy, patch

Created on 2011-12-18 18:20 by meador.inge, last changed 2012-01-16 01:35 by meador.inge. This issue is now closed.

Files
File name Uploaded Description Edit
parser-debug-output.patch meador.inge, 2011-12-18 18:20 review
renumber-tokens.patch meador.inge, 2012-01-13 03:53 review
Messages (5)
msg149789 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-12-18 18:20
When making the changes to remove backticks in eb2f70fdbf32, the _PyParser_TokenNames table was incorrectly updated.  Now the indexes into _PyParser_TokenNames don't match the token numbers.  This can be seen in the output of 'python -d':

   $ echo '2 << 1' | ./python -d | grep Token
   ...
   Token NUMBER/'2' ... It's a token we know
   Token RIGHTSHIFT/'<<' ... It's a token we know
   Token NUMBER/'1' ... It's a token we know
   Token NEWLINE/'' ... It's a token we know
   Token NEWLINE/'' ... It's a token we know
   Token ENDMARKER/'' ... It's a token we know

The fix is trivial.  Patch attached.
msg149801 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-12-18 19:54
Is there a reason not to renumber token.h?
msg149809 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-12-19 00:53
> Is there a reason not to renumber token.h?

I thought about that, but at the time I wasn't sure whether or not that
would break anything.  I went with the current patch because it is lower
risk.

On the other hand, proper clients of token.h should only be using the macros and should have no knowledge of the numeric codes.  If they do use the numeric codes directly somehow, then that is their issue.  So, I guess it is OK to renumber token.h.
msg151161 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2012-01-13 03:53
Here is a patch that renumber the tokens.  I also regenerated token.py.
msg151313 - (view) Author: Roundup Robot (python-dev) Date: 2012-01-16 01:34
New changeset 1c0c6fa7341c by Meador Inge in branch '3.2':
Issue #13629: Renumber the tokens in token.h to match the _PyParser_TokenNames indexes.
http://hg.python.org/cpython/rev/1c0c6fa7341c

New changeset c0660d7cc1fe by Meador Inge in branch 'default':
Issue #13629: Renumber the tokens in token.h to match the _PyParser_TokenNames indexes.
http://hg.python.org/cpython/rev/c0660d7cc1fe
History
Date User Action Args
2012-01-16 01:35:52meador.ingesetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2012-01-16 01:34:27python-devsetnosy: + python-dev
messages: + msg151313
2012-01-13 03:53:43meador.ingesetfiles: + renumber-tokens.patch
nosy: + georg.brandl
messages: + msg151161

2011-12-19 00:53:09meador.ingesetmessages: + msg149809
2011-12-18 19:54:15loewissetnosy: + loewis
messages: + msg149801
2011-12-18 18:20:22meador.ingecreate