This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author serhiy.storchaka
Recipients Albert-Jan Nijburg, benjamin.peterson, matrixise, meador.inge, r.david.murray, serhiy.storchaka, vstinner
Date 2017-05-30.13:02:08
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1496149328.29.0.825653192179.issue30455@psf.upfronthosting.co.za>
In-reply-to
Content
PR 1860 makes following files be generated from token.py:

* Include/token.h
* Parser/token.c. New file containing the array of token names _PyParser_TokenNames, and functions PyToken_OneChar(), PyToken_TwoChars(), PyToken_ThreeChars(), moved from Parser/tokenizer.c.
* Doc/library/token-list.inc. New file containing the list of token.py constants, it is included in Doc/library/token.rst.

New Makefile target regen-token regenerates these files.

The dict EXACT_TOKEN_TYPES that maps operator strings to token names now is automatically generated and moved from tokenize.py to token.py. Tokens COMMENT, NL and ENCODING used only in tokenize.py now are added in token.py as in issue25324.
History
Date User Action Args
2017-05-30 13:02:08serhiy.storchakasetrecipients: + serhiy.storchaka, vstinner, benjamin.peterson, r.david.murray, meador.inge, matrixise, Albert-Jan Nijburg
2017-05-30 13:02:08serhiy.storchakasetmessageid: <1496149328.29.0.825653192179.issue30455@psf.upfronthosting.co.za>
2017-05-30 13:02:08serhiy.storchakalinkissue30455 messages
2017-05-30 13:02:08serhiy.storchakacreate