Rietveld Code Review Tool
Help | Bug tracker | Discussion group | Source code | Sign in
(15)

#3353: make built-in tokenizer available via Python C API

Can't Edit
Can't Publish+Mail
Start Review
Created:
5 years, 7 months ago by fredrik
Modified:
4 years, 9 months ago
Reviewers:
ncoghlan
CC:
fredrik_effbot.org, amaury.forgeotdarc, djmitche, flossyfriend_hotmail.com, meadori, berkerpeksag, storchaka, superluser, andrew.carrera_gmail.com, Jim Fasarakis-Hilliard
Visibility:
Public.

Patch Set 1 #

Patch Set 2 #

Total comments: 5

Patch Set 3 #

Unified diffs Side-by-side diffs Delta from patch set Stats Patch
Doc/c-api/tokenizer.rst View 1 2 1 chunk +96 lines, -0 lines 0 comments Download
Doc/c-api/utilities.rst View 1 2 1 chunk +1 line, -0 lines 0 comments Download
Include/Python.h View 1 2 1 chunk +1 line, -0 lines 0 comments Download
Include/token.h View 1 2 1 chunk +61 lines, -61 lines 0 comments Download
Include/tokenizer.h View 1 2 1 chunk +83 lines, -0 lines 0 comments Download
Lib/token.py View 1 2 1 chunk +1 line, -1 line 0 comments Download
Makefile.pre.in View 1 2 1 chunk +1 line, -1 line 0 comments Download
Modules/parsermodule.c View 1 2 57 chunks +111 lines, -111 lines 0 comments Download
Parser/acceler.c View 1 2 2 chunks +3 lines, -3 lines 0 comments Download
Parser/firstsets.c View 1 2 2 chunks +2 lines, -2 lines 0 comments Download
Parser/grammar1.c View 1 2 3 chunks +4 lines, -4 lines 0 comments Download
Parser/grammar.c View 1 2 5 chunks +12 lines, -12 lines 0 comments Download
Parser/listnode.c View 1 2 2 chunks +5 lines, -5 lines 0 comments Download
Parser/parser.c View 1 2 6 chunks +16 lines, -16 lines 0 comments Download
Parser/parsetok.c View 1 2 6 chunks +8 lines, -8 lines 0 comments Download
Parser/pgen.c View 1 2 10 chunks +17 lines, -17 lines 0 comments Download
Parser/tokenizer.c View 1 2 61 chunks +135 lines, -135 lines 0 comments Download
Parser/tokenizer.h View 1 2 1 chunk +0 lines, -82 lines 0 comments Download
Python/ast.c View 1 2 50 chunks +99 lines, -99 lines 0 comments Download
Python/pythonrun.c View 1 2 1 chunk +3 lines, -3 lines 0 comments Download

Messages

Total messages: 1
Nick Coghlan
4 years, 9 months ago #1
Main missing piece is to also publish the token definitions to actually make use
of the results of PyTokenizer_Get()

https://bugs.python.org/review/3353/diff/14564/Doc/c-api/tokenizer.rst
File Doc/c-api/tokenizer.rst (right):

https://bugs.python.org/review/3353/diff/14564/Doc/c-api/tokenizer.rst#newcode5
Doc/c-api/tokenizer.rst:5: Tokenizing Python Code
This could use a usage example to demonstrate chaining the pieces together
create a tokenizer, use it to turn a data stream into a token stream and
comparing the token stream against the token header #defines.

https://bugs.python.org/review/3353/diff/14564/Include/tokenizer.h
File Include/tokenizer.h (right):

https://bugs.python.org/review/3353/diff/14564/Include/tokenizer.h#newcode11
Include/tokenizer.h:11: #include "token.h"      /* For token types */
This suggests the token #defines will also need to be published and
appropriately prefixed.

https://bugs.python.org/review/3353/diff/14564/Include/tokenizer.h#newcode13
Include/tokenizer.h:13: typedef struct PyTokenizer_State PyTokenizer_State;
As a now public API, this should be included from Python.h.

https://bugs.python.org/review/3353/diff/14564/Include/tokenizer.h#newcode16
Include/tokenizer.h:16: #define MAXINDENT 100   /* Max indentation level */
This needs a PYTOKENIZER_* prefix.

https://bugs.python.org/review/3353/diff/14564/Include/tokenizer.h#newcode21
Include/tokenizer.h:21: STATE_NORMAL        /* have a codec associated with
input */
PYTOKENIZER_* prefix
Sign in to reply to this message.

RSS Feeds Recent Issues | This issue
This is Rietveld 894c83f36cb7+