Message149815
The proposed documentation text seems too complicated and language expert speaky to me. We should try to link to standard definitions when possible to reduce the text here. For example, I believe the "Operators" and "Delimiters" tokens in the "Lexical Analysis" section of the docs (http://docs.python.org/dev/reference/lexical_analysis.html#operators) are exactly what we are trying to describe when referencing "literal tokens" and "affected tokens".
I like Nick's idea to introduce a new attribute for the exact type, while keeping the tuple structure itself backwards compatible. Attached is a patch for 3.3 that updates the docs, adds exact_type, adds new unit tests, and adds a new CLI option for displaying token names using the exact type.
An example of the new CLI option is:
$ echo '1+2**4' | ./python -m tokenize
1,0-1,1: NUMBER '1'
1,1-1,2: OP '+'
1,2-1,3: NUMBER '2'
1,3-1,5: OP '**'
1,5-1,6: NUMBER '4'
1,6-1,7: NEWLINE '\n'
2,0-2,0: ENDMARKER ''
$ echo '1+2**4' | ./python -m tokenize -e
1,0-1,1: NUMBER '1'
1,1-1,2: PLUS '+'
1,2-1,3: NUMBER '2'
1,3-1,5: DOUBLESTAR '**'
1,5-1,6: NUMBER '4'
1,6-1,7: NEWLINE '\n'
2,0-2,0: ENDMARKER '' |
|
Date |
User |
Action |
Args |
2011-12-19 04:51:23 | meador.inge | set | recipients:
+ meador.inge, akuchling, terry.reedy, ncoghlan, gpolo, ezio.melotti, docs@python, eric.snow |
2011-12-19 04:51:23 | meador.inge | set | messageid: <1324270283.73.0.22714523648.issue2134@psf.upfronthosting.co.za> |
2011-12-19 04:50:23 | meador.inge | link | issue2134 messages |
2011-12-19 04:50:22 | meador.inge | create | |
|