➜
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2014-03-17 11:56 by FunkyBob, last changed 2022-04-11 14:58 by admin. This issue is now closed.
| Files | ||||
|---|---|---|---|---|
| File name | Uploaded | Description | Edit | |
| 20956-tokenize-docs.diff | sam.kimbrel, 2014-04-14 19:06 | review | ||
| Messages (5) | |||
|---|---|---|---|
| msg213853 - (view) | Author: Curtis Maloney (FunkyBob) | Date: 2014-03-17 11:56 | |
Although 3.x does yield a named-tuple, 2.x does not, despite the documentation:
The exact type can be determined by checking the token string field on the named tuple returned from tokenize.tokenize() for the character sequence that identifies a specific operator token.
(I'd much rather this functionality be back-ported from 3, as it's quite simple)
|
|||
| msg213854 - (view) | Author: Eric V. Smith (eric.smith) *  |
Date: 2014-03-17 12:41 | |
The 2.x documentation says generate_tokens returns a 5-tuple, not a namedtuple: http://docs.python.org/2/library/tokenize.html "The generator produces 5-tuples with these members: ..." The section you quote is talking about tokenize.tokenize, and I think that documentation should be fixed. Changing to namedtuple would be a new feature, which are not being added to 2.7. |
|||
| msg213856 - (view) | Author: Curtis Maloney (FunkyBob) | Date: 2014-03-17 12:57 | |
Corrected title. And, yeah, figured it wouldn't be added as a new feature, hence reporting this as a Documentation bug. |
|||
| msg216174 - (view) | Author: Sam Kimbrel (sam.kimbrel) * | Date: 2014-04-14 19:06 | |
Attached patch to update 2.7 docs to refer to the plain-old-tuple returned from generate_tokens(). |
|||
| msg216197 - (view) | Author: Roundup Robot (python-dev)  |
Date: 2014-04-14 20:17 | |
New changeset d4f5a88b94b4 by Terry Jan Reedy in branch '2.7': Closes #20956: 2.7 tokenize does not produce named tuples. Patch by Sam Kimbrel. http://hg.python.org/cpython/rev/d4f5a88b94b4 |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:58:00 | admin | set | github: 65155 |
| 2014-04-14 20:17:44 | python-dev | set | status: open -> closed nosy: + python-dev messages: + msg216197 resolution: fixed stage: resolved |
| 2014-04-14 20:01:54 | terry.reedy | set | assignee: docs@python -> terry.reedy |
| 2014-04-14 19:06:09 | sam.kimbrel | set | files:
+ 20956-tokenize-docs.diff nosy: + sam.kimbrel messages: + msg216174 keywords: + patch |
| 2014-03-21 21:51:50 | terry.reedy | set | nosy:
+ terry.reedy |
| 2014-03-17 12:57:21 | FunkyBob | set | messages:
+ msg213856 title: tokenize module claims generate_tokens returns namedtuple, but it doesn't -> tokenize module claims tokenize.tokenize returns namedtuple, but it doesn't |
| 2014-03-17 12:41:56 | eric.smith | set | nosy:
+ eric.smith messages: + msg213854 |
| 2014-03-17 11:56:05 | FunkyBob | create | |