classification
Title: tokenize.generate_tokens doesn't always return logical line
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: LambertDW, docs@python, duncf, jackdied, miss-islington
Priority: normal Keywords: easy, patch, patch, patch, patch

Created on 2009-01-22 01:39 by duncf, last changed 2019-05-30 22:06 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11683 merged python-dev, 2019-01-26 18:12
PR 11683 merged python-dev, 2019-01-26 18:12
PR 11683 merged python-dev, 2019-01-26 18:12
PR 11683 merged python-dev, 2019-01-26 18:12
PR 13686 merged Anthony Sottile, 2019-05-30 21:48
Messages (4)
msg80353 - (view) Author: Duncan Findlay (duncf) Date: 2009-01-22 01:38
According to the documentation for tokenize.generate_tokens:

"The generator produces 5-tuples with these members: the token type; the
token string; a 2-tuple (srow, scol) of ints specifying the row and
column where the token begins in the source; a 2-tuple (erow, ecol) of
ints specifying the row and column where the token ends in the source;
and the line on which the token was found. The line passed (the last
tuple item) is the logical line; continuation lines are included."

It seems though that the "logical line" -- the last element of the tuple
is the physical line unless the token being returned spans beyond the
end of the line. As an example, consider a test file test.py:

foo = """
%s """ % 'bar'

>>> import pprint, tokenize
>>> pprint.pprint(list(tokenize.generate_tokens(open('test.py').readline)))
[(1, 'foo', (1, 0), (1, 3), 'foo = """\n'),
 (51, '=', (1, 4), (1, 5), 'foo = """\n'),
 (3, '"""\n%s """', (1, 6), (2, 6), 'foo = """\n%s """ % \'bar\'\n'),
 (51, '%', (2, 7), (2, 8), '%s """ % \'bar\'\n'),
 (3, "'bar'", (2, 9), (2, 14), '%s """ % \'bar\'\n'),
 (4, '\n', (2, 14), (2, 15), '%s """ % \'bar\'\n'),
 (0, '', (3, 0), (3, 0), '')]
>>> 

Since there is only one logical line, I would expect the first 6 tokens
to have the same 5th element.
msg84222 - (view) Author: Jack Diederich (jackdied) * (Python committer) Date: 2009-03-26 21:21
+1 for a docbug.  The last item is always the physical line and not the
logical line.  Some other examples:

if True and \
   False: pass

if (True and
    False): pass
msg343995 - (view) Author: miss-islington (miss-islington) Date: 2019-05-30 19:31
New changeset 1e36f75d634383eb243aa1798c0f2405c9ceb5d4 by Miss Islington (bot) (Andrew Carr) in branch 'master':
bpo-5028: fix doc bug for tokenize (GH-11683)
https://github.com/python/cpython/commit/1e36f75d634383eb243aa1798c0f2405c9ceb5d4
msg344008 - (view) Author: miss-islington (miss-islington) Date: 2019-05-30 22:06
New changeset 2a58b0636d1f620f8a85a2e4c030cc10551936a5 by Miss Islington (bot) (Anthony Sottile) in branch 'master':
bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686)
https://github.com/python/cpython/commit/2a58b0636d1f620f8a85a2e4c030cc10551936a5
History
Date User Action Args
2019-05-30 22:06:44miss-islingtonsetnosy: + miss-islington
messages: + msg344008
2019-05-30 21:48:48Anthony Sottilesetpull_requests: + pull_request13574
2019-05-30 19:33:23cheryl.sabellasetstatus: open -> closed

nosy: - miss-islington
keywords: patch, patch, patch, patch, easy
resolution: fixed
stage: patch review -> resolved
2019-05-30 19:31:56miss-islingtonsetnosy: + miss-islington
messages: + msg343995
2019-01-26 18:13:16python-devsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request11521
2019-01-26 18:13:07python-devsetkeywords: + patch
stage: needs patch -> needs patch
pull_requests: + pull_request11520
2019-01-26 18:12:57python-devsetkeywords: + patch
stage: needs patch -> needs patch
pull_requests: + pull_request11519
2019-01-26 18:12:47python-devsetkeywords: + patch
stage: needs patch -> needs patch
pull_requests: + pull_request11518
2019-01-23 22:45:26cheryl.sabellasetkeywords: + easy
stage: needs patch
components: + Documentation, - Library (Lib)
versions: + Python 3.8, - Python 2.6
2010-08-22 00:25:39georg.brandlsetassignee: docs@python

nosy: + docs@python
2009-03-26 21:21:08jackdiedsetnosy: + jackdied
messages: + msg84222
2009-01-22 02:46:48LambertDWsetnosy: + LambertDW
2009-01-22 01:39:02duncfcreate