classification
Title: tokenize module: normal lines, not "logical"
Type: Stage:
Components: Documentation Versions: Python 2.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: georg.brandl Nosy List: georg.brandl, noam, orsenthil
Priority: normal Keywords:

Created on 2008-06-02 11:33 by noam, last changed 2008-06-08 14:48 by noam. This issue is now closed.

Messages (4)
msg67635 - (view) Author: Noam Yorav-Raphael (noam) Date: 2008-06-02 11:33
Hello,

The documentation of the tokenize module says: "The line passed is the
*logical* line; continuation lines are included."

Some background: The tokenize module splits a python source into tokens,
and says for each token where it begins and where it ends, in the format
of (row, offset). This note in the documentation made me think that
continuation lines are considered as one line, and made me break my head
how I should find the offset of the token in the original string. The
truth is that the row number is simply the index of the line as returned
by the readline function, and it's very simple to reconstruct the string
offset.

I suggest that this will be changed to something like "The line passed
is the index of the string returned by the readline function, plus 1.
That is, the first string returned is called line 1, the second is
called line 2, and so on."

Thanks,
Noam
msg67660 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2008-06-03 17:14
> The documentation of the tokenize module says: "The line passed is the
> *logical* line; continuation lines are included."
> 
> I suggest that this will be changed to something like "The line passed
> is the index of the string returned by the readline function, plus 1.
> That is, the first string returned is called line 1, the second is
> called line 2, and so on."

The emphasis of *logical* may help us understand that it is a complete line.

I find the wording of solution bit awkward, tough I am able to get what it is
trying to say.

- Index of string returned by readline function ??  and plus 1. ??

How about, 

The line passed is the *logical* non-blank line; continuation lines are included.
The row counting starts from one.
msg67829 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-06-08 09:00
I got it wrong too at first; the "logical line" refers to the last tuple
item, the line string, not the line *number* passed.

It should now be clear as of r64037.
msg67832 - (view) Author: Noam Yorav-Raphael (noam) Date: 2008-06-08 14:47
Can I suggest that you also add something like "The row indices in the
(row, column) tuples, however, are physical, and don't treat
continuation lines specially."?

It's just that it took me some time to understand your clarification,
since the row indices I thought the documentation talks about are also
tuple items, they just happen to be the first in the tuple, not the last.
History
Date User Action Args
2008-06-08 14:48:00noamsetmessages: + msg67832
2008-06-08 09:00:27georg.brandlsetstatus: open -> closed
resolution: fixed
messages: + msg67829
2008-06-03 17:15:37orsenthilsetnosy: + orsenthil
messages: + msg67660
2008-06-02 11:33:25noamcreate