This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author jhermann
Recipients
Date 2001-03-21.03:21:12
SpamBayes Score
Marked as misclassified
Message-id
In-reply-to
Content
Logged In: YES 
user_id=39128

tokenize.NL seems to be an empty line, as opposed to 
token.NEWLINE which is a logical line end. Insofar NL is 
comparable to COMMENT, since it has no semantic content and 
is ignorable by a parser.

1 a = \
2 2
3
4 b=4

emits these tokens:

1,0-1,1:        NAME    'a'
1,2-1,3:        OP      '='
2,0-2,1:        NUMBER  '2'
2,1-2,2:        NEWLINE '\012'
3,0-3,1:        NL      '\012'
4,0-4,1:        NAME    'b'
4,1-4,2:        OP      '='
4,2-4,3:        NUMBER  '4'
4,3-4,4:        NEWLINE '\012'
5,0-5,0:        ENDMARKER       ''
History
Date User Action Args
2007-08-23 13:53:37adminlinkissue410155 messages
2007-08-23 13:53:37admincreate