➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author jaredgrubb
Recipients jaredgrubb
Date 2008-02-25.02:06:47
SpamBayes Score 0.03303177
Marked as misclassified No
Message-id <1203905207.87.0.633637741257.issue2182@psf.upfronthosting.co.za>
In-reply-to
Content 
tokenize recognizes '\n' and '\r\n' as newlines, but does not tolerate '\r':

>>> s = "print 1\nprint 2\r\nprint 3\r"
>>> open('temp.py','w').write(s)
>>> exec(open('temp.py','r'))
1
2
3
>>> tokenize.tokenize(open('temp.py','r').readline)
1,0-1,5:	NAME	'print'
1,6-1,7:	NUMBER	'1'
1,7-1,8:	NEWLINE	'\n'
2,0-2,5:	NAME	'print'
2,6-2,7:	NUMBER	'2'
2,7-2,9:	NEWLINE	'\r\n'
3,0-3,5:	NAME	'print'
3,6-3,7:	NUMBER	'3'
3,7-3,8:	ERRORTOKEN	'\r'
4,0-4,0:	ENDMARKER	''
History
Date User Action Args
2008-02-25 02:06:48jaredgrubbsetspambayes_score: 0.0330318 -> 0.03303177
recipients: + jaredgrubb
2008-02-25 02:06:47jaredgrubbsetspambayes_score: 0.0330318 -> 0.0330318
messageid: <1203905207.87.0.633637741257.issue2182@psf.upfronthosting.co.za>
2008-02-25 02:06:47jaredgrubblinkissue2182 messages
2008-02-25 02:06:47jaredgrubbcreate