This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author jaredgrubb
Recipients jaredgrubb
Date 2008-02-25.01:59:16
SpamBayes Score 0.00511074
Marked as misclassified No
Message-id <1203904758.19.0.84784431119.issue2180@psf.upfronthosting.co.za>
In-reply-to
Content
tokenize does not handle line joining properly, as the following string
fails the CPython tokenizer but passes the tokenize module.

Example 1:
>>> s = "if 1:\n  \\\n  #hey\n  print 1"
>>> exec s
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 3
    #hey
       ^
SyntaxError: invalid syntax

>>> tokenize.tokenize(StringIO(s).readline)
1,0-1,2:	NAME	'if'
1,3-1,4:	NUMBER	'1'
1,4-1,5:	OP	':'
1,5-1,6:	NEWLINE	'\n'
2,0-2,2:	INDENT	'  '
3,2-3,6:	COMMENT	'#hey'
3,6-3,7:	NEWLINE	'\n'
4,2-4,7:	NAME	'print'
4,8-4,9:	NUMBER	'1'
5,0-5,0:	DEDENT	''
5,0-5,0:	ENDMARKER	''
History
Date User Action Args
2008-02-25 01:59:18jaredgrubbsetspambayes_score: 0.00511074 -> 0.00511074
recipients: + jaredgrubb
2008-02-25 01:59:18jaredgrubbsetspambayes_score: 0.00511074 -> 0.00511074
messageid: <1203904758.19.0.84784431119.issue2180@psf.upfronthosting.co.za>
2008-02-25 01:59:17jaredgrubblinkissue2180 messages
2008-02-25 01:59:16jaredgrubbcreate