➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author lys.nikolaou
Recipients gvanrossum, lys.nikolaou, pablogsal, serhiy.storchaka
Date 2020-04-10.23:01:31
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1586559691.81.0.857739000475.issue40246@roundup.psfhosted.org>
In-reply-to
Content 
At the moment, if the prefix is invalid, it gets tokenized as a NAME node and the string following is tokenized as a STRING node with no prefix.

╰─ echo "ur''" | ./python -m tokenize
1,0-1,2:            NAME           'ur'           
1,2-1,4:            STRING         "''"           
1,4-1,5:            NEWLINE        '\n'           
2,0-2,0:            ENDMARKER      ''

╰─ echo "f''" | ./python -m tokenize  
1,0-1,3:            STRING         "f''"          
1,3-1,4:            NEWLINE        '\n'           
2,0-2,0:            ENDMARKER      ''
History
Date User Action Args
2020-04-10 23:01:31lys.nikolaousetrecipients: + lys.nikolaou, gvanrossum, serhiy.storchaka, pablogsal
2020-04-10 23:01:31lys.nikolaousetmessageid: <1586559691.81.0.857739000475.issue40246@roundup.psfhosted.org>
2020-04-10 23:01:31lys.nikolaoulinkissue40246 messages
2020-04-10 23:01:31lys.nikolaoucreate