This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Recipients daniel.urban, eric.snow, ezio.melotti,, sandro.tosi
Date 2011-08-05.20:42:40
SpamBayes Score 2.94599e-07
Marked as misclassified No
Message-id <>
I think I can make these changes independently and issue two patches, one fixing the problems with untokenize listed here, and another improving tokenize.

I've just noticed a third bug in untokenize: in full mode, it doesn't handle backslash-continued lines correctly.

    Python 3.3.0a0 (default:c099ba0a278e, Aug  2 2011, 12:35:03) 
    [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from io import BytesIO
    >>> from tokenize import tokenize, untokenize
    >>> untokenize(tokenize(BytesIO('1 and \\\n not 2'.encode('utf8')).readline))
    b'1 andnot 2'
Date User Action Args
2011-08-05 20:42:42gdr@garethrees.orgsetrecipients: +, ezio.melotti, daniel.urban, sandro.tosi, eric.snow
2011-08-05 20:42:41gdr@garethrees.orgsetmessageid: <>
2011-08-05 20:42:41gdr@garethrees.orglinkissue12691 messages
2011-08-05 20:42:40gdr@garethrees.orgcreate