Author Joshua.Landau
Recipients Joshua.Landau
Date 2014-06-02.18:24:00
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1401733440.77.0.120432390188.issue21642@psf.upfronthosting.co.za>
In-reply-to
Content
By the docs,

    Except at the beginning of a logical line or in
    string literals, the whitespace characters space,
    tab and formfeed can be used interchangeably to
    separate tokens. Whitespace is needed between two
    tokens only if their concatenation could otherwise
    be interpreted as a different token
    (e.g., ab is one token, but a b is two tokens).

"_ if 1else _" should compile equivalently to "_ if 1 else _".

The tokenize module does this correctly:

    import io
    import tokenize

    def print_tokens(string):
        tokens = tokenize.tokenize(io.BytesIO(string.encode("utf8")).readline)    

        for token in tokens:
            print("{:12}{}".format(tokenize.tok_name[token.type], token.string))

    print_tokens("_ if 1else _")
    #>>> ENCODING    utf-8
    #>>> NAME        _
    #>>> NAME        if
    #>>> NUMBER      1
    #>>> NAME        else
    #>>> NAME        _
    #>>> ENDMARKER   

but it fails when compiled with, say, "compile", "eval" or "ast.parse".

    import ast

    compile("_ if 1else _", "", "eval")
    #>>> Traceback (most recent call last):
    #>>>   File "", line 32, in <module>
    #>>>   File "<string>", line 1
    #>>>     _ if 1else _
    #>>>           ^
    #>>> SyntaxError: invalid token

    eval("_ if 1else _")
    #>>> Traceback (most recent call last):
    #>>>   File "", line 40, in <module>
    #>>>   File "<string>", line 1
    #>>>     _ if 1else _
    #>>>           ^
    #>>> SyntaxError: invalid token

    ast.parse("_ if 1else _")
    #>>> Traceback (most recent call last):
    #>>>   File "", line 48, in <module>
    #>>>   File "/usr/lib/python3.4/ast.py", line 35, in parse
    #>>>     return compile(source, filename, mode, PyCF_ONLY_AST)
    #>>>   File "<unknown>", line 1
    #>>>     _ if 1else _
    #>>>           ^
    #>>> SyntaxError: invalid token

Further, some other forms work:

    1 if 0b1else 0
    #>>> 1

    1 if 1jelse 0
    #>>> 1

See

    http://stackoverflow.com/questions/23998026/why-isnt-this-a-syntax-error-in-python

particularly,

    http://stackoverflow.com/a/23998128/1763356

for details.
History
Date User Action Args
2014-06-02 18:24:00Joshua.Landausetrecipients: + Joshua.Landau
2014-06-02 18:24:00Joshua.Landausetmessageid: <1401733440.77.0.120432390188.issue21642@psf.upfronthosting.co.za>
2014-06-02 18:24:00Joshua.Landaulinkissue21642 messages
2014-06-02 18:24:00Joshua.Landaucreate