Author belopolsky
Recipients Retro, belopolsky, brian.curtin, eric.araujo, georg.brandl, lemburg, mark.dickinson, michael.foord, pitrou, rhettinger
Date 2010-12-03.05:33:33
SpamBayes Score 5.71788e-05
Marked as misclassified No
Message-id <>
In-reply-to <>
On Thu, Dec 2, 2010 at 11:49 PM, Éric Araujo <> wrote:
> Whether 1+2j is a literal or an expression is debatable.
> I think +1 is an expression but 1+2j is a literal; neither should have a space.

With respect to implementation there is no debate:

[TokenInfo(type=57 (ENCODING), string='utf-8', start=(0, 0), end=(0,
0), line=''),
 TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+1j'),
 TokenInfo(type=53 (OP), string='+', start=(1, 1), end=(1, 2), line='1+1j'),
 TokenInfo(type=2 (NUMBER), string='1j', start=(1, 2), end=(1, 4), line='1+1j'),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]

[TokenInfo(type=57 (ENCODING), string='utf-8', start=(0, 0), end=(0,
0), line=''),
 TokenInfo(type=53 (OP), string='-', start=(1, 0), end=(1, 1), line='-1'),
 TokenInfo(type=2 (NUMBER), string='1', start=(1, 1), end=(1, 2), line='-1'),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]

(Who designed the tokenize interface, btw?  I took me 3 tries to come
up with the incantation above.)

> I’m not sure the language reference and the actual implementation are in agreement here
> (I have peephole optimizations in mind).

Literals are atomic to the tokenizer.  AST processes a stream of
tokens.  Peephole optimizations are irrelevant because these are hacks
that operate on the bytecode when the lexical structure is all but
Date User Action Args
2010-12-03 05:36:26belopolskyunlinkissue10562 messages
2010-12-03 05:33:35belopolskysetrecipients: + belopolsky, lemburg, georg.brandl, rhettinger, mark.dickinson, pitrou, eric.araujo, Retro, michael.foord, brian.curtin
2010-12-03 05:33:33belopolskylinkissue10562 messages
2010-12-03 05:33:33belopolskycreate