Type something like the following at the interpreter prompt:

>>> 04208
  File "<stdin>", line 1
    04208
        ^
SyntaxError: invalid token


This is not very descriptive.  I suggest "SyntaxError: invalid octal digit".

I was looking at the current hg tip.  The lexer emits E_TOKEN errors for the following cases:
- invalid hex digit
- invalid octal digit
- invalid binary digit
- invalid digit in float exponent
- old-style octal constant (e.g. 001), which is no longer accepted

I think I can come up with a patch that replaces them all with different error codes (E_BAD_HEX_DIGIT etc.) and different error messages.  Does that sound like an acceptable change?  (I never contributed non-documentation patches to CPython before.)

Note that "invalid token" is emitted only on invalid first digit:

>>> 0b2
  File "<stdin>", line 1
    0b2
     ^
SyntaxError: invalid token
>>> 0b02
  File "<stdin>", line 1
    0b02
       ^
SyntaxError: invalid syntax


See also issue1634034.

Oh, hey, PEP 3127 actually asks for a better error message than "invalid token" for this case: http://www.python.org/dev/peps/pep-3127/#tokenizer-exception-handling

So here's a tentative patch to test the waters.  I still haven't figured out how to write tests for it (is Lib/test/test_tokenize.py the right place for that?), and I haven't manually tested it either, because building CPython tip fails for me with a strange link error about _PyTraceMalloc_Init/_PyTraceMalloc_Fini.

If there's some documentation I should read about submitting CPython patches, or some IRC channel I should join, please tell me!

I resolved my compilation problems (by running 'make distclean').

There are some problems with my patch:
- "leading" is misspelled (as "lleading")
- literals like 0x1z, 0o18, 0b12, 1.2e-1x produce a generic "invalid syntax" message instead of the specific "bad digit in hex/octal/binary/float literal"
- 1.2e-x produces "bad digit in float literal" correctly, but the caret points to the '-' sign instead of the 'x' character

I see that I misunderstood Serhiy's comment.  I assumed he meant the caret will be pointing to the 1st digit that is invalid.  Instead what actually happens is that E_TOKEN is emitted only if the 1st digit after the 0x/0o/0b prefix is invalid.

So, I get the nice error messages for 0b2, 0o8, 0xz and 0e-x (but the caret incorrectly points to the previous character).

Here's version 2 of the patch:
- spelling error fixed
- 0b2, 0o8, 0xg, 0e-x show the expected error at the expected place
- 0b02, 0o08, 0x0g, 0e-0x continue produce a generic "syntax error" because the tokenizer thinks these are a pair of valid tokens (0b0 followed by 2 etc.), and the error comes from the parser

Version 3 of the patch catches bad digits in the middle of a literal, like this:

>>> 0o01010118001
  File "<stdin>", line 1
    0o01010118001
             ^
SyntaxError: bad digit in octal literal

Here are some unit tests for the new syntax errors (in test_syntax.py; test_tokenize.py turned out to be totally unrelated).

One possible shortcoming: they do not test the column of the syntax error.

Updated test that checks the syntax error offset as well.

I think I'm done with the iterations.  I'll be waiting for feedback.

Marius, you can remove unneeded patches (click on the "edit" link and then press the "Unlink" button).

I can look at this when the time for 3.5 rolls around.

I had filed issue 29146 but eventually found this, which has patches to fix the exception messages. Could someone please look at this and get it incorporated into python?

Fixed elsewhere.

>>> 04208
  File "<stdin>", line 1
    04208
        ^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers
>>> 0o38
  File "<stdin>", line 1
    0o38
       ^
SyntaxError: invalid digit '8' in octal literal