Python parser (Parser/tokenizer.c) has a translate_into_utf8() function to decode a string from the input encoding and encode it to UTF-8.

This function is unnecessary if the input string is already encoded to UTF-8, which is something common nowadays. Linux, Mac OS X and many other operating systems are now using UTF-8 as the default locale encoding, UTF-8 is the default encoding for Python scripts, etc. compile(), eval() and exec() functions pass UTF-8 encoded strings to the parser.

Attached patch adds an input_is_utf8 flag to the tokenizer to skip translate_into_utf8() if the input string is already encoded to UTF-8.

The patch has an issue, importing test.bad_coding2 (UTF-8 with a BOM) does not raise a SyntaxError anymore.

The parser should check that the input is actually valid UTF-8 data.

> The parser should check that the input is actually valid UTF-8 data.

Ah yes, correct. It looks like input data is still checked for valid
UTF-8 data. I suppose that the byte strings should be decoded from
UTF-8 because Python 3 manipulates Unicode strings, not byte strings.

The patch only skips calls to translate_into_utf8(str, tok->encoding),
calls to translate_into_utf8(str, tok->enc) are unchanged (notice:
encoding != enc :-)).

But it looks like translate_into_utf8(str, tok->enc) is not called if
tok->enc is NULL.

If tok->encoding is "utf-8" and tok->enc is NULL, maybe the input
string is not decoded from UTF-8. But it sounds strange, because
Python uses Unicode strings.

Don't trust me, I would prefer an explanation of Benjamin who knows
better than me the parser internals :-)

tok->enc and tok->encoding should always have the same value, except that tok->enc gets set earlier.

tok->enc is used when parsing from strings, to remember what codec to use. For file based parsing, the codec object created knows what encoding to use; for string-based parsing, tok->enc stores the encoding.

If the code is to be simplified, unifying the cases of string-based parsing and file-based parsing might be a worthwhile goal.

> If the code is to be simplified, unifying the cases of string-based parsing and file-based parsing might be a worthwhile goal.

Ah yes, it enc and encoding attributes are almost the same, it would be nice to merge them! But I'm not sure that I understand, do you prefer to merge them in this issue or in a new issue?