This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Non-UTF8 encoding line
Type: behavior Stage: needs patch
Components: Interpreter Core, Library (Lib) Versions: Python 3.11
Status: open Resolution:
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Arfrever, benjamin.peterson, eric.snow, iritkatriel, loewis, meador.inge, serhiy.storchaka, terry.reedy, vstinner
Priority: normal Keywords:

Created on 2013-09-07 15:35 by serhiy.storchaka, last changed 2022-04-11 14:57 by admin.

File name Uploaded Description Edit serhiy.storchaka, 2013-09-07 15:35
Messages (6)
msg197169 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-09-07 15:35
Here is a file which is accepted by Python interpreter but is rejected by the tokenize module.

$ ./python 
$ ./python -m tokenize error: invalid or missing encoding declaration for ''

Python itself checks that a line is UTF-8 encoded only if not found a magic comment. The tokenize module checks it before searching (issue14629).
msg197621 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-09-13 17:48
Which behavior do you propose to change? Does PEP263 specify the response to a self-contradictory encoding comment? What do you think it should say?

I raising is the better behavior. 

Idle notices that it cannot save the file with iso8859-15 encoding, so it saves it with utf-8 encoding instead. Any Python-aware editor that recognizes and uses the encoding cookie would have to either do the same or refuse to save until the encoding specification were changed.

Trying to run the file brings up a error box with SyntaxError: encoding problem: utf-8. Note that Idle runs a syntax check before creating a new subprocess, connecting to it, and trying to actually run the file. Closing the box takes one back to the editor as the spot where the error was noticed, which in this case is at the end of the line.
msg197918 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2013-09-16 16:44
Terry: the comment isn't self-contradictory. Instead, the file constitutes a perfectly fine iso-8859-15 byte sequence (albeit a meaningless one: any byte sequence is perfectly fine iso-8559-15, and, in a comment, any characters are allowed by the Python syntax).

AFAICT, the file also follows the wording of PEP 263, as it does not disallow additional text on the encoding line. Therefore, I think tokenize should accept the file.
msg197925 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-09-16 18:27
What about first line? Currently both Python interpreter and the tokenize module decode it from UTF-8 (actually due to bug #18960 Python interprets it twice, in different encodings). PEP 263 says:

    1. The complete Python source file should use a single encoding.
       Embedding of differently encoded data is not allowed and will
       result in a decoding error during compilation of the Python
       source code.

I conclude that the first line should be decoded with the encoding specified in the second line. We first should read the first line, check if it isn't a comment or contains encoding cookie, otherwise read the second line, determine the encoding, and decode read lines. Perhaps it will untangle issue18960 too.
msg197953 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2013-09-16 23:45
One issue at a time, please, and issue18960 is already its own issue.

In any case, item "1" in "Concepts" of PEP 263 is clear that any deviation from the declared encoding should cause a decoding error.
msg396104 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-06-18 22:17
I've reproduced the same in 3.11:

> .\python.bat
Running Release|x64 interpreter...
> .\python.bat -m tokenize
Running Release|x64 interpreter... error: invalid or missing encoding declaration for ''
Date User Action Args
2022-04-11 14:57:50adminsetgithub: 63161
2021-06-18 22:17:37iritkatrielsetnosy: + iritkatriel

messages: + msg396104
versions: + Python 3.11, - Python 3.3, Python 3.4
2020-03-06 20:30:24brett.cannonsetnosy: - brett.cannon
2016-12-06 12:14:59serhiy.storchakalinkissue28884 dependencies
2013-09-16 23:45:25loewissetmessages: + msg197953
2013-09-16 18:27:41serhiy.storchakasetassignee: serhiy.storchaka
messages: + msg197925
stage: needs patch
2013-09-16 16:44:24loewissetmessages: + msg197918
2013-09-13 22:05:35vstinnersetnosy: + vstinner
2013-09-13 17:48:59terry.reedysetnosy: + terry.reedy
messages: + msg197621
2013-09-07 15:36:32serhiy.storchakasetnosy: + meador.inge
2013-09-07 15:35:49serhiy.storchakacreate