PEP-597 is accepted.

I replied to INADA-san message on bpo-43552:
https://bugs.python.org/issue43552#msg389091

I had forgot to consider about UTF-8 mode while finishing PEP-597. If possible, I want to ignore UTF-8 mode when encoding="locale" is specified from Python 3.10.

In this case, the PEP-597 statement that open(filename, encoding="locale") is the same than open(filename) is wrong. It would mean that users which got the UTF-8 Mode enabled (implicitly or explicitly) would switch to a legacy encoding like latin1 rather than using the UTF-8 encoding, if they add encoding="locale" to their open() calls?

Since the final goal is to move everybody towards to UTF-8, I'm not sure how it's a good thing.

Since the final goal is to move everybody towards to UTF-8, I'm not sure how it's a good thing.

The final goal (the third motivation of the PEP-597) is changing the default encoding (i.e. encoding used when it is not specified) to UTF-8.

But forcing people to use UTF-8 even they specify locale encoding explicitly is not the goal. That's why I want to ignore UTF-8 mode when encoding="locale" is specified.

I think this is almost Windows-only issue, and "mbcs" can be used in Windows already. It is documented in https://docs.python.org/3/using/windows.html#utf-8-mode

So this is not a blocker. Just my preference.

I see different cases when open() is called with no encoding argument:

(A) User wants to use UTF-8: add encoding="utf-8"

(B) Windows user wants to use the ANSI code page of their computer, local file not intended to be shared with other computers: add encoding="mbcs". This makes the code specific to Windows ("mbcs" alias doesn't exist on Unix).

(C) User wants to use the locale encoding and is fine with the UTF-8 Mode: add encoding=getpreferredencoding(False)

(D) Unix user wants to use the locale encoding but not the UTF-8 Mode: encoding=get_current_locale_encoding() (function proposed in bpo-43552) or nl_langinfo(CODESET) (should work on any Python version). I don't know if nl_langinfo(CODESET) is available on Windows.

(E) User has no idea of what they are doing and don't understand anything to Unicode: please trust us and specify explicitly UTF-8 :-)

Apart the encoding="utf-8" case, I understand that they are two main complex cases:

(1) "UTF-8" in the UTF-8 Mode, or the locale encoding
(2) Always use the locale encoding, ignore the UTF-8 Mode

What I don't expect is the current behavior, before PEP-597. Who uses open() without specifying an encoding but always want to use the locale encoding? (case 2) So this use case is already broken when the UTF-8 Mode is enabled explicitly?

(1) "UTF-8" in the UTF-8 Mode, or the locale encoding
(2) Always use the locale encoding, ignore the UTF-8 Mode

What I don't expect is the current behavior, before PEP-597. Who uses open() without specifying an encoding but always want to use the locale encoding? (case 2) So this use case is already broken when the UTF-8 Mode is enabled explicitly?

Yes, it is broken already. So they can not use UTF-8 mode.

If encoding="locale" ignore UTF-8 mode, it save the use case. They can add encoding="locale" where they need to use locale/GetACP encoding and enable UTF-8 mode.

That's why it is important If we enable UTF-8 mode by default in the future.

New changeset 4827483 by Inada Naoki in branch 'master':
bpo-43510: Implement PEP-597 opt-in EncodingWarning. (GH-19481)
4827483

Yeah! Congrats INADA-san for implementing your PEP!

I created bpo-43651 to track fixing EncodingError in Python stdlibs.
I close this issue for now.

In bpo-43651, I found code pattern that it's difficult to use io.text_encoding():

    class OpenWrapper:
        def __new__(cls, *args, **kwargs):
            return open(*args, **kwargs)

kwargs["encoding"] = text_encoding(kwargs.get("encoding) doesn't work because open(filename, "b", encoding="locale") raises ValueError: binary mode doesn't take an encoding argument.

I think we should accept encoding="locale" even in binary mode. It makes easy to use text_encoding() and encoding="locale".

New changeset ff3c973 by Inada Naoki in branch 'master':
bpo-43510: PEP-597: Accept encoding="locale" in binary mode (GH-25103)
ff3c973

I'm sorry, I was wrong. Allowing encoding="locale" didn't help OpenWrapper. See #69294.

If we use encoding = text_encoding(encoding) in binary mode, open(filename, "rb") will be warned. This doesn't make sense at all.

Adding mode parameter to the text_encoding() doesn't make sense too. Because it is used for functions wrapping not only open(), but also TextIOWrapper().

So we must not call text_encoding() in binary mode. Allowing encoding="locale" in binary mode doesn't make it easy. I will revert #69290.

To me, it sounds really weird to accept an encoding when a file is opened in binary mode. open(filename, "rb", encoding="locale") looks like a bug.

On 31.03.2021 11:30, STINNER Victor wrote:

To me, it sounds really weird to accept an encoding when a file is opened in binary mode. open(filename, "rb", encoding="locale") looks like a bug.

Same here.

If encoding is used as an argument and then not used, this is a bug,
not a feature :-)

New changeset cfa1766 by Inada Naoki in branch 'master':
Revert "bpo-43510: PEP-597: Accept encoding="locale" in binary mode (GH-25103)" (bpo-25108)
cfa1766

New changeset bec8c78 by Inada Naoki in branch 'master':
bpo-43510: Fix emitting EncodingWarning from _io module. (GH-25146)
bec8c78