classification
Title: os.device_encoding() doesn't respect the UTF-8 Mode
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: methane, vstinner
Priority: normal Keywords: patch

Created on 2020-11-01 18:12 by vstinner, last changed 2020-11-04 10:26 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23083 merged vstinner, 2020-11-01 18:47
PR 23086 merged vstinner, 2020-11-01 21:40
PR 23109 merged vstinner, 2020-11-02 15:06
PR 23119 merged vstinner, 2020-11-02 22:03
Messages (8)
msg380156 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-01 18:12
When the UTF-8 Mode is enabled, sys.stdout.encoding is always UTF-8, whereas os.devide_encoding(sys.stdout.fileno()) returns the locale encoding. os.devide_encoding() must return UTF-8 when the UTF-8 Mode is used.
msg380158 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-01 18:45
In bpo-42208, I added C _Py_GetLocaleEncoding() function and Python _locale._get_locale_encoding() function.
msg380159 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-01 19:59
New changeset 82458b6cdbae3b849dc11d0d7dc2ab06ef0451c4 by Victor Stinner in branch 'master':
bpo-42236: Enhance _locale._get_locale_encoding() (GH-23083)
https://github.com/python/cpython/commit/82458b6cdbae3b849dc11d0d7dc2ab06ef0451c4
msg380166 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-01 22:07
New changeset e662c398d87f136497f8ec672e83657ae3a599e0 by Victor Stinner in branch 'master':
bpo-42236: Use UTF-8 encoding if nl_langinfo(CODESET) fails (GH-23086)
https://github.com/python/cpython/commit/e662c398d87f136497f8ec672e83657ae3a599e0
msg380234 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-02 15:50
New changeset 4b9aad49992a825d8c76e428ed1aca81dd3878b2 by Victor Stinner in branch 'master':
bpo-42236: Enhance init and encoding documentation (GH-23109)
https://github.com/python/cpython/commit/4b9aad49992a825d8c76e428ed1aca81dd3878b2
msg380261 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-11-03 05:15
I don't think UTF-8 mode should override os.device_encoding() on Windows.

UTF-8 mode can be used to ignore legacy locale encoding, and os.device_encoding() uses the locale encoding on Unix. So overriding it make sense.

But locale encoding and console cp are different on Windows. Users may want to know console cp even when they want to use UTF-8 by default for reading/writing text files.
msg380313 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-04 10:20
New changeset 3529718925f40d14ed48d281d809187bc7314a14 by Victor Stinner in branch 'master':
bpo-42236: os.device_encoding() respects UTF-8 Mode (GH-23119)
https://github.com/python/cpython/commit/3529718925f40d14ed48d281d809187bc7314a14
msg380314 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-04 10:26
> But locale encoding and console cp are different on Windows. Users may want to know console cp even when they want to use UTF-8 by default for reading/writing text files.

You're right. My final change leaves Windows unchanged.
History
Date User Action Args
2020-11-04 10:26:56vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg380314

stage: patch review -> resolved
2020-11-04 10:20:19vstinnersetmessages: + msg380313
2020-11-03 05:15:52methanesetnosy: + methane
messages: + msg380261
2020-11-02 22:03:11vstinnersetpull_requests: + pull_request22036
2020-11-02 15:50:02vstinnersetmessages: + msg380234
2020-11-02 15:06:40vstinnersetpull_requests: + pull_request22026
2020-11-01 22:07:32vstinnersetmessages: + msg380166
2020-11-01 21:40:25vstinnersetpull_requests: + pull_request22003
2020-11-01 19:59:42vstinnersetmessages: + msg380159
2020-11-01 18:47:17vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request22001
2020-11-01 18:45:39vstinnersetmessages: + msg380158
2020-11-01 18:12:56vstinnercreate