classification
Title: _PyCoreConfig: add stdio_encoding and stdio_errors
Type: Stage: resolved
Components: Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ncoghlan, vstinner
Priority: normal Keywords: patch

Created on 2018-08-23 23:44 by vstinner, last changed 2018-09-03 15:32 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 8881 merged vstinner, 2018-08-23 23:45
PR 8978 merged vstinner, 2018-08-28 20:10
PR 8979 merged vstinner, 2018-08-28 22:00
PR 8981 merged vstinner, 2018-08-28 22:22
PR 8982 merged vstinner, 2018-08-28 23:02
PR 8986 merged vstinner, 2018-08-29 07:43
PR 8987 merged vstinner, 2018-08-29 08:11
PR 8988 merged vstinner, 2018-08-29 09:09
PR 9002 merged vstinner, 2018-08-29 20:29
PR 9041 merged vstinner, 2018-09-03 11:58
PR 9045 merged vstinner, 2018-09-03 15:10
Messages (15)
msg323980 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-23 23:44
Currently, the code to select the encoding and error handler of sys.stdin, sys.stdout and sys.stderr is run later in Py_Initialize(). I propose to move most of this code into _PyCoreConfig_Read() and add stdio_encoding and stdio_errors to _PyCoreConfig to easily let Python embedders to select the encoding.

Attached PR implements this idea. During Py_Initialize(), the code still reads the LC_CTYPE locale to choose the error handler if stdio_errors is NULL. This check cannot be done earlier, since the LC_CTYPE locale is only set to the user locale in Py_Initialize(): after _PyCoreConfig_Read().

The Py_SetStandardStreamEncoding() function has been added by Nick Coghlan in bpo-16129.

    This new pre-initialization API allows embedding
    applications like Blender to force a particular
    encoding and error handler for the standard IO streams.
msg324290 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-28 21:26
New changeset 9e4994d410970fb4e75168401d159ba47a8f7108 by Victor Stinner in branch 'master':
bpo-34485: Enhance init_sys_streams() (GH-8978)
https://github.com/python/cpython/commit/9e4994d410970fb4e75168401d159ba47a8f7108
msg324294 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-28 22:16
New changeset 2c8ddcf4f14f3e4c87a6fe6678ab5ad09130c6ab by Victor Stinner in branch 'master':
bpo-34485: Fix _Py_InitializeCore() for C locale coercion (GH-8979)
https://github.com/python/cpython/commit/2c8ddcf4f14f3e4c87a6fe6678ab5ad09130c6ab
msg324297 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-28 23:13
New changeset 98c49c6ab239875e35a3c271bc8fabde6c9be804 by Victor Stinner in branch '3.7':
bpo-34485: Fix _Py_InitializeCore() for C locale coercion (GH-8979) (GH-8981)
https://github.com/python/cpython/commit/98c49c6ab239875e35a3c271bc8fabde6c9be804
msg324298 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-28 23:29
New changeset 124b9eb4e41ba1676dad954eec9a1fb3480794e4 by Victor Stinner in branch 'master':
bpo-34485: Add _Py_ClearStandardStreamEncoding() (GH-8982)
https://github.com/python/cpython/commit/124b9eb4e41ba1676dad954eec9a1fb3480794e4
msg324306 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-29 07:58
New changeset 315877dc361d554bec34b4b62c270479ad36a1be by Victor Stinner in branch 'master':
bpo-34485: stdout uses surrogateescape on POSIX locale (GH-8986)
https://github.com/python/cpython/commit/315877dc361d554bec34b4b62c270479ad36a1be
msg324309 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-29 09:01
New changeset 0b9ea4b211b24464c7d38f63e45e51c275c52dcd by Victor Stinner in branch '3.7':
[3.7] bpo-34485: stdout uses surrogateescape on POSIX locale (GH-8986) (GH-8987)
https://github.com/python/cpython/commit/0b9ea4b211b24464c7d38f63e45e51c275c52dcd
msg324311 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-29 09:25
New changeset 177d921c8c03d30daa32994362023f777624b10d by Victor Stinner in branch 'master':
bpo-34485, Windows: LC_CTYPE set to user preference (GH-8988)
https://github.com/python/cpython/commit/177d921c8c03d30daa32994362023f777624b10d
msg324315 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-29 09:47
New changeset dfe0dc74536dfb6f331131d9b2b49557675bb6b7 by Victor Stinner in branch 'master':
bpo-34485: Add _PyCoreConfig.stdio_encoding (GH-8881)
https://github.com/python/cpython/commit/dfe0dc74536dfb6f331131d9b2b49557675bb6b7
msg324316 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-29 09:50
Oh wow, I didn't expect that it would be so complicated to "just" get the encoding of sys.stdout... I had to fix many bugs to be able to do that!
msg324349 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-29 20:56
New changeset cf21504194927b2f22132f48effea69eb8ade751 by Victor Stinner in branch 'master':
bpo-34485: Emit C locale coercion warning later (GH-9002)
https://github.com/python/cpython/commit/cf21504194927b2f22132f48effea69eb8ade751
msg324382 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-08-30 13:53
Yeah, there were some good reasons I went with the relatively brute force option of provideding Blender with a new config API call back in bpo-16129 - as we've seen, actually fixing it properly has been a multi-year multi-person effort :)
msg324384 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-30 14:46
> Yeah, there were some good reasons I went with the relatively brute force option of provideding Blender with a new config API call back in bpo-16129 - as we've seen, actually fixing it properly has been a multi-year multi-person effort :)

Yeah, I see ;-) Thanks for Py_SetStandardStreamEncoding(), it fixed Blender use case!
msg324511 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-03 12:38
New changeset f01b2a1b84ee08df73a78cf1017eecf15e3cb995 by Victor Stinner in branch 'master':
bpo-34544: Fix setlocale() in pymain_read_conf() (GH-9041)
https://github.com/python/cpython/commit/f01b2a1b84ee08df73a78cf1017eecf15e3cb995
msg324526 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-03 15:32
New changeset 73b00becbdd40f6a80cfa00abf1ae650a2b5e454 by Victor Stinner in branch 'master':
bpo-34544: pymain_read_conf() don't change LC_ALL (GH-9045)
https://github.com/python/cpython/commit/73b00becbdd40f6a80cfa00abf1ae650a2b5e454
History
Date User Action Args
2018-09-03 15:32:34vstinnersetmessages: + msg324526
2018-09-03 15:10:17vstinnersetpull_requests: + pull_request8507
2018-09-03 12:38:24vstinnersetmessages: + msg324511
2018-09-03 11:58:12vstinnersetpull_requests: + pull_request8503
2018-08-30 14:46:23vstinnersetmessages: + msg324384
2018-08-30 13:53:23ncoghlansetnosy: + ncoghlan
messages: + msg324382
2018-08-29 20:56:09vstinnersetmessages: + msg324349
2018-08-29 20:29:00vstinnersetpull_requests: + pull_request8473
2018-08-29 09:50:04vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg324316

stage: patch review -> resolved
2018-08-29 09:47:33vstinnersetmessages: + msg324315
2018-08-29 09:25:23vstinnersetmessages: + msg324311
2018-08-29 09:09:15vstinnersetpull_requests: + pull_request8461
2018-08-29 09:01:45vstinnersetmessages: + msg324309
2018-08-29 08:11:13vstinnersetpull_requests: + pull_request8460
2018-08-29 07:58:21vstinnersetmessages: + msg324306
2018-08-29 07:43:18vstinnersetpull_requests: + pull_request8459
2018-08-28 23:29:09vstinnersetmessages: + msg324298
2018-08-28 23:13:33vstinnersetmessages: + msg324297
2018-08-28 23:02:01vstinnersetpull_requests: + pull_request8456
2018-08-28 22:22:32vstinnersetpull_requests: + pull_request8455
2018-08-28 22:16:58vstinnersetmessages: + msg324294
2018-08-28 22:00:52vstinnersetpull_requests: + pull_request8453
2018-08-28 21:26:36vstinnersetmessages: + msg324290
2018-08-28 20:10:28vstinnersetpull_requests: + pull_request8452
2018-08-23 23:45:06vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request8358
2018-08-23 23:44:09vstinnercreate