classification
Title: Py_Initialize affects the console
Type: behavior Stage: resolved
Components: C API, IO, Unicode, Windows Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, ezio.melotti, paul.moore, steve.dower, tim.golden, twoone3, vstinner, zach.ware
Priority: normal Keywords:

Created on 2020-10-05 08:51 by twoone3, last changed 2020-10-05 13:26 by eryksun. This issue is now closed.

Files
File name Uploaded Description Edit
Screenshots.zip twoone3, 2020-10-05 10:31 Test result
Messages (11)
msg378004 - (view) Author: twoone3 (twoone3) Date: 2020-10-05 08:51
When I set Py_LegacyWindowsStdioFlag to 0, after Py_Initialize, the console will forcibly change what encoding is displayed. At this time, my standard stream output is utf-8, py's print is utf-8, chcp 65001 is useless, and still garbled.  When I set Py_LegacyWindowsStdioFlag to 1, the console encoding is mbcs, the standard stream output is UTF-8 and not garbled, but the print of py is gbk encoding, and my system is Windows server 2016. I hope that Py_Initialize will not affect the console in subsequent versions.  Do any behavior, I suggest you test the coding problem in Chinese
msg378012 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-05 09:47
Python supports Unicode in a Windows console session by using the console API's wide-character functions (i.e. ReadConsoleW and WriteConsoleW) with UTF-16 encoded text. This is implemented in the io stack via io._WindowsConsoleIO, and for PyOS_Readline (e.g. builtin `input`) via _PyOS_WindowsConsoleReadline. The UTF-16 aspect is an internal detail that's presented externally as UTF-8 by automatically converting between the two.

Notwithstanding the behavior of third-party packages, CPython intentionally makes no changes to a console session's global settings, including:

    * input & output codepages
    * input & output modes (except for msvcrt.getwch, etc)
    * cursor size and visibility
    * screen-buffer size, font, colors, & attributes
    * window size
    * window title
msg378014 - (view) Author: twoone3 (twoone3) Date: 2020-10-05 10:31
So how does this explain
msg378016 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-05 10:45
I suggest you to use https://docs.python.org/dev/c-api/init_config.html#c.PyConfig.PyConfig_InitIsolatedConfig API instead of the legacy Py_Initialize() API.
msg378021 - (view) Author: twoone3 (twoone3) Date: 2020-10-05 12:31
This problem has bothered me for a month, thank you for helping me solve this problem, you can close it
msg378023 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-05 13:03
> This problem has bothered me for a month, thank you for helping me solve this problem, you can close it

I close the issue, but I'm not sure how you fixed it. Don't hesitate commenting to explain how you solved it ;-)
msg378024 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-05 13:07
Having looked at the screenshots, it seems that your issue is in part due to non-legacy mode not setting the standard I/O files to binary mode (_O_BINARY) from their default ANSI text mode (_O_TEXT). This is not a problem on its own. 

The real issue is that you're not using an isolated configuration, as Victor suggested, so the LC_CTYPE locale gets set to the default user locale instead of "C". When writing to the console in ANSI text mode with a configured locale other than the default "C" locale, the C runtime _write() function does a double translation from the locale encoding to the console codepage encoding via the internal function write_double_translated_ansi_nolock().
msg378026 - (view) Author: twoone3 (twoone3) Date: 2020-10-05 13:12
I used the link you gave me https://docs.python.org/dev/c-api/init_config.html#c.PyConfig.PyConfig_InitIsolatedConfig Click to open and turn up to see the box, I follow that box  The example inside is done, and the coding problem of the console is solved, but I hope that cpython can provide a more concise API, remove useless functions or variables, so that it is convenient for you to maintain and easy for users to read.
msg378027 - (view) Author: twoone3 (twoone3) Date: 2020-10-05 13:14
That's it
msg378028 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-05 13:14
PyConfig_InitIsolatedConfig() reduces side effects on the process. For example, it doesn't set the LC_CTYPE locale and it doesn't change the standard streams (stdio).

config_init_stdio() is not called in an isolated configuration:

https://github.com/python/cpython/blob/dcc54215ac1eb4b6fab2a9ffe1abcdf3ac4bb77e/Python/initconfig.c#L1855

https://github.com/python/cpython/blob/dcc54215ac1eb4b6fab2a9ffe1abcdf3ac4bb77e/Python/initconfig.c#L1809-L1817
msg378029 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-05 13:26
> config_init_stdio() is not called in an isolated configuration:

config_init_stdio wasn't being called anyway since Py_Initialize uses _PyConfig_InitCompatConfig. The issue was primarily due to the LC_CTYPE locale being set to the default user locale, as I discussed in msg378024.

Note that for legacy mode, i.e. Py_LegacyWindowsStdioFlag = 1, there's no simple way to not modify the standard I/O files. The io stack needs binary mode. You'd have to use and modify duped file descriptors.
History
Date User Action Args
2020-10-05 13:26:53eryksunsetmessages: + msg378029
2020-10-05 13:14:44vstinnersetmessages: + msg378028
2020-10-05 13:14:35twoone3setmessages: + msg378027
2020-10-05 13:12:34twoone3setmessages: + msg378026
2020-10-05 13:07:39eryksunsetmessages: + msg378024
2020-10-05 13:03:41vstinnersetstatus: open -> closed
resolution: not a bug
messages: + msg378023

stage: resolved
2020-10-05 12:31:22twoone3setmessages: + msg378021
2020-10-05 10:45:00vstinnersetmessages: + msg378016
2020-10-05 10:31:27twoone3setfiles: + Screenshots.zip

messages: + msg378014
2020-10-05 09:50:04eryksunsetnosy: + paul.moore, tim.golden, ezio.melotti, vstinner, zach.ware, steve.dower
components: + Unicode, Windows, IO
2020-10-05 09:47:56eryksunsetnosy: + eryksun
messages: + msg378012
2020-10-05 08:51:42twoone3create