This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author eryksun
Recipients eryksun, mdileep, xtreak
Date 2019-04-21.20:46:42
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1555879602.79.0.247655990559.issue35149@roundup.psfhosted.org>
In-reply-to
Content
In Windows, Python defaults to the system ANSI codepage (e.g. 1252 in the West) for non-console standard I/O. For the case of a `for /f` loop in CMD, stdout is a pipe, so Python defaults to writing ANSI encoded text to its end of the pipe. I recommend overriding the encoding to UTF-8 using the PYTHONIOENCODING environment variable. 

CMD uses the console's output codepage to decode bytes read from its end of the pipe, so the batch script should temporarily change the console codepage to UTF-8 via `chcp.com 65001`. Note that this won't work if CMD is running without a console (i.e. a DETACHED_PROCESS), in which case it defaults to ANSI. (I don't recommend running without a console. If no window is required, use CREATE_NO_WINDOW or a hidden window instead.) First save the current console codepage, parsed from the output of running `chcp.com` without arguments. Then you can restore the original console codepage after the loop.

After decoding the text, CMD's `echo` command writes to the console using the wide-character WriteConsoleW function, so there's no problem at this stage -- up to the limits of the console's text support. FYI, in lieu of Python getting the blame for this too, the Windows console can only render Basic Multilingual Plane (i.e. UCS-2) text, and it doesn't support automatic font fallback or complex scripts. If the console can't display a character, it displays the font's default glyph (e.g. an empty rectangle), or two default glyphs for a surrogate pair. However, we can still copy text from the console in this case.
History
Date User Action Args
2019-04-21 20:46:42eryksunsetrecipients: + eryksun, xtreak, mdileep
2019-04-21 20:46:42eryksunsetmessageid: <1555879602.79.0.247655990559.issue35149@roundup.psfhosted.org>
2019-04-21 20:46:42eryksunlinkissue35149 messages
2019-04-21 20:46:42eryksuncreate